This is a great question, because grid cells, which are involved in processing spatial information about our surroundings, are located in a brain region that is part of a larger memory system thought to be responsible for the feeling of familiarity. After considering their function in detail, however, I think it seems more likely that a different system of neurons, place cells, plays a stronger role in providing us with the sense that a new locale is familiar—a feeling called “déjà visité.”
In any environment, the brain must keep track of the distinct locations within the surrounding area (say, at the kitchen table versus in front of the refrigerator). It also must note how these different locales relate to one another (the table is three feet to the right of the fridge, for instance). Place cells are involved in the former type of processing; each place cell corresponds to a specific location in an environment and fires when you pass through that spot.
In contrast, grid cells work in a network to produce a kind of internal coordinate system, noting information about distance and direction. These neurons do not correspond to a specific location but become active across several regularly spaced points in any setting. The geometric arrangement of these cells, relative to one another and to the external setting, ultimately helps us form a mental map of a certain environment.
Grid cells are located in the entorhinal cortex, a brain region that processes information before sending it to the hippocampus, the area where place cells are located. Because we know that place cells have a unique firing pattern for nearly every experience, it is likely that the hippocampus, and not primarily the entorhinal cortex, decides whether a location is novel or being revisited. When a strange place is experienced as familiar, it may be because the activated ensemble of place cells at that location happens to be similar to a pattern of activity that was elicited by a previous locale. Link
Is our sense of familiarity more tied to visual modalities than others? Consider the overwhelming familiarity aroused through smell. The key question here is not to consider which sense modalities yield familiarity, so much as which sense modalities yield false familiarity and why. As far as I know, I've never had an olfactory sense of déjà vu.
However, I think I have experienced déjà vu sparked by conversational content rather than visual scene. Considering the discussion above, perhaps the sense of familiarity is neurochemically triggered because I'm in some particular location and not strictly because of auditory or conceptual features of the conversation?
Deja vu might reveal yet another failure of our introspective capacity to pinpoint the causal underpinnings of our conscious experience.