Gemini Embedding 2 and True Multimodal Vectors

Gemini Embedding 2 went GA one week ago, and one detail in Google's architecture diagram is worth a closer look.

Each modality enters as its own input. Text as text, image as image, video as video. All of them are then mapped into a single shared vector space. At first sight that looks like the same setup as previous multimodal systems. The difference is in how the model is trained.

Earlier "multimodal" embedding models were essentially several unimodal encoders bolted together. A text encoder, an image encoder, a video encoder, each trained separately and aligned through projection layers afterwards. The cross-modal relationships were patched in rather than learned end-to-end.

Gemini Embedding 2 is trained jointly across all modalities from the start. The shared semantic space is what the model directly optimizes for, not something assembled later.

You can send a photo, a description, and an audio review in a single request, and get back one vector that encodes the whole thing. The model covers five modalities natively (text, image, video, audio, documents) rather than the usual two-modality pairings, which means you can mix any combination of them in your inputs. And the backbone is shared across modalities rather than duplicated per encoder, so the model benefits from cross-modal transfer during training.

The inputs are kept separate despite being trained together because early fusion creates an optimization conflict. When you mix all modalities at the input level, the dominant one (almost always text) tends to crowd out the others during training. Each modality ends up underrepresented compared to what a dedicated model would give you. Keeping unimodal inputs but sharing the downstream space avoids that trap.

The benchmarks line up with the design choice:

▸ Text-to-image: 93.4 vs 84.0 for the closest competitor ▸ Text-to-video retrieval: 68.8 vs 60.3 ▸ Multilingual MTEB: 69.9 (top of leaderboard)

For RAG pipelines or multimodal search, this removes a real engineering tax. No more juggling separate models per modality, no more alignment layers to maintain, and the vectors actually encode cross-modal meaning out of the box.

Curious what you'd build on top of this. Anyone already migrating a pipeline to try it out?