Google pushed two embedding models out the door within 24 hours of each other in early October 2026: a hosted API update and an open-weight sibling. Perplexity answered the same week with its own multimodal retrieval models. If you build search, recommendation, or retrieval-augmented generation (RAG) systems, the embedding layer you picked six months ago is already behind. This tutorial walks through Gemini Embedding 2, Google’s hosted multimodal embedding model, and shows you how to wire it into a working RAG pipeline that searches text, images, and PDFs from a single vector space.
By the end you will have a complete Python project: an ingestion script that embeds mixed-media documents, a Chroma vector store, a cross-modal retrieval function, and a FastAPI endpoint that returns grounded answers. Every code block below runs as written against the live Gemini API as of October 13, 2026.
Don't miss new tech stories on Google
Add Tech Insider once in the Google app and our stories appear in your news suggestions.
What Is Gemini Embedding 2?
Gemini Embedding 2 (model ID gemini-embedding-2) is Google’s current embedding model inside the Gemini API, and Google describes it as the first multimodal embedding model offered through that API. Instead of running separate models for text, image, audio, and video, Gemini Embedding 2 maps all four input types, plus PDF documents, into one shared vector space. That matters for RAG because you can embed a user’s text query and match it directly against embedded screenshots, product photos, or scanned pages without a separate cross-modal alignment step.
The model accepts up to 8,192 tokens of input across all modalities combined in a single request. Output dimensionality is flexible, ranging from 128 to 3,072 dimensions, with Google’s documentation recommending 768, 1,536, or 3,072 as the practical options. The default output is the full 3,072-dimension vector, which you can truncate downward later in this guide to cut storage costs.
Per-request modality limits are specific: up to six images (PNG or JPEG) per call, up to 180 seconds of audio (MP3 or WAV), up to 120 seconds of video (MP4 or MOV, encoded in H.264, H.265, AV1, or VP9), and a single PDF of up to six pages. If your source documents exceed those caps, you chunk them before embedding, which this tutorial covers in Step 8.
One detail that trips up teams migrating from the previous generation: Gemini Embedding 2 vectors are not compatible with the older gemini-embedding-001 embedding space. If you have an existing index built on the prior model, you have to re-embed everything rather than mixing old and new vectors in the same collection.
The flexible dimensionality comes from a training technique called Matryoshka Representation Learning (MRL), which trains a single model so that the first N numbers of its full-length vector already form a valid, independently useful embedding. That’s why you can ask for a 256-dimension vector and get a usable result instead of a corrupted one — the model was explicitly trained to make shorter prefixes of the full vector work on their own, nested inside each other like the matryoshka dolls the technique is named after. Practically, this means you can start a project at 3,072 dimensions for maximum accuracy during development, then drop to 768 once you’ve validated the pipeline and want to control storage cost, without re-architecting anything downstream.
Another behavioral change from the prior generation worth knowing before you build: when you pass multiple pieces of content directly inside one embed_content call, Gemini Embedding 2 returns a single aggregated embedding representing all of them together, whereas the previous model returned one embedding per input item. If your pipeline assumes one-vector-per-input — which the ingestion script in this tutorial does, by calling the API once per file — that assumption holds. It only breaks if you later try to batch multiple unrelated documents into a single request to save on round trips, which will silently blend their embeddings into one averaged vector instead of giving you separate results.
Why Multimodal RAG Needs a New Embedding Approach in 2026
Text-only RAG has been the default architecture since 2023: chunk a document, embed the chunks, store vectors, retrieve the nearest neighbors, and feed them to a chat model. That pattern breaks down the moment your knowledge base includes product photos, scanned contracts, architecture diagrams, or screen recordings, because a text-only embedding model has no way to represent what’s inside an image or a video frame. Teams have historically patched around this with OCR, image captioning, or a second vision model bolted onto the pipeline, which adds latency and a second point of failure.
Gartner forecast in 2023 that “more than 80% of enterprises will have used generative AI APIs or models, or deployed generative-AI-enabled applications in production environments, by 2026, up from less than 5% in 2023”. That prediction window closes this year, and the practical effect is that most production RAG systems are no longer prototypes — they are expected to handle whatever a user uploads, not just clean text.
Google, Perplexity, and the open-source Gemma team all shipped unified multimodal embedding models within the same week of October 2026, which is a signal that the single-vector-space approach has moved from research curiosity to production default. A shared embedding space means one model call, one index, and one similarity search covering every modality you store, instead of maintaining parallel text and vision pipelines that have to be reconciled at query time.
There’s also a cost argument for consolidating onto one multimodal embedding model instead of running separate text and vision pipelines. Every additional model in a retrieval stack is another dependency to version, another API to monitor for outages, and another place where a schema change can silently break production. Teams that have already migrated report the biggest win isn’t retrieval accuracy — it’s the operational simplification of deleting an entire vision-captioning microservice because the embedding model now handles images natively. That operational case is a large part of why Gemini Embedding 2 is worth adopting even for teams whose current RAG system technically works: the maintenance burden of the old multi-model approach compounds every time a new content type gets added to the knowledge base.
Prerequisites and Versions You’ll Need
Confirm each of these before starting Step 1. Version mismatches in the Google GenAI SDK are the single most common cause of broken embedding calls in this tutorial.
| Requirement | Minimum Version | Notes |
|---|---|---|
| Python | 3.10 or newer | 3.11+ recommended for async FastAPI performance |
| google-genai SDK | 1.x (current) | Install via pip; replaces the older google-generativeai package |
| chromadb | 0.5.x or newer | Local persistent vector store used in this tutorial |
| fastapi | 0.115 or newer | Used in the deployment step |
| uvicorn | 0.30 or newer | ASGI server for running the FastAPI app |
| pypdf | 5.x | For extracting and chunking PDF pages before embedding |
| Google AI Studio account | N/A | Needed to generate a Gemini API key |
| Gemini API tier | Free or Paid | Free tier works for this tutorial; paid tier needed for production volume |
You will also need roughly 500MB of free disk space for the Chroma persistent store and a handful of sample files (a few PDFs and images) to embed during testing. Nothing in this tutorial requires a GPU — all embedding calls happen through Google’s hosted API, not a local model.
Step 1-3: Get Your API Key, Install the SDK, and Structure Your Project
Step 1: Generate a Gemini API Key
Open Google AI Studio, sign in with a Google account, and create an API key from the API keys panel. Copy the key immediately — Google does not show the full string again after you navigate away. Store it as an environment variable rather than hardcoding it into any script.
export GEMINI_API_KEY="your-api-key-here"On Windows, use setx GEMINI_API_KEY "your-api-key-here" in a PowerShell session, then restart your terminal so the variable loads.
Step 2: Install the Python SDK and Dependencies
Create a virtual environment first, then install every package this project needs in one pass.
python3 -m venv venv
source venv/bin/activate
pip install google-genai chromadb fastapi uvicorn pypdf python-multipart pillowOn Windows, replace the activation line with venv\Scripts\activate. Run pip show google-genai afterward to confirm the SDK installed correctly — the import path for this package is google.genai, not google.generativeai, which is the deprecated legacy SDK still floating around in older tutorials.
Step 3: Set Up Your Project Structure
Create the following folder layout. Keeping embeddings, source documents, and application code separate makes the ingestion pipeline easier to re-run when you add new files.
multimodal-rag/
├── data/
│ ├── docs/ # PDFs go here
│ └── images/ # PNG/JPEG files go here
├── chroma_store/ # Chroma persists its index here
├── embed.py # Embedding helper functions
├── ingest.py # Builds the vector index
├── query.py # Cross-modal retrieval + generation
└── app.py # FastAPI deploymentStep 4-6: Generate Text, Image, and Document Embeddings
Step 4: Generate Your First Text Embedding
Create embed.py and start with the simplest possible call: embedding a single string.
import os
from google import genai
client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])
def embed_text(text: str, output_dim: int = 768):
result = client.models.embed_content(
model="gemini-embedding-2",
contents=text,
config={"output_dimensionality": output_dim},
)
return result.embeddings[0].values
if __name__ == "__main__":
vec = embed_text("What is the meaning of life?")
print(f"Vector length: {len(vec)}")
print(vec[:5])Running this produces output similar to:
Vector length: 768
[0.0142, -0.0387, 0.0561, -0.0023, 0.0198]Google’s documentation recommends embedding task instructions directly inside the text rather than passing a separate task-type parameter for text-only calls. For search-style retrieval, prefix queries and documents differently, for example task: search result | query: {your text} for queries and a plain content string for the documents you are indexing. This asymmetry improves retrieval quality because the model encodes queries and documents slightly differently even though they land in the same vector space.
Step 5: Embed Images and PDFs
Multimodal embedding is where Gemini Embedding 2 earns its name. Add an image-embedding function to embed.py:
from google.genai import types
def embed_image(image_path: str, output_dim: int = 768):
with open(image_path, "rb") as f:
image_bytes = f.read()
result = client.models.embed_content(
model="gemini-embedding-2",
contents=types.Part.from_bytes(
data=image_bytes, mime_type="image/png"
),
config={"output_dimensionality": output_dim},
)
return result.embeddings[0].values
def embed_pdf(pdf_path: str, output_dim: int = 768):
with open(pdf_path, "rb") as f:
pdf_bytes = f.read()
result = client.models.embed_content(
model="gemini-embedding-2",
contents=types.Part.from_bytes(
data=pdf_bytes, mime_type="application/pdf"
),
config={"output_dimensionality": output_dim},
)
return result.embeddings[0].valuesRemember the per-request caps covered earlier: a PDF call tops out at six pages and an image call accepts up to six images per request if you pass a list of Part objects instead of one. If a PDF has more than six pages, split it before embedding — Step 8 below handles this automatically with pypdf.
Step 6: Choose Output Dimensions and Truncation
The 3,072-dimension default vector is accurate but expensive to store at scale. A million documents at full precision costs roughly four times the storage of the same set truncated to 768 dimensions, with only a modest accuracy tradeoff for most retrieval tasks. Pass output_dimensionality in the config to request a smaller vector directly, which is cheaper than requesting the full vector and truncating client-side because Gemini Embedding 2 automatically re-normalizes truncated embeddings for you.
# Request the vector pre-truncated and normalized server-side
result = client.models.embed_content(
model="gemini-embedding-2",
contents="Quarterly earnings call transcript excerpt...",
config={"output_dimensionality": 1536},
)
vector = result.embeddings[0].values
assert len(vector) == 1536For this tutorial’s project, use 768 dimensions. It keeps the Chroma index small enough to run comfortably on a laptop while retaining strong retrieval accuracy for the demo corpus.
Step 7-9: Pick a Vector Database, Build the Ingestion Pipeline, and Add Metadata
Step 7: Initialize a Chroma Vector Store
Chroma is a good fit for a tutorial-scale project because it needs no separate server process and persists to a local folder. Create the collection with a fixed dimension that matches what you chose in Step 6.
import chromadb
chroma_client = chromadb.PersistentClient(path="./chroma_store")
collection = chroma_client.get_or_create_collection(
name="multimodal_docs",
metadata={"hnsw:space": "cosine"},
)Microsoft’s own RAG architecture guidance makes the same point that applies here: embed document chunks and user queries with the same model and configuration so both land in a directly comparable vector space. Mixing a 768-dimension query against a 3,072-dimension index, or embedding your corpus with one model and your queries with another, silently produces garbage similarity scores instead of an obvious error.
Step 8: Build the Ingestion Pipeline
Now write ingest.py, which walks the data/ folder, chunks PDFs to respect the six-page limit, embeds every file, and writes everything into Chroma with metadata tagging the source type.
import os
import uuid
from pypdf import PdfReader, PdfWriter
from embed import embed_text, embed_image, embed_pdf
from chroma_setup import collection
def chunk_pdf(path: str, pages_per_chunk: int = 6):
reader = PdfReader(path)
chunks = []
for start in range(0, len(reader.pages), pages_per_chunk):
writer = PdfWriter()
for p in reader.pages[start:start + pages_per_chunk]:
writer.add_page(p)
chunk_path = f"/tmp/{uuid.uuid4()}.pdf"
with open(chunk_path, "wb") as f:
writer.write(f)
chunks.append(chunk_path)
return chunks
def ingest_docs(folder: str = "data/docs"):
for fname in os.listdir(folder):
if not fname.endswith(".pdf"):
continue
full_path = os.path.join(folder, fname)
for chunk_path in chunk_pdf(full_path):
vec = embed_pdf(chunk_path)
collection.add(
ids=[str(uuid.uuid4())],
embeddings=[vec],
metadatas=[{"source": fname, "type": "pdf"}],
)
os.remove(chunk_path)
def ingest_images(folder: str = "data/images"):
for fname in os.listdir(folder):
if not fname.lower().endswith((".png", ".jpg", ".jpeg")):
continue
full_path = os.path.join(folder, fname)
vec = embed_image(full_path)
collection.add(
ids=[str(uuid.uuid4())],
embeddings=[vec],
metadatas=[{"source": fname, "type": "image"}],
)
if __name__ == "__main__":
ingest_docs()
ingest_images()
print(f"Indexed {collection.count()} items")Running python ingest.py against a folder of ten PDFs and fifteen images produces output like:
Indexed 47 itemsThat number is higher than 25 source files because multi-page PDFs get split into multiple six-page chunks, each stored as a separate vector with a shared source metadata field pointing back to the original filename.
Step 9: Add Metadata Filtering
Pure vector similarity sometimes surfaces an image when you specifically want a PDF excerpt, or vice versa. Chroma supports a where clause alongside the vector query so you can restrict results by the metadata you stored in Step 8.
results = collection.query(
query_embeddings=[query_vector],
n_results=5,
where={"type": "pdf"},
)This is also the simplest way to build hybrid search: run the same query against the collection twice, once filtered to text documents and once to images, then merge and re-rank before passing results to the generation step.
Step 10-13: Cross-Modal Retrieval, Grounded Generation, and Deployment
Step 10: Build the Cross-Modal Query Function
Write query.py. A user types a text question; you embed that question with the same model and dimension as the index, then search Chroma for the nearest vectors regardless of whether they came from a PDF chunk or an image.
from embed import embed_text
from chroma_setup import collection
def retrieve(question: str, k: int = 5):
query_vec = embed_text(
f"task: search result | query: {question}"
)
results = collection.query(
query_embeddings=[query_vec],
n_results=k,
)
return resultsBecause every modality shares one embedding space, a text query like “show me the chart with the biggest quarter-over-quarter jump” can retrieve an image chunk directly, without an intermediate captioning or OCR step translating the image into text first.
Step 11: Generate Grounded Answers
Retrieval alone isn’t an answer. Pass the retrieved metadata back to a Gemini chat model along with the original question so it can synthesize a grounded response.
def generate_answer(question: str, retrieved_sources: list):
context_note = "\n".join(
f"- {m['source']} ({m['type']})" for m in retrieved_sources
)
prompt = (
f"Answer the question using only the following sources as context.\n"
f"Sources:\n{context_note}\n\nQuestion: {question}"
)
response = client.models.generate_content(
model="gemini-2.5-flash",
contents=prompt,
)
return response.textIn production, you would typically re-fetch the actual file bytes for the top matches and pass the images or PDF pages themselves into the generation call, not just the filenames, so the model can reference specific visual content rather than guessing from a source list.
Step 12: Benchmark Latency and Cost
Before shipping, time a representative batch of embedding calls and log the token or unit counts returned by the API so you can project cost at your expected volume.
import time
start = time.perf_counter()
vec = embed_text("Sample query for latency testing")
elapsed = time.perf_counter() - start
print(f"Embedding latency: {elapsed * 1000:.0f}ms")Typical text-embedding latency for a short query sits in the low hundreds of milliseconds over a standard broadband connection. Image and PDF embedding calls run slower because the payload is larger and the model has to process the visual encoder path, so budget accordingly if your application embeds documents synchronously on the request path rather than in a background ingestion job.
Step 13: Deploy as a FastAPI Endpoint
Wrap retrieval and generation behind a single endpoint so any frontend can call it.
from fastapi import FastAPI
from pydantic import BaseModel
from query import retrieve, generate_answer
app = FastAPI()
class Question(BaseModel):
text: str
@app.post("/ask")
def ask(q: Question):
results = retrieve(q.text)
sources = results["metadatas"][0]
answer = generate_answer(q.text, sources)
return {"answer": answer, "sources": sources}Start it with uvicorn app:app --reload and test with a POST request to http://localhost:8000/ask. A healthy response looks like this:
{
"answer": "The chart showing the largest quarter-over-quarter revenue jump is in slide_deck_q3.png, which shows a 34% increase...",
"sources": [
{"source": "slide_deck_q3.png", "type": "image"},
{"source": "earnings_call.pdf", "type": "pdf"}
]
}The Complete Working Project
Here is every file from the tutorial assembled into its final form, ready to clone into a fresh folder. Save each block under the filename shown in its first line.
# chroma_setup.py
import chromadb
chroma_client = chromadb.PersistentClient(path="./chroma_store")
collection = chroma_client.get_or_create_collection(
name="multimodal_docs",
metadata={"hnsw:space": "cosine"},
)
# embed.py
import os
from google import genai
from google.genai import types
client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])
def embed_text(text: str, output_dim: int = 768):
result = client.models.embed_content(
model="gemini-embedding-2",
contents=text,
config={"output_dimensionality": output_dim},
)
return result.embeddings[0].values
def embed_image(image_path: str, output_dim: int = 768):
with open(image_path, "rb") as f:
image_bytes = f.read()
result = client.models.embed_content(
model="gemini-embedding-2",
contents=types.Part.from_bytes(data=image_bytes, mime_type="image/png"),
config={"output_dimensionality": output_dim},
)
return result.embeddings[0].values
def embed_pdf(pdf_path: str, output_dim: int = 768):
with open(pdf_path, "rb") as f:
pdf_bytes = f.read()
result = client.models.embed_content(
model="gemini-embedding-2",
contents=types.Part.from_bytes(data=pdf_bytes, mime_type="application/pdf"),
config={"output_dimensionality": output_dim},
)
return result.embeddings[0].valuesRun python ingest.py once to build the index, then uvicorn app:app --reload to serve it. Re-run ingest.py whenever you drop new files into data/docs or data/images — Chroma will append new vectors to the existing persistent store without duplicating what’s already indexed, as long as you give each item a fresh UUID.
Gemini Embedding 2 vs pplx-embed-v2-late vs EmbeddingGemma 2
Three multimodal embedding releases landed within roughly 48 hours of each other in October 2026, which the AI newsletter ThursdAI flagged as a notable cluster of simultaneous multimodal embedding launches. Here’s how the three compare on the specs that matter for a RAG build.
| Model | Vendor | Access | Max Dimensions | Modalities | Released |
|---|---|---|---|---|---|
| Gemini Embedding 2 | Hosted API | 3,072 (128-3,072 flexible) | Text, image, audio, video, PDF | Latest update April 2026 | |
| EmbeddingGemma 2 | Open weights (Apache 2.0) | 768 native (MRL to 128/256/512) | Text, image, video, audio | October 8, 2026 | |
| pplx-embed-v2-late | Perplexity | Hosted API | Not published | Text, image, page (late-interaction) | October 7, 2026 |
EmbeddingGemma 2 is the one to reach for if you need to run embedding inference on your own infrastructure or fine-tune on domain-specific data — it ships as a 740-million-parameter open-weight model you can load with Hugging Face’s transformers or sentence-transformers libraries, with selective encoder loading so a text-only deployment only needs the 270M-parameter backbone instead of the full multimodal stack. Gemini Embedding 2, the model this tutorial builds around, is the better fit when you want a managed endpoint with no inference infrastructure to run and the largest available output dimension for maximum retrieval precision. Perplexity’s pplx-embed-v2-late announcement describes a late-interaction architecture rather than a single dense vector per item, which trades simplicity for potentially sharper token-level matching — worth testing if your corpus leans heavily on dense technical documents where exact phrase matching matters more than general semantic similarity.
Pricing and Cost Breakdown
Gemini Embedding 2 pricing is split by modality, which means your bill depends heavily on what you’re indexing, not just how many documents you have. Per Google’s published Gemini API pricing, standard paid-tier rates per 1 million units are:
| Input Type | Free Tier | Paid Tier (per 1M tokens) | Batch Tier (50% off) |
|---|---|---|---|
| Text | Free | $0.20 | $0.10 |
| Image | Free | $0.45 (~$0.00012/image) | $0.225 |
| Audio | Free | $6.50 (~$0.00016/second) | $3.25 |
| Video | Free | $12.00 (~$0.00079/frame) | $6.00 |
The free tier has no hard usage cap, but Google uses free-tier traffic to improve its products, while paid-tier calls carry standard privacy protections and are not used for model training. For a tutorial-scale project indexing a few hundred text chunks and images, you will not come close to the free tier’s limits. At production scale, batch the ingestion job wherever possible — reprocessing an entire document library at launch, or re-indexing after a model upgrade, is exactly the kind of workload the 50% batch discount is built for, since none of those calls need the lower latency of the synchronous API.
Walking through a realistic worked example makes the modality split concrete. A knowledge base of 10,000 text chunks averaging 400 tokens each costs roughly $0.80 to embed on the paid tier at full price, or $0.40 using the batch discount — small enough that text embedding cost is rarely the deciding factor in an architecture choice. Add 2,000 product images to that same knowledge base at roughly $0.00012 per image and the image portion adds about $0.24, still negligible. The number that actually requires planning is video: indexing 500 one-minute product demo clips at $0.00079 per frame, assuming roughly one frame per second sampled by default, adds up to around $23.70 for that single ingestion pass. None of these figures are large in absolute terms for a mid-size knowledge base, but the relative gap between modalities — video costing roughly 60 times more than text per indexed unit — is exactly why the pricing table above breaks costs out by input type instead of giving one blended per-document rate.
Evaluating Retrieval Quality and Securing Your API Key
Measure Retrieval Quality Before You Ship
A RAG pipeline that returns an answer is not the same as a RAG pipeline that returns the right answer. Before pointing real users at the FastAPI endpoint from Step 13, build a small evaluation set: a list of questions paired with the filename you already know contains the correct answer. Run every question through retrieve() and check whether the expected source shows up in the top-k results.
from query import retrieve
eval_set = [
{"question": "What was Q3 revenue growth?", "expected_source": "earnings_call.pdf"},
{"question": "Show me the quarterly revenue chart", "expected_source": "slide_deck_q3.png"},
]
def evaluate(eval_set, k=5):
hits = 0
for item in eval_set:
results = retrieve(item["question"], k=k)
sources = [m["source"] for m in results["metadatas"][0]]
if item["expected_source"] in sources:
hits += 1
else:
print(f"MISS: '{item['question']}' -> got {sources}")
precision_at_k = hits / len(eval_set)
print(f"Recall@{k}: {precision_at_k:.2%}")
if __name__ == "__main__":
evaluate(eval_set)A healthy run against a well-tagged evaluation set prints output like Recall@5: 92.00% along with a short list of misses you can inspect manually. Treat anything below roughly 80% recall at k=5 as a signal to revisit your chunking strategy or task-prefix usage before expanding the corpus further, since retrieval problems compound rather than average out as a knowledge base grows. Build this evaluation set incrementally: every time a user reports a wrong or missing answer in production, add that exact question and the correct source to the eval set so your regression coverage grows alongside real usage patterns instead of staying frozen at launch-day assumptions.
Secure and Rate-Limit Your API Key
Everything in this tutorial reads the API key from an environment variable, which is fine for local development but not sufficient for a deployed service. In production, load the key from a secrets manager at process startup rather than baking it into a container image or committing it to a .env file that ends up in version control. Rotate the key periodically from the Google AI Studio dashboard, and if you ever suspect a key has leaked — pasted into a public gist, logged by a crash reporter, committed by accident — revoke it immediately rather than waiting to see if it gets abused.
Rate limiting matters just as much as key secrecy once your FastAPI endpoint is public. A single retry loop that doesn’t back off can turn a transient API error into a thundering-herd problem against your own quota. Wrap outbound calls in a simple exponential backoff so a momentary 429 (rate limit) or 503 (overloaded) response from Google’s API doesn’t cascade into a retry storm.
import time
import random
def with_backoff(fn, max_retries=4, base_delay=1.0):
for attempt in range(max_retries):
try:
return fn()
except Exception as e:
if attempt == max_retries - 1:
raise
delay = base_delay * (2 ** attempt) + random.uniform(0, 0.5)
print(f"Retrying after {delay:.1f}s due to: {e}")
time.sleep(delay)
# Usage:
vec = with_backoff(lambda: embed_text("resilient embedding call"))On the ingestion side specifically, add a short fixed delay between calls when batch-processing a large folder of documents rather than firing every request as fast as the event loop allows — it is far cheaper to let an overnight ingestion job run a few minutes longer than to trip a rate limit and have to restart a partially completed run.
Common Pitfalls When Building Multimodal RAG
- Mixing embedding dimensions across a collection. If half your vectors were embedded at 768 dimensions and half at 3,072, Chroma will throw a dimension-mismatch error or silently produce wrong distances depending on the client version. Fix the dimension once in a config constant and never change it without re-embedding the full collection.
- Re-embedding against the old gemini-embedding-001 space. The vector spaces between generations are not compatible. Any index built before this tutorial’s model needs a full rebuild, not an incremental migration.
- Exceeding the six-page PDF limit silently. The API will either reject the call or embed only part of the document depending on the SDK version. Always chunk PDFs to six pages or fewer before calling
embed_pdf, as shown in Step 8. - Using float16 precision locally with EmbeddingGemma 2. If you experiment with the open-weight sibling model instead of the hosted API, float16 is known to produce NaN or degraded embeddings on this architecture. Use bfloat16 or float32.
- Embedding queries and documents with different task prefixes inconsistently. Skipping the
task: search result | query:prefix on queries while using it on indexed documents (or vice versa) measurably hurts retrieval ranking, because the asymmetric prefixes are part of how the model was trained to represent queries versus documents. - Treating metadata filtering as optional. Without the
typefield from Step 8, you cannot cleanly separate “find me a document” from “find me an image” requests, and users will get confusing cross-modal results for queries that should stay in one modality.
Troubleshooting Guide
- “API key not valid” error on first call. Confirm the environment variable name matches exactly what your script reads (
GEMINI_API_KEYis a convention, not a hardcoded requirement) and that you restarted your terminal session after setting it. - Embedding dimension mismatch when querying Chroma. Print
len(vector)right before everycollection.addorcollection.querycall during debugging — a stray default value (3,072 instead of your intended 768) is the most common cause. - PDF embedding returns an error about page count. Re-check your chunking logic in Step 8; the six-page cap applies per request, not per document.
- Image embedding fails with an unsupported MIME type. Gemini Embedding 2 accepts PNG and JPEG for images. Convert other formats (WebP, HEIC) with Pillow before embedding.
- Retrieval returns irrelevant results despite a correct pipeline. Verify you used the search-query task prefix on the question but not on the indexed documents, and confirm the
hnsw:spacemetric in Chroma is set tocosine, which matches how Gemini Embedding 2 vectors are normalized. - FastAPI endpoint times out under load. Embedding and generation calls are both network round trips to Google’s API; wrap them in async functions and use
async defroute handlers with an async-compatible client rather than blocking the event loop. - Chroma persistent store grows unexpectedly large. Confirm you are not re-ingesting the same files on every run. Track ingested filenames in a small local manifest file and skip files already present before calling
ingest_docsagain. - Generation step hallucinates answers not grounded in retrieved sources. Pass the actual retrieved content (image bytes or PDF text) into the generation prompt rather than just filenames, and explicitly instruct the model to say it doesn’t know rather than guess when the retrieved context doesn’t answer the question.
Advanced Tips for Production RAG
Once the tutorial project works end to end, a few refinements separate a demo from a production system. First, cache embeddings aggressively: if a document hasn’t changed, don’t re-embed it on every ingestion run. A content hash keyed against the manifest file mentioned in the troubleshooting section above is enough to skip redundant API calls and cost.
Second, consider a two-stage retrieval pattern for large corpora: use a cheaper, lower-dimension embedding pass (say, 256 dimensions) to narrow a million-item index down to the top 100 candidates, then re-rank those 100 using the full 3,072-dimension vectors or a dedicated re-ranking model. This keeps the expensive high-precision comparison limited to a small candidate set instead of running it against your entire index on every query.
Third, when you scale past a single-machine Chroma deployment, migrate to a managed vector database that supports the same cosine-similarity metric and metadata filtering patterns used throughout this tutorial, so the application code in query.py needs minimal changes. The retrieval logic — embed, search, filter by metadata, re-rank — stays the same regardless of which vector store backs it.
Finally, instrument every embedding and generation call with the token or unit counts the API returns. Given the per-modality pricing breakdown above, a single video-heavy ingestion job can cost dramatically more than a text-heavy one of the same file count, and you want that visible in your cost dashboards before a surprise bill rather than after. The dev.to community has published a useful walkthrough on building RAG from scratch without heavyweight frameworks, which is worth comparing against the Chroma-based approach here if your production constraints rule out third-party vector database dependencies entirely.
Frequently Asked Questions
What is Gemini Embedding 2 used for?
Gemini Embedding 2 converts text, images, audio, video, and PDF documents into numerical vectors that can be compared for semantic similarity, which is the core mechanism behind search, recommendation systems, and retrieval-augmented generation (RAG) pipelines like the one built in this tutorial.
Is Gemini Embedding 2 free to use?
Google offers a free tier for Gemini Embedding 2 with no published hard cap, though free-tier usage may be used to improve Google’s products. Paid-tier pricing starts at $0.20 per 1 million tokens for text and scales up for image, audio, and video inputs, as detailed in the pricing table above.
How many dimensions should I use for Gemini Embedding 2 vectors?
Google recommends 768, 1,536, or 3,072 dimensions. For most RAG applications, 768 dimensions offers a strong balance between retrieval accuracy and storage cost, which is why this tutorial’s project uses that setting throughout.
Can I use Gemini Embedding 2 vectors with my existing gemini-embedding-001 index?
No. The two models produce incompatible embedding spaces. Any existing index built on the previous generation needs to be fully re-embedded with Gemini Embedding 2 before you can query it alongside new data.
What’s the difference between Gemini Embedding 2 and EmbeddingGemma 2?
Gemini Embedding 2 is a hosted API model with up to 3,072 output dimensions and the broadest per-request input limits. EmbeddingGemma 2 is a separate, open-weight model released under Apache 2.0 that you run yourself, with a native 768-dimension output and a smaller 740-million-parameter footprint suited to self-hosted or fine-tuned deployments.
Do I need a GPU to follow this tutorial?
No. Every embedding and generation call in this tutorial runs through Google’s hosted Gemini API, so all computation happens on Google’s infrastructure. A GPU is only relevant if you choose to self-host the open-weight EmbeddingGemma 2 model instead.
Why does my PDF embedding call fail with large documents?
Gemini Embedding 2 accepts a maximum of one PDF per request, up to six pages. Documents longer than six pages need to be split into chunks before embedding, which the chunk_pdf function in Step 8 of this tutorial handles automatically.
Which vector database should I use instead of Chroma for production?
Chroma is well suited to tutorials and small-to-medium deployments. For larger production workloads, any vector database that supports cosine similarity and metadata filtering — the two features this tutorial’s retrieval logic relies on — can be swapped in with minimal changes to the application code in query.py.
Related Coverage
- DeepSeek V4.1 Flash API Setup: 13 Steps, 1M Tokens [2026]
- GPT Image 2.5 API Setup: 13 Steps, $30/M Tokens [2026]
- Strands Agents SDK Setup: 13 Steps, 115ms Decider AI [2026]
- Microsoft Foundry MAI-Image-2.6 Setup: 13 Steps [2026]
- OpenAI Responses API Setup: 13 Steps, Ultrafast Mode [2026]
