Production RAG on Google Cloud

Concept

Retrieval-Augmented Generation (RAG) grounds an LLM in your own data. At query time you retrieve relevant chunks from a vector (and/or keyword) store and pass them as context so the model answers from facts you control, with citations. On GCP you assemble it from Cloud Storage, Document AI, an embedding model, a vector store (Vertex AI Vector Search, AlloyDB AI, or others), and a Gemini model — served from Cloud Run.

Problem it solves. LLMs hallucinate and don't know your private or fresh data. RAG injects authoritative context per query, giving grounded, cite-able answers without retraining — and lets you update knowledge by re-indexing documents instead of fine-tuning.

Limitations

• Quality is bounded by retrieval — bad chunks → bad answers
• Ingestion is a data pipeline with its own failure/cost/latency concerns
• Vector store endpoints bill continuously; embeddings cost per token
• Requires evaluation to catch regressions (retrieval and faithfulness)

Interactive architecture

flowchart TD
  subgraph Ingest[Ingestion loop - async]
    UP[Upload to Cloud Storage] --> EVT[Eventarc / Pub/Sub]
    EVT --> JOB[Cloud Run Job]
    JOB --> DOCAI[Document AI - parse / OCR]
    DOCAI --> CHUNK[Chunk + enrich metadata]
    CHUNK --> EMB[Embeddings - Vertex AI]
    EMB --> IDX[(Vector store: Vector Search / AlloyDB AI)]
    JOB --> STATE[(Ingestion state table)]
  end
  subgraph Query[Query loop - sync]
    Q[User question] --> API[FastAPI on Cloud Run]
    API --> QEMB[Embed query]
    QEMB --> RET[Hybrid retrieve + metadata filter]
    RET --> IDX
    RET --> RR[Rerank]
    RR --> CTX[Build context]
    CTX --> GEN[Gemini - generate + cite]
    GEN --> API
  end
  classDef gcp fill:#e8f0fe,stroke:#4285f4,color:#174ea6;
  class UP,EVT,JOB,DOCAI,CHUNK,EMB,IDX,STATE,API,QEMB,RET,RR,CTX,GEN gcp;

flowchart TD
  subgraph Ingest[Ingestion loop - async]
    UP[Upload to Cloud Storage] --> EVT[Eventarc / Pub/Sub]
    EVT --> JOB[Cloud Run Job]
    JOB --> DOCAI[Document AI - parse / OCR]
    DOCAI --> CHUNK[Chunk + enrich metadata]
    CHUNK --> EMB[Embeddings - Vertex AI]
    EMB --> IDX[(Vector store: Vector Search / AlloyDB AI)]
    JOB --> STATE[(Ingestion state table)]
  end
  subgraph Query[Query loop - sync]
    Q[User question] --> API[FastAPI on Cloud Run]
    API --> QEMB[Embed query]
    QEMB --> RET[Hybrid retrieve + metadata filter]
    RET --> IDX
    RET --> RR[Rerank]
    RR --> CTX[Build context]
    CTX --> GEN[Gemini - generate + cite]
    GEN --> API
  end
  classDef gcp fill:#e8f0fe,stroke:#4285f4,color:#174ea6;
  class UP,EVT,JOB,DOCAI,CHUNK,EMB,IDX,STATE,API,QEMB,RET,RR,CTX,GEN gcp;

How it works

RAG has two loops: an ingestion loop (documents → parse → chunk → embed → index, usually asynchronous) and a query loop (question → retrieve → rerank → build context → generate → cite). Production quality lives in the retrieval and evaluation details, not the generation call.

1. 1 · Ingest & validateCloud Storage + Cloud Run Job

   Files land in a bucket; an event triggers a Job that validates MIME type/size and (conceptually) scans before processing.

   ⚠ Reject invalid/oversized files to a quarantine prefix 📈 Ingestion state table, DLQ depth 💰 Storage + job compute
2. 2 · Parse & extractDocument AI / parsers

   Extract text (and layout) from PDFs/images via OCR; keep page/section metadata for citations.

   ⚠ OCR failures → retry / dead-letter 📈 Parse success rate
3. 3 · Chunk & enrichChunker

   Split into semantically coherent chunks with overlap; attach metadata (source, page, tenant, ACL) for filtering.

   ⚠ Bad chunking silently degrades retrieval 📈 Chunk size distribution
4. 4 · Embed & indexEmbeddings + vector store

   Embed chunks and upsert into the vector store with metadata. Store the chunk text alongside for context assembly.

   ⚠ Partial index → inconsistent answers; use idempotent upserts 📈 Index size, upsert errors 💰 Embedding tokens; continuous index endpoint cost
5. 5 · RetrieveRetriever

   Embed the query, do dense (+ optional sparse/keyword) search with metadata filters (e.g. tenant ACL), fetch top-k.

   ⚠ Low recall → missing context; wrong filter → data leakage 📈 Recall@k, latency
6. 6 · Rerank & build contextReranker

   Reorder candidates by relevance, trim to the token budget, and construct a grounded prompt with source markers.

   📈 Context relevance score
7. 7 · Generate & citeGemini via Vertex AI

   Generate the answer constrained to the context, returning citations to the source chunks.

   ⚠ Hallucination if grounding is weak; handle safety blocks 📈 Faithfulness, citation accuracy, token cost

Hands-on example

Cloud Console

1. Create a Cloud Storage bucket for source documents (uniform bucket-level access, public access prevention)
2. Enable Vertex AI, Document AI, and (if used) Vector Search APIs
3. Prototype prompts in Vertex AI Studio; decide on chunk size and top-k
4. Choose a vector store: Vertex AI Vector Search, AlloyDB AI (pgvector), or Vertex AI RAG Engine (managed)

Code

PROJECT=my-project; REGION=us-central1
gcloud config set project "$PROJECT"
gcloud services enable aiplatform.googleapis.com documentai.googleapis.com \
  run.googleapis.com eventarc.googleapis.com pubsub.googleapis.com

# Secure document bucket
gcloud storage buckets create gs://$PROJECT-rag-docs \
  --location=$REGION --uniform-bucket-level-access --public-access-prevention

# Deploy the ingestion worker as a Cloud Run Job
gcloud run jobs deploy rag-ingest \
  --image "$REGION-docker.pkg.dev/$PROJECT/apps/rag-ingest:latest" \
  --region "$REGION" \
  --service-account rag-ingest-sa@$PROJECT.iam.gserviceaccount.com \
  --set-env-vars EMBED_MODEL=text-embedding-005

# Trigger the Job when objects are finalized in the bucket (Eventarc)
gcloud eventarc triggers create rag-ingest-trigger \
  --location=$REGION \
  --destination-run-job=rag-ingest \
  --event-filters="type=google.cloud.storage.object.v1.finalized" \
  --event-filters="bucket=$PROJECT-rag-docs" \
  --service-account=eventarc-sa@$PROJECT.iam.gserviceaccount.com

# pip install google-genai
import os
from google import genai
from google.genai import types

client = genai.Client()  # Vertex AI via ADC
GEN_MODEL = os.getenv("GENAI_MODEL", "gemini-2.0-flash")
EMBED_MODEL = os.getenv("EMBED_MODEL", "text-embedding-005")

def embed_query(q: str) -> list[float]:
    r = client.models.embed_content(model=EMBED_MODEL, contents=[q])
    return r.embeddings[0].values

def retrieve(vector: list[float], tenant: str, k: int = 8) -> list[dict]:
    """Query your vector store (AlloyDB AI / Vector Search).
    ALWAYS filter by tenant to prevent cross-tenant leakage."""
    # Pseudocode for AlloyDB AI (pgvector):
    #   SELECT chunk_id, source, page, text
    #   FROM chunks
    #   WHERE tenant = %s
    #   ORDER BY embedding <=> %s LIMIT %s
    ...
    return []  # [{chunk_id, source, page, text}, ...]

def answer(question: str, tenant: str) -> dict:
    chunks = retrieve(embed_query(question), tenant)
    # Build a grounded prompt with numbered sources for citations
    context = "\n\n".join(f"[{i+1}] ({c['source']} p.{c['page']}) {c['text']}"
                           for i, c in enumerate(chunks))
    system = ("Answer ONLY from the sources. Cite sources as [n]. "
              "If the answer isn't in the sources, say you don't know.")
    resp = client.models.generate_content(
        model=GEN_MODEL,
        contents=f"{system}\n\nSources:\n{context}\n\nQuestion: {question}",
        config=types.GenerateContentConfig(temperature=0.1, max_output_tokens=1024),
    )
    return {
        "answer": resp.text,
        "citations": [{"n": i + 1, "source": c["source"], "page": c["page"]}
                      for i, c in enumerate(chunks)],
        "tokens": resp.usage_metadata.total_token_count,
    }

# Retrieval hits your vector store's API; generation uses the Vertex AI
# generateContent endpoint (see the Vertex AI topic for the exact URL pattern).

resource "google_storage_bucket" "docs" {
  name                        = "${var.project_id}-rag-docs"
  location                    = var.region
  uniform_bucket_level_access = true
  public_access_prevention    = "enforced"
  versioning { enabled = true }
}

# AlloyDB as the vector store (enable the vector extension in SQL after creation)
resource "google_alloydb_cluster" "rag" {
  cluster_id = "rag-cluster"
  location   = var.region
  network_config { network = google_compute_network.vpc.id }
}

resource "google_alloydb_instance" "primary" {
  cluster       = google_alloydb_cluster.rag.name
  instance_id   = "rag-primary"
  instance_type = "PRIMARY"
  machine_config { cpu_count = 2 }
}

Troubleshooting

• Answers cite nothing / say 'I don't know' for known facts → retrieval recall problem (chunking, k, embedding model, filters)
• Hallucinated facts → weaken by tightening the system prompt and lowering temperature; verify context actually reached the model
• Cross-tenant leakage → missing/incorrect metadata filter on retrieval
• Slow queries → cache query embeddings, reduce k, add a reranker only where it pays off
• Ingestion stuck → check the DLQ, Job logs, and idempotent upsert logic

Production considerations

What interviewers expect you to know

Common mistakes

• Skipping evaluation — shipping RAG with no recall/faithfulness metrics
• No per-tenant metadata filter → cross-tenant data leakage
• Poor chunking (too big/small, no overlap, no metadata) tanking retrieval
• Re-embedding everything on every change instead of deltas (cost blowup)
• Synchronous ingestion in the request path (should be async Jobs/Batch)
• Leaving Vector Search index endpoints deployed and forgotten (continuous cost)
• Not returning citations, so answers can't be trusted or audited
• Ignoring safety/blocked responses and PII in logs

Comparisons

Knowledge check