Embeddings · 58 topics
What are embeddings, and how are they used in retrieval?
An embedding is a dense vector (for example 384, 768 or 1,536 numbers) produced by a neural network so that texts with similar meaning land close together in vector space.
In retrieval, documents (chunks) are embedded once and stored; at query time the query is embedded with the same model, and the nearest vectors by a similarity metric are returned.
This matches meaning rather than exact words, so "how do I return an item" can find a passage about refunds.
Embeddings are also used for clustering, deduplication, classification, recommendations and anomaly detection.
How are text embedding models trained?
Most are transformer encoders trained with contrastive learning: pairs of related texts (query and relevant passage, question and answer, paraphrases) are pulled together while unrelated texts are pushed apart.
A typical loss is InfoNCE: for each query, the matching passage is the positive and the other passages in the batch (in-batch negatives) are negatives, so larger batches give a harder task.
Hard negatives (passages that look relevant but are not) improve quality a lot.
Training usually starts with large weakly supervised pairs from the web, followed by fine-tuning on curated labeled data.
What is the difference between a bi-encoder and a cross-encoder?
A bi-encoder embeds the query and each document independently, so document vectors can be precomputed and searched with a vector index: fast and scalable, but less precise because query and document never interact directly.
A cross-encoder feeds the query and document together into one transformer and outputs a relevance score: much more accurate, but it must run per query-document pair, so it is too slow for searching a whole corpus.
The standard pipeline uses a bi-encoder (plus keyword search) to retrieve the top 20 to 100 candidates and a cross-encoder to rerank them.
How do dense and sparse embeddings differ?
Dense embeddings are short vectors where every dimension carries some meaning; they capture semantics and paraphrases but are opaque and can blur exact terms like IDs and rare names.
Sparse representations (BM25, TF-IDF, learned sparse models such as SPLADE) have one dimension per vocabulary term with mostly zeros; they match exact words, are interpretable, and work well for rare keywords.
Dense helps with meaning and sparse with precision on terms, which is why hybrid search combining both usually beats either alone.
Cosine similarity, dot product or Euclidean distance: which should you use for embeddings?
Use the metric the embedding model was trained with (check its documentation); most text models use cosine similarity.
Cosine measures only the angle (direction); dot product also rewards larger magnitude; Euclidean measures absolute distance.
If vectors are normalized to unit length, cosine similarity and dot product give identical rankings, and Euclidean distance becomes a monotonic function of them. Normalizing lets an inner-product index emulate cosine.
Use dot product when magnitude is meaningful (for example some recommendation models).
How does embedding dimensionality affect retrieval, and what is Matryoshka representation learning?
Higher dimensions can hold more information and often give slightly better quality, but cost more storage, memory and search time (a 1,536-dimension FP32 vector takes about 6 KB).
Matryoshka embeddings are trained so that the first k dimensions form a good embedding on their own; you can truncate vectors to 256 or 512 dimensions with a small quality loss, and rerank with the full vectors.
Choose dimensions by measuring recall on your own data against storage and latency budgets.
How do you scale embeddings: dimensionality reduction versus quantization?
Dimensionality reduction (PCA, Matryoshka truncation) shrinks the number of values per vector but can discard information and may need retraining or fitting.
Quantization keeps all dimensions but stores each with fewer bits: scalar quantization (FP32 to INT8) cuts memory about 4 times with a small accuracy loss; binary quantization (1 bit per dimension) cuts it about 32 times and speeds up search with Hamming distance, but loses more accuracy, so it is usually paired with rescoring using full-precision vectors.
Quantization generally preserves more quality per byte saved, and the two techniques can be combined.
How do you choose an embedding model?
Define requirements: language coverage, domain (legal, code, medical), maximum input length, dimension, latency and cost, and whether it must run on your own infrastructure (license and hosting).
Shortlist from public leaderboards such as MTEB, but decide by evaluating candidates on your own labeled queries and documents (recall@k, MRR), since leaderboard tasks may not match your data.
Consider the operational cost: switching models later means re-embedding everything, so test with realistic scale and keep the choice versioned.
What are asymmetric search and query or document prefixes?
In retrieval, queries are short questions while documents are long passages, so models are often trained for asymmetric matching.
Many models expect instructions or prefixes (for example a query marker versus a passage marker) and give noticeably worse results if you omit them.
Always follow the model's documented usage for queries and documents, and embed both with the same model version.
What happens when the input is longer than the embedding model's maximum length?
Most models truncate silently beyond their limit (often 512 to 8,192 tokens), so the tail of a long chunk is ignored by the embedding.
Keep chunks within the limit, and note that very long inputs also dilute meaning into one vector.
For long documents, chunk first, or use long-context embedding models, or techniques such as late chunking.
How do you handle multilingual or cross-lingual retrieval?
Use a multilingual embedding model that maps different languages into one shared space, so a query in one language can retrieve documents in another.
Evaluate per language with native-speaker labeled sets, because quality varies widely, especially for low-resource languages.
Watch tokenization (some languages use many more tokens per word), and consider language detection with metadata filters or translation as a fallback.
When should you fine-tune an embedding model, and how?
Consider it when a domain has specialized vocabulary or notions of relevance that general models miss, and you have measured a retrieval gap even after tuning chunking, hybrid search and reranking.
Data: pairs of queries and relevant passages (from click logs, labeled questions, or synthetic questions generated by an LLM for each chunk), plus hard negatives mined with the current model.
Train with a contrastive loss, hold out an evaluation set, and compare against the base model on recall and MRR as well as general-purpose tasks to check for regressions.
It requires re-embedding the whole corpus afterward.
What can embeddings not capture well?
Exact identifiers (order numbers, error codes, version strings), rare entities and spelling variants: dense vectors blur them, so add keyword or hybrid search.
Negation and fine distinctions ("covered" versus "not covered"), numeric comparisons and precise dates, and relationships spread across many chunks.
Queries needing reasoning over multiple documents; address these with query decomposition, reranking, structured filters and agentic retrieval.
What is embedding drift, and how do you change embedding models without downtime?
Vectors from different models (or even different versions of one model) live in incompatible spaces and often have different dimensions, so you cannot mix them in one index or convert them.
Safe migration: create a new versioned index, dual-write new documents to both indexes, backfill and re-embed the corpus in the background, evaluate the new index offline and on shadow traffic, then switch reads and keep the old index for rollback.
Store the model name and version with every vector so mismatches are detectable.
What are multimodal embeddings?
Models such as CLIP embed images and text into one shared space, so a text query can retrieve images and the other way round.
Uses: image search, product search, matching screenshots or diagrams to documents, and retrieval over slides and PDFs with figures.
Challenges: larger storage and compute costs, weaker fine-grained text understanding inside images, and the need to evaluate on realistic visual queries.
Why does chunking strategy matter so much in RAG?
The chunk is the unit of retrieval: if chunks are too large, each embedding blurs several topics and the prompt fills with noise; if too small, chunks lose context and the answer is split across pieces.
Good chunks are self-contained enough to be understood alone and focused enough to match a specific question.
Chunking determines recall, answer quality, latency and token cost, and it is often the cheapest place to gain retrieval quality.
What are the common chunking methods?
Fixed-size: split every N tokens or characters; simple, but can cut sentences and ideas in half.
Recursive character or token splitting: try paragraph breaks, then line breaks, then sentences, then words, keeping pieces under the size limit; a strong default for plain text.
Sentence or paragraph-based, and structure-aware splitting by headings, sections, list items or HTML and Markdown structure.
Semantic chunking: start a new chunk where the meaning shifts, detected by embedding similarity between neighboring sentences.
Document-specific: AST-based splitting for code, row or section-level for tables, and LLM-based (agentic) chunking for messy documents.
How do you choose chunk size and overlap?
Match the content and the questions: FAQ-style answers suit small chunks (roughly 200 to 400 tokens); manuals, contracts and technical documentation often need larger ones (roughly 500 to 1,000 tokens) or section-level chunks.
Overlap (often 10% to 20%) prevents an idea at a boundary from being cut apart, at the cost of more storage, duplicate results and slightly higher indexing cost.
Do not guess: run a small grid of sizes and overlaps against your labeled queries and compare recall@k, MRR and end-to-end answer quality.
Stay within the embedding model's maximum input length.
What is the purpose of overlap between chunks?
It preserves context across boundaries: a sentence or concept near the end of one chunk also appears at the start of the next, so a query about it can still match.
Too much overlap inflates the index and returns near-duplicate chunks; too little risks losing boundary information.
Structure-aware splitting reduces the need for overlap because boundaries fall on natural breaks.
What are the pros and cons of semantic chunking?
Pros: chunks follow topic boundaries, so each embedding is more coherent and retrieval can improve on documents with long, flowing prose.
Cons: it needs an embedding pass over every sentence at indexing time (cost and latency), produces variable and sometimes tiny or huge chunks, depends on a similarity threshold you must tune, and does not always beat a well-tuned recursive splitter on structured documents.
Evaluate it against simpler baselines on your own data before adopting it.
What are parent-child retrieval and sentence-window retrieval?
Both separate the unit that is searched from the unit that is given to the LLM.
Parent-child (small-to-big): index small child chunks for precise matching, then return the larger parent section, so the model gets surrounding context.
Sentence-window: index individual sentences and, when one matches, return it with a window of neighboring sentences.
Benefits: precise retrieval with enough context for generation; costs: more complex indexing, and larger prompts if many parents are returned (deduplicate parents).
What is the contextual chunk header or contextual retrieval technique?
Each chunk is stored with extra context prepended before embedding (and often before keyword indexing): the document title, section headings, or a short LLM-generated summary of where the chunk sits in the document.
This fixes chunks that are ambiguous alone, such as "the company's revenue grew 3% over the previous quarter" without saying which company or period.
Costs: an LLM call per chunk at indexing time (reduced with prompt caching) and longer chunk text; gains are often substantial for retrieval accuracy.
What is late chunking?
Instead of chunking first and embedding each chunk in isolation, a long-context embedding model processes the whole document, and chunk embeddings are produced by pooling the token embeddings within each chunk span.
Each chunk embedding therefore carries context from the rest of the document (for example what a pronoun refers to).
It needs a long-context embedding model and more compute per document, but avoids per-chunk LLM calls.
How do you chunk tables, PDFs and scanned documents?
Use layout-aware parsing and OCR for scans, and verify the extracted text with spot checks and automatic checks (empty pages, garbled characters).
Keep tables intact: convert to Markdown or structured text with the header repeated, rather than splitting rows from their column names; for very large tables, index rows with the headers or use text-to-SQL instead.
Attach captions, headings and page numbers as metadata, and describe charts and images with a vision model so they become searchable text.
How do you chunk source code?
Split on syntax boundaries (functions, classes, methods) using an AST or a language-aware splitter, instead of fixed character counts that cut through functions.
Include context in each chunk: file path, class and function names, imports and docstrings.
Consider code-specific embedding models, hybrid search (exact identifiers matter), and retrieval that can follow references such as callers and definitions.
What metadata should you store with each chunk?
Source document ID and URL, title, section path, page number, chunk position, language, and created and updated timestamps.
Access-control information (tenant, team, permission groups), document type, product or version, and a content hash.
Metadata powers filtering (by product, date or permission), citations, freshness handling, debugging and de-duplication, so decide the schema before indexing at scale.
What chunk enhancement techniques exist, and what are their pros and cons?
Techniques: prepend titles or summaries to chunks, generate hypothetical questions each chunk answers and index those, extract keywords or entities, and store both a short summary and the full text.
Pros: better matching between user questions and chunk content, more robust retrieval on vague chunks.
Cons: extra LLM cost and latency at indexing time, more storage, risk of generated text introducing errors, and re-processing when documents change.
Validate each technique against a baseline using retrieval metrics.
How do you handle very long documents or many levels of detail?
Use hierarchical indexing: summaries or section-level nodes at the top and detailed chunks below, so retrieval can first find the right section and then the right passage (RAPTOR-style tree summaries are one approach).
Combine small-to-big retrieval with metadata such as section paths, and route broad questions to summaries and specific questions to detailed chunks.
For one-off long documents, long-context prompting can be simpler than building an index, at higher cost per query.
How do you deduplicate content and keep the index up to date?
Compute a content hash per document and per chunk to skip unchanged content and detect exact duplicates; use near-duplicate detection (for example MinHash or embedding similarity) for repeated boilerplate across documents.
For updates, re-process only changed documents, replace their old chunks atomically, and delete chunks of removed documents so stale answers disappear.
Run ingestion through a queue with idempotent, retryable steps, and track document versions and last-indexed times so you can monitor staleness.
Compare keyword-based (BM25) and semantic retrieval.
BM25 scores documents by how often query terms appear (term frequency with saturation), how rare the terms are across the corpus (inverse document frequency), and normalizes for document length; it needs exact word matches, is fast, interpretable, and excellent for names, IDs and error codes.
Semantic (dense) retrieval matches meaning through embeddings, handling paraphrases and conceptual queries, but can miss exact tokens and is harder to explain.
Neither dominates, which is why production systems usually combine them.
How does hybrid search work, and when do you use it instead of pure semantic search?
Run keyword (BM25 or learned sparse) and vector search in parallel, then merge the ranked lists.
Reciprocal rank fusion (RRF) is a popular merge: each document's score is the sum of 1 divided by (k + rank) across lists, with k around 60; it needs no score normalization. The alternative is a weighted sum of normalized scores (for example 0.7 semantic and 0.3 keyword as a starting point), tuned on evaluation data.
Prefer hybrid when queries contain identifiers, product names, acronyms, version numbers or stack traces, when the corpus has specialized jargon, or when you need robustness across query types.
What is approximate nearest neighbor (ANN) search, and why is it needed?
Exact search compares the query with every vector (brute force): perfect recall but cost grows linearly with corpus size, which is too slow for millions of vectors at low latency.
ANN indexes build a structure (graph, clusters, compressed codes) that finds very likely nearest neighbors while examining a small fraction of vectors.
The core trade-off is recall versus latency versus memory; tuning parameters moves along that curve, and recall is measured against exact search results.
Brute force remains fine for small corpora (a few hundred thousand vectors) and for offline evaluation.
How does HNSW work, and what do M, efConstruction and efSearch control?
HNSW (Hierarchical Navigable Small World) is a multi-layer graph: each vector is a node linked to nearby nodes; upper layers are sparse long-range shortcuts, lower layers are denser.
Search starts at the top layer, greedily hops to closer neighbors, drops to the next layer, and at the bottom layer explores a candidate list to collect the nearest neighbors.
M is the number of links per node (higher gives better recall but more memory); efConstruction is the candidate list size while building (higher gives a better graph but slower build); efSearch is the candidate list size at query time (higher gives better recall but slower queries) and can be tuned on a live index, while M and efConstruction are fixed at build time.
Strengths: high recall and low latency, good for steady inserts. Weaknesses: large memory (vectors plus links), slow bulk builds, and awkward deletes.
How do IVF and product quantization (PQ) work?
IVF (inverted file) clusters vectors with k-means into cells; at query time only the nprobe nearest cells are scanned. More probes give higher recall and higher latency; recall drops for vectors near cell boundaries.
Product quantization splits each vector into m sub-vectors and replaces each with the ID of its nearest entry in a small learned codebook (256 entries fit in one byte), compressing a 768-dimension float32 vector (3,072 bytes) to, for example, 96 bytes.
IVF-PQ therefore scales to hundreds of millions or billions of vectors: a billion 768-dimension float32 vectors need about 3 TB raw, versus roughly 100 GB of PQ codes, at some cost in recall (often recovered by reranking with original vectors).
How do you choose between flat, HNSW and IVF-PQ indexes?
Flat (brute force): small data, exact results, simplest.
HNSW: best recall and speed when the vectors fit in RAM, with steady small writes; the default for millions of vectors.
IVF-Flat or IVF-PQ: when memory is the constraint (tens of millions and up), data is relatively static or periodically retrained, and some recall loss is acceptable.
Disk-based graph indexes (DiskANN) and scalar or binary quantization extend these ideas when RAM is limited. Always benchmark recall and latency on your own vectors.
You are at 0.90 recall and need 0.97 without changing the embedding model. What do you change?
Raise the query-time search effort: efSearch for HNSW or nprobe for IVF, at the cost of higher latency and CPU.
Rebuild with higher M and efConstruction (HNSW) or more cells and better training (IVF), at the cost of memory and build time.
Use less aggressive compression, or keep quantized vectors for search and rerank the top candidates using full-precision vectors.
Retrieve more candidates (a larger k) and rerank with a cross-encoder, or add hybrid search; measure recall against exact search on a held-out query set after each change.
What is metadata filtering, and what is the difference between pre-filtering and post-filtering?
Filtering restricts results by attributes such as tenant, product, date or permission group.
Post-filtering runs the vector search first and then removes non-matching results: simple, but you may end up with fewer than k (even zero) results if most top hits are filtered out.
Pre-filtering (or filtered search inside the index) applies the constraint during search, which guarantees k matching results but can slow HNSW traversal sharply under very selective filters, so many systems switch to brute force over the filtered subset when the filter is very selective.
Index frequently used filter fields and test tail latency with selective filters.
What is reranking, and why use a cross-encoder?
A reranker rescoring the top 20 to 100 candidates from first-stage retrieval reads the query and each candidate together, capturing fine-grained relevance that independent embeddings miss.
Cross-encoders are much more accurate than bi-encoders but add latency (tens of milliseconds per document on CPU, less on GPU), so they only run on a small candidate set.
Typical pipeline: retrieve 50 candidates with hybrid search, rerank, and pass the top 5 to 10 to the LLM. LLM-based rerankers are an option when quality matters more than latency and cost.
How do you balance relevance and diversity in retrieved chunks?
Plain top-k often returns near-duplicates of the same passage, wasting context.
Maximal marginal relevance (MMR) selects results iteratively, balancing similarity to the query against dissimilarity to the chunks already selected, controlled by a lambda parameter.
Other options: deduplicate parents when using small-to-big retrieval, cluster results, and cap chunks per source document.
What query transformation techniques improve retrieval?
Rewriting: turn a vague or conversational message into a clear standalone query, using chat history to resolve pronouns and references.
Multi-query: generate several paraphrases or sub-questions, retrieve for each, and merge results (often with RRF).
HyDE: have an LLM write a hypothetical answer, embed it, and retrieve documents similar to it, which helps when queries and documents look very different.
Step-back prompting and decomposition: ask a broader question first, or split a complex question into sub-questions answered separately.
All add LLM latency and cost, so apply them selectively and check that they actually improve retrieval metrics.
What is ColBERT, and what are multi-vector retrieval and late interaction?
Instead of one vector per document, ColBERT stores one vector per token; at query time each query token finds its best-matching document token, and the maximum similarities are summed (MaxSim).
This gives much of a cross-encoder's fine-grained matching while still allowing precomputed document representations, so it is stronger than single-vector bi-encoders, especially for out-of-domain queries.
Costs: much larger index (many vectors per document) and more complex serving; compressed variants and use as a reranker reduce that.
How do you choose how many chunks (top-k) to retrieve and pass to the LLM?
Retrieve a generous candidate set (for example 20 to 100) to protect recall, then rerank and keep a small set (often 3 to 10) that fits the token budget.
More context raises cost and latency and can hurt accuracy: models attend less to information in the middle of long prompts, so put the best chunks first (and last) and avoid padding with weak matches.
Tune k on evaluation data, and consider adaptive k based on score gaps or similarity thresholds.
What should the system do when nothing relevant is retrieved?
Use a similarity or reranker score threshold, and check whether the top results actually answer the question.
If evidence is weak, respond with "I don't know" or ask a clarifying question, fall back to another source (web search, human handoff), or try a rewritten query, rather than letting the LLM answer from nothing.
Log low-confidence queries; they reveal missing documents and gaps in the knowledge base.
How do you handle follow-up questions in conversational retrieval?
Rewrite each user turn into a standalone query using recent conversation history (an LLM call or a small model), so "what about the premium plan?" becomes a full question.
Retrieve with the rewritten query, but give the LLM the original conversation and the retrieved context.
Keep summaries of long histories, and decide when a new topic should reset retrieval context.
What is agentic or iterative retrieval, and when is it worth it?
Instead of one retrieval step, an LLM-driven loop decides what to search, evaluates the results, and searches again with refined queries, sources or filters until it has enough evidence.
It helps with multi-hop questions, ambiguous queries and questions spanning several data sources, and can combine search tools such as vector search, keyword search and SQL.
Costs: more latency and tokens, harder evaluation, and risk of loops, so cap the number of steps and measure whether accuracy gains justify it versus a well-tuned single-step pipeline.
What are the key metrics for evaluating retrieval quality?
Recall@k: the share of all relevant chunks that appear in the top k results; the most important metric because the LLM cannot use what was not retrieved.
Precision@k: the share of the top k results that are relevant; low precision wastes context and can distract the model.
Hit rate@k: the share of queries with at least one relevant result in the top k.
MRR (mean reciprocal rank): the average of 1 divided by the rank of the first relevant result, rewarding relevant items near the top.
NDCG: handles graded relevance and discounts relevant results that appear lower in the list.
Also track latency, and, for ANN indexes, recall against exact search.
How do you build a retrieval evaluation set?
Collect real user queries (logs, support tickets, search queries) and have domain experts mark which chunks or documents answer them.
To scale up or bootstrap, generate synthetic questions for sampled chunks with an LLM, filter out vague or leaky ones, and treat the source chunk as the relevant item (verify a sample by hand).
Include hard cases: ambiguous questions, exact-identifier queries, multi-chunk answers, and questions the corpus cannot answer.
Start with 100 to 300 queries, version the set, and re-run it after every change to chunking, embeddings, index parameters or reranking.
How do you debug poor retrieval in a RAG system?
Take a failing query and inspect each stage: was the document ingested and parsed correctly, was the right chunk created (not split or missing context), did the query embedding and rewrite make sense, and where did the relevant chunk rank in raw results?
If it is absent from the top 100, suspect parsing, chunking, filters or the embedding model; if it ranks 30th, add hybrid search or a reranker; if it ranks well but the answer is wrong, look at prompt assembly and generation instead.
Fix the class of problem, add the query to the evaluation set as a regression test, and change one thing at a time.
Retrieval metrics look good but users get wrong answers. What do you check?
Whether the relevant chunk is actually in the prompt (context budget truncation, ordering and the lost-in-the-middle effect) and whether conflicting or outdated chunks are also included.
Generation: grounding instructions, faithfulness of the answer to the context (measure with a judge or claim-checking), and whether the evaluation set resembles real queries.
Metric mismatch: document-level relevance may be marked correct while the specific passage needed is missing, so evaluate at chunk level and by end-to-end answer correctness.
What is a sensible latency budget for a retrieval pipeline, and how do you cache?
Example budget for a 1 to 2 second interactive answer: query rewrite 100 to 300 ms (if used), query embedding 20 to 50 ms, vector and keyword search 20 to 100 ms, reranking 50 to 200 ms, then LLM generation as the largest part.
Cache embeddings of repeated queries, cache results for popular queries, and add a semantic cache that returns stored answers for near-identical questions (with care for freshness and permissions).
Reduce work with approximate indexes, a smaller k before reranking, quantization, parallel calls, and streaming the answer.
How do you scale retrieval to tens of millions or billions of vectors?
Capacity math first: 50 million chunks of 768-dimension float32 vectors is about 154 GB raw (about 38 GB with INT8 quantization), before index overhead.
Techniques: scalar, binary or product quantization; Matryoshka truncation; sharding vectors across nodes with scatter-gather queries and replication for throughput and availability; tiered retrieval (fast coarse search then precise reranking with full vectors); and disk-based indexes when RAM is limited.
Add result and embedding caches, pre-filtering to shrink the search space, and monitor recall as parameters change.
How do you handle multi-tenancy and access control in retrieval?
Store tenant ID and access-control information with every chunk, and enforce it inside the search (metadata filters or per-tenant namespaces), never by filtering after generation.
Options: separate indexes or namespaces per tenant (strong isolation, more overhead, good for large tenants) versus a shared index with tenant filters (efficient for many small tenants, but needs careful filter performance and testing).
Watch for noisy-neighbor effects with per-tenant rate limits and quotas, test for leakage across tenants, and log which chunks were retrieved for audits.
How do you keep the index fresh and handle deleted or outdated documents?
Detect changes with content hashes, modified timestamps or event hooks from the source system, and re-chunk and re-embed only what changed.
Delete chunks for removed documents promptly (soft-delete with a filter first if the index handles deletes poorly, then compact), and store updated-at fields so retrieval can prefer recent versions.
Decide freshness targets per source (minutes for tickets, daily for wikis), monitor index lag, and handle superseded policies by versioning or filtering on effective dates.
How do you choose a vector database?
Consider scale (millions versus billions of vectors), latency targets, filtering and hybrid search support, multi-tenancy, update and delete behavior, operational burden (managed versus self-hosted), cost, and ecosystem integration.
Options: purpose-built stores (Pinecone, Weaviate, Qdrant, Milvus), search engines with vector support (Elasticsearch or OpenSearch), vector extensions of databases you already run (pgvector), and libraries (FAISS, Chroma) for embedded or prototype use.
For moderate scale where you already run PostgreSQL, pgvector avoids an extra system; choose specialized systems when you need very large scale, advanced filtering or high write throughput. Benchmark with your own vectors and filters.
What would you monitor in production for retrieval?
Zero-result and low-score queries (they reveal gaps in the knowledge base), distributions of top-1 similarity and reranker scores, and latency percentiles per stage.
Quality signals: thumbs up or down, re-asked questions, citation clicks, escalations to humans, and periodic sampled review with LLM judges calibrated to human labels.
Drift in query topics and languages, index freshness, ingestion failures, and the share of answers with sources.
How do you reduce embedding and retrieval cost?
Batch embedding calls, cache embeddings by content hash so unchanged text is never re-embedded, and skip boilerplate and duplicates before embedding.
Use smaller or truncated embeddings and quantization to lower storage and memory, and cheaper first-stage retrieval with reranking only the top candidates.
Retrieve fewer, better chunks to cut LLM tokens, cache frequent queries, and compare the cost of an open-source embedding model you host with a paid API at your volume.
What security risks exist in retrieval, and how do you mitigate them?
Data leakage across users or tenants: enforce permissions in the retrieval query and test with adversarial users.
Indirect prompt injection: retrieved documents can contain instructions; treat them as untrusted data, delimit them in the prompt, and limit what tools the model can use after reading them.
Knowledge-base poisoning: control who can write to the corpus, track provenance, and review or scan new content; also redact secrets and personal data before indexing.
Design the retrieval layer for a 50-million-chunk knowledge base with 50 queries per second and a 300 ms p95 retrieval budget.
Ingestion: chunk with structure-aware splitting, add contextual headers, embed in batches, and store vectors plus metadata (tenant, ACL, version, timestamps) with content hashes for incremental updates.
Index: HNSW with quantization (or IVF-PQ if memory is tight), sharded and replicated across nodes; keep a keyword index alongside for hybrid search.
Query path: rewrite follow-ups, run hybrid retrieval with permission filters (about 50 to 100 candidates), rerank the top candidates with a cross-encoder, and return 5 to 10 chunks.
Reliability and speed: embedding and result caches, timeouts with fallbacks (skip reranking if the reranker is slow), autoscaling on query load, and monitoring of recall, latency and cost.
Evaluation: labeled query set with recall@k and MRR in CI, plus online feedback and sampled review.