Retrieval tuning

Retrieval quality is the single biggest lever on answer quality. This guide covers every knob CRTX exposes — the per-collection strategy, Top-K, chunk size, neighbour stitching, and optional reranking — and when to reach for each.

---

#The retrieval pipeline

question ──► (contextualize with prior turns) ──► embed
                                                    │
                                                    ▼
                             Pinecone query in collection namespace
                                     (fetch_k candidates)
                                                    │
                          ┌─────────────────────────┼───────────────────────┐
                          ▼                         ▼                        ▼
                     similarity                    mmr                   threshold
                   (top_k by score)      (diversify via MMR)      (score ≥ 0.70, top_k)
                          │                         │                        │
                          └───────── optional Cohere rerank ────────────────┘
                                                    │
                                                    ▼
                              neighbour stitching (± NEIGHBOR_WINDOW)
                                                    │
                                                    ▼
                                   numbered, source-labelled context

---

#Retrieval strategies

Set per collection via PUT /collections/{id}/config (or the Pipeline Config UI). Three strategies are available:

MMR vs. reranking are mutually exclusive per query. When the strategy is mmr, reranking is skipped (MMR needs vector values and does its own reordering).

---

#Top-K

Top-K is how many chunks are fed to the model as context. It defaults to 5 (TOP_K env var) and is applied at query time.

• Higher Top-K → more context, better recall, higher token cost, and more chance of diluting the prompt with marginally-relevant chunks.
• Lower Top-K → tighter, cheaper prompts, but you may miss the chunk that held the answer.

For most document QA, 4–6 is the sweet spot. Raise it for broad "summarize everything about X" questions; lower it for pinpoint factual lookups.

---

#Chunk size & overlap

Chunking happens at ingestion time, governed by CHUNK_SIZE and CHUNK_OVERLAP (see Configuration). The recursive splitter also keeps structured units (tables, slides, spreadsheet row batches) whole up to TABLE_MAX_CHARS before splitting them into prose.

Because chunk size is set at ingestion, changing it only affects newly-ingested documents. To apply a new chunk size to existing content, re-ingest it (safe — deterministic vector IDs overwrite cleanly).

The per-collection chunk_size in the pipeline config is stored on the collection and surfaced in the config UI; the character-level splitting used by the worker is driven by the CHUNK_SIZE / CHUNK_OVERLAP environment values. Set those at the deployment level to change ingestion chunking.

---

#Neighbour stitching

Retrieval returns individual chunks, but the answer to a question often spans a chunk boundary. CRTX mitigates this by stitching neighbours: for each retrieved text chunk, it fetches the adjacent chunks (same source, chunk_index ± NEIGHBOR_WINDOW) by their deterministic IDs and splices them back together before building the prompt.

• Controlled by NEIGHBOR_WINDOW (default 1 = one chunk on each side).
• Adds one extra Pinecone fetch (no extra embedding) — cheap.
• Skipped for image-description chunks (their order isn't contiguous prose) and when querying without a collection namespace.
• Set NEIGHBOR_WINDOW=0 to disable.

This is why answers often read coherently even when the exact sentence sat at the edge of a chunk.

---

#Reranking (optional, Cohere)

A cross-encoder reranker re-scores candidates by reading the query and each document together, which is far more accurate than embedding similarity alone. CRTX supports Cohere reranking as an opt-in layer:

1. Set RERANK_ENABLED=true and provide COHERE_API_KEY.
2. On similarity and threshold queries, CRTX over-fetches top_k × RERANK_FETCH_MULTIPLIER (default ×4) candidates, reranks them with RERANK_MODEL (default rerank-english-v3.0), and keeps the Top-K.
3. Fail-safe: any rerank error (outage, rate limit) silently falls back to embedding-score order, so a reranker problem never fails a query.

Reranking is the highest-leverage upgrade for precision-sensitive corpora. Enable it when "the right chunk exists but doesn't always rank first."

---

#Query contextualization for follow-ups

Before embedding, the last two user turns are prepended to the current question so follow-ups embed with their antecedents. This is a cheap, LLM-free step that prevents pronoun-y follow-ups ("and its limitations?") from retrieving the wrong material. It's automatic when you pass a session_id.

---

#A tuning playbook

Measure the effect of every change on the Evaluation dashboard — don't tune by vibes.

Next: Evaluation & quality →