Querying & chat

Querying is the core interaction: you ask a question, CRTX retrieves the most relevant chunks from your collection, and GPT-4o generates a grounded, cited answer that streams back token-by-token.

---

#The streaming model

Answers are delivered over Server-Sent Events (SSE). This matters because it lets the UI show sources before the answer starts and render tokens as they arrive. A single POST /query/ produces a stream of four event types:

The sources array in metadata is the grounding — render it immediately so the user sees citations while the answer types out.

#Request

POST /query/
{
  "question": "What were Q3 gross margins?",  // 1–5000 chars, required
  "collection_id": "…",                        // optional; empty = personal namespace
  "session_id": "…"                            // optional; ties the exchange to a chat session
}

#Consuming the stream (JavaScript)

const res = await fetch(`${BASE_URL}/query/`, {
  method: "POST",
  headers: { "Content-Type": "application/json", Authorization: `Bearer ${token}` },
  body: JSON.stringify({ question, collection_id, session_id }),
});

const reader = res.body!.getReader();
const decoder = new TextDecoder();
let buffer = "";

while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  buffer += decoder.decode(value, { stream: true });
  const events = buffer.split("\n\n");     // SSE events are double-newline separated
  buffer = events.pop() ?? "";
  for (const block of events) {
    let type = "", data = "";
    for (const line of block.split("\n")) {
      if (line.startsWith("event: ")) type = line.slice(7).trim();
      if (line.startsWith("data: ")) data = line.slice(6);
    }
    if (!data) continue;
    const payload = JSON.parse(data);
    if (type === "metadata") renderSources(payload.sources);
    else if (type === "token") appendToken(payload.token);
    else if (type === "done") finish(payload);
    else if (type === "error") showError(payload.message);
  }
}

#Consuming the stream (curl, for debugging)

curl -N -X POST "$BASE_URL/query/" \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{ "question": "Summarize the risks section", "collection_id": "'"$COLLECTION_ID"'" }'

-N disables buffering so you see events as they arrive.

---

#Sources

Each entry in the sources array describes one retrieved chunk:

{
  "source": "annual-report.pdf",
  "chunk_index": 42,
  "text": "Gross margin for the quarter was 61.2% …",
  "score": 0.83,                       // similarity score
  "chunk_type": "text",                // or "image_description"
  "image_url": null                    // populated for figure sources
}

• chunk_type: "image_description" marks a chunk derived from a figure, chart, or scanned page. When image_url is set, the UI renders the actual image inline as a source (with an "IMG" badge).
• score is the raw retrieval score, useful for surfacing confidence or filtering.

The model is instructed to cite sources inline as [N] matching the numbered context it was given, and to only answer from that context — if the answer isn't supported, it says so rather than guessing.

---

#Chat sessions

A session is a saved conversation within a collection. Passing session_id on a query does two things: it loads recent history for context, and it persists the new exchange (question + answer + sources) into that session.

Messages come back as { id, role, content, sources, created_at }, so a reloaded conversation shows the same cited sources it did live.

#Follow-up questions just work

CRTX handles conversational follow-ups by:

1. Contextualizing retrieval. The last two user turns are prepended to the current question for embedding only, so "what about its pricing?" retrieves chunks about the subject you were just discussing — not generic pricing.
2. Passing history to the model. Up to the last ~10 exchanges are included in the prompt so the model understands what you're building on.

You don't do anything special — just keep sending queries with the same session_id.

---

#Latency, cost, and what gets logged

Every query records timing and token counts to query_logs (retrieval strategy, embedding/retrieval/generation latency, time_to_first_token_ms, prompt/completion tokens, model, retrieval scores). This is the raw material behind the evaluation dashboard and your cost analysis. See Operations & scaling.

Typical shape of the pipeline latency:

• Embedding + retrieval happen first (tens to low-hundreds of ms) and gate the metadata event.
• Time-to-first-token is the perceived responsiveness — sources render before it, so the UI never looks frozen.
• Generation streams until done.

---

#Error handling

• A malformed request (empty or >5000-char question) returns 422 before streaming.
• No access to the collection returns 403.
• If OpenAI fails after streaming starts, you get an error event ({ "message": "Generation failed" }) rather than a broken connection — handle it in your consumer.
• If the client disconnects mid-stream, the server logs a partial-generation event and stops cleanly.

Next: Retrieval tuning →