CRTX
DocsSign in →

Scaling Audit — CRTX (10 → 100–1000 concurrent users)

Audit date: 2026-07-02. Deploy target: Vercel (frontend) + Railway (backend, single uvicorn process, no --workers), Supabase (Postgres + auth + storage), Pinecone (vectors).

The architecture is fundamentally sound for scale: ingestion is offloaded to an arq/Redis worker, Pinecone absorbs vector load, streaming is async, and Supabase/OpenAI/Pinecone clients are module-level singletons rather than per-request. The bottlenecks are concentrated in the query hot path and a few whole-table reads.

Findings are ranked by severity. #1 has been fixed — see backend/app/routers/query.py (the three pre-stream Supabase calls are now wrapped in asyncio.to_thread). The rest are documented below for later work.

#✅ 1. (FIXED) Blocking Supabase calls inside the async def query endpoint

query() is async but made three synchronous, blocking Supabase HTTP round-trips (_can_query, _get_collection_config, _get_session_history) directly on the single event-loop thread before streaming. With one uvicorn process / one event loop, each of those froze all other concurrent requests, including in-flight token streaming.

Fixed by wrapping each call in asyncio.to_thread. Still recommended as a follow-up: run uvicorn with multiple workers (--workers N) so a single process is not the ceiling.

#2. list_users() reads the entire auth table into memory on every members/shares view

Files: backend/app/routers/chat.py:123-124 (get_collection_members), backend/app/routers/collections.py:139-140 (list_shares).

_db.auth.admin.list_users() fetches all auth users and filters in Python to build an email map.

  • Doesn't scale: O(total users) work + full-table transfer on every "view members" / "view shares" request, independent of the collection's actual member count.
  • Already subtly broken past ~50 users: Supabase list_users paginates (default 50/page); this code reads only page 1, so beyond 50 total users the email map silently drops people. Looping all pages to "fix" it makes the scaling problem worse.

Fix direction: look up only the specific candidate_ids (a targeted batch query by id), never enumerate the whole user base.

#✅ 3. (FIXED) Per-query LLM eval scoring + title generation ran in the web process threadpool

Every query scheduled a background task making up to 5 sequential OpenAI calls (title generation with up to 3 retries + 2 gpt-4o-mini eval scorers) plus blocking Supabase writes. FastAPI runs sync background tasks in the shared ~40-thread Starlette threadpool — the same pool that serves every sync def route — so under concurrency the pool saturated and plain CRUD endpoints queued behind eval scoring.

Fixed: the persist + score work moved to the arq worker.

  • New job module: backend/app/jobs/query_followup.py (persist_and_score + arq wrapper persist_query_result); _save_exchange / _generate_title moved here out of the router.
  • Registered in backend/app/worker.py.
  • backend/app/routers/query.py after_stream is now async and enqueues the job (a fast Redis call). If Redis is unavailable it falls back to running the work inline via asyncio.to_thread, so chat history is still saved (degraded but correct).

Deploy note: this makes the arq worker part of the query path, not just ingestion — so the worker process must be running in production (see the operational aside below).

#4. Missing indexes on the collections-list hot path → sequential scans

Verified live (Supabase project crux / lzmkprrszsvfndgdjzhq, 2026-07-02):

  • collections has only its primary key — no index on user_id.
  • collection_members has composite unique (collection_id, user_id), which does not serve lookups filtered by user_id alone.

list_collections (backend/app/routers/collections.py:70-72) filters both tables by user_id on every dashboard load → two sequential scans. Negligible today (tiny tables), but grows linearly with total collections/memberships across all tenants.

Fix direction:

CREATE INDEX IF NOT EXISTS idx_collections_user_id ON collections (user_id);
CREATE INDEX IF NOT EXISTS idx_collection_members_user_id ON collection_members (user_id);

(Lower-priority companion: collection_shares.collection_id has no dedicated index either; add one if the shares list becomes hot.)

#5. get_eval_stats pulls 500 full rows and aggregates in Python

File: backend/app/routers/evals.py:29-45.

Each analytics load selects up to 500 rag_evals rows with select("*") — including full retrieved_chunks JSON blobs and answer text — then computes averages/trend/worst-10 in Python. Heavy payload + CPU that grows with eval volume.

Fix direction: push aggregation into SQL (avg(), group by date, order by ... limit) via a Postgres view or RPC; return only what the dashboard renders. Lower severity because it's a dashboard/admin endpoint, not per-user query traffic.

#Deliberately NOT flagged (scale fine at 1000)

  • JWKS client cache (backend/app/auth.py) — in-memory, per-instance, correct.
  • lru_cache'd Pinecone index (backend/app/db/vector_store.py).
  • Module-level Supabase/OpenAI client singletons.
  • The arq ingestion pipeline itself.

Operational aside (not a scaling bottleneck, but verify): the arq worker is not in backend/Procfile (only web:). Confirm it runs as a separate Railway service, or ingestion silently never processes.