Engineering study 01

Retrieval that knows when to stop

An AI system that declines is worth more than one that guesses. This is how we built one that does, and why declining is also cheaper.

EnterpriseRAGPython 3.11 · FastAPIPublic repository
354
BACKEND TESTS, 13 SUITES
14
PIPELINE STAGES
8
ARCHITECTURE DECISION RECORDS
0.35
RELEVANCE THRESHOLD

The problem

A retrieval system that answers from thin context sounds exactly like one that answers correctly. That is the whole difficulty. The failure is silent, it is confident, and by the time anyone notices, the answer has already been acted on.

So the design question is not how the model composes an answer. It is what happens when the corpus does not contain one.

The query path, and the gate

The gate is the design. Retrieval runs top-K 5 against a 0.35 cosine threshold, with a payload filter pinning results to status = COMPLETED. If nothing clears the threshold, the Response Composer is never invoked and the API returns HTTP 200 with is_grounded: false. The system declines, and it declines cheaply, because a refusal costs one model call instead of two.

Ingestion is a state machine, not a script

Fourteen stages, each with its own persisted status, its own error code and an explicit retry policy. A document can be inspected at any point and you can see exactly which stage it reached.

Every state is persisted. Work runs asynchronously in Celery with task_acks_late and a prefetch of 1, a soft limit of 1800s and a hard limit of 2100s. Transient failures retry three times at 10s, 20s and 40s; permanent ones stop and record a machine-readable code.

OCR only when needed

Tesseract 5.5.3 with OpenCV preprocessing, invoked only for pages that have no text layer. Running it unconditionally would multiply ingestion cost for no gain.

Chunking that respects pages

1000 characters with 150 of overlap, and a chunk never spans a page boundary. A citation that points across two pages is a citation nobody can check.

Local embeddings

all-mpnet-base-v2 at 768 dimensions, on the machine. ADR-003 records the reason: no per-document API cost. A hosted model is used for classification and composition only.

Duplicates are caught twice, for different reasons

The same document arriving twice is common. The same content arriving as a different file is more common still, and more expensive, because it silently doubles the weight of one source in every future answer.

ADR-005. Byte-identical files never reach storage. Content-identical files are kept for audit but never indexed, so retrieval cannot be skewed by the same passage appearing twice under two filenames.

Six decisions worth a technical reader’s attention

Refusal is a designed feature

Not an error branch. Below 0.35 the Response Composer is never called and the answer is an honest empty one. It is the cheapest path through the system, not the most expensive.

Filters run inside the query

The status = COMPLETED payload filter is part of the Qdrant search, not a list comprehension afterwards. That is the difference between correct retrieval and retrieval that looks correct on a small corpus.

No agent failure reaches the user

Three agents, each with an explicit timeout and a documented fallback. Identity extraction: 30s, returns null. Query Planner: 25s, zero retries, falls back to searching the raw question. Response Composer: 60s, one retry, falls back to a template over the retrieved passages.

Citations are verified, not trusted

Cited IDs are intersected with the IDs actually supplied to the model, then intersected again against live documents rows. A model that invents a source cannot produce a working citation.

Point IDs are deterministic

uuid5(namespace, "documentId:chunkIndex"). Reprocessing a document overwrites its own points rather than duplicating them, and the collection is created once and never recreated (ADR-006).

The test suite enumerates its own routes

One test walks the application route table and asserts every endpoint returns 401 without a token. A new endpoint cannot be added without authentication and quietly pass review.

Security is layered, and it fails closed

The browser never sees a token. Next.js route handlers act as a backend-for-frontend and attach credentials server-side. The Redis jti denylist fails closed, so a Redis outage revokes access rather than granting it.
ControlImplementation
Passwordsbcrypt at 12 rounds. Login enumeration defeated by identical body, status and timing.
Rate limitsLogin 5/min per IP · upload 20/min per user · chat 30/min per user · 300/min default
Upload validationExtension, MIME type, magic-byte signature and size. 20 MB documents, 2 MB images.
Headersnosniff · X-Frame-Options: DENY · CSP · Referrer-Policy · HSTS in production
PIIPAN and Aadhaar redacted in every log sink, and masked in chat answers before storage or return.
DataTen tables, each indexed to its query pattern, with an append-only audit log carrying user, request ID, IP and user agent.

What it costs to run

Most RAG cost estimates ignore ingestion and assume every question reaches the model. This one is counted per operation, which is why the refusal path matters commercially and not only editorially.

OperationModel calls
Ingesting a standard document3 model calls
Ingesting an identity document4 model calls
Answering a grounded question2 model calls
Refusing an ungrounded question1 model call
Embedding, at any volume0, runs locally
A refusal is the cheapest thing the system does. Designing the honest path to also be the cheap path is what makes it survive a cost review six months later.

Fourteen known limitations, documented in the open

The repository lists what the system does not do. Among them: chunk text is stored in plaintext, there is no virus scanning on upload, no redaction before text reaches the model provider, prompt injection is reduced rather than solved, the encoder truncates silently past 384 word-pieces, all-mpnet-base-v2 is English-centric, there is no reranking, fixed-window rate limiting allows double the intended rate across a boundary, Redis failures fail open for context and rate limits, there is no answer streaming, no Playwright end-to-end suite, and no coverage gating.

We publish that list deliberately. Every system has one. Most are discovered by the client six months in. You should expect the same candour about anything we build for you.

Apply this discipline to your retrieval

Sixty minutes on your data, your queries and where a threshold belongs in your system.

A 60-minute architecture review · no charge · the notes are yours either way