Hire a RAG engineer
RAG that still works at document two hundred thousand.
Most retrieval systems are impressive in a demo and disappointing in production, and the gap is almost never the model. It is chunking that destroyed the structure of the source, an embedding choice nobody revisited, no way to tell whether a change made answers better or worse, and a retriever that happily returns documents the asking user was never allowed to see. I build the unglamorous half — the part that decides whether the answers are trustworthy.
Why retrieval systems disappoint in production
The demo used twenty clean documents and everyone was impressed. The pilot used twenty thousand real ones — scanned contracts, spreadsheets exported to PDF, three generations of template — and the answers started drifting. The model did not get worse. The retrieval did, and nothing in the system was measuring it.
Nearly every failure I am called in to fix traces back to one of four decisions that got made by default. Chunking that split on a character count, cutting tables in half and severing clauses from the headings that gave them meaning. Embeddings chosen once and never compared, which is fine until your corpus is full of part numbers and abbreviations that semantic similarity treats as noise. No evaluation set, so every tuning change is argued from anecdote and nobody notices that fixing one question broke three others. And permissionsbolted on afterwards, usually as an instruction in the prompt, which is not access control — it is a polite request to a system that has already been handed the document.
The model layer is the commodity now. What separates a retrieval feature people rely on from one they quietly stop opening is whether the right passage reliably arrives in the context window, whether the answer points back at a source a domain expert can check, and whether the system is willing to say it does not know. That is infrastructure work, and it is the work I do.
How I build it
Start from the questions, not the corpus
Before anything is indexed, we write down the twenty questions the system has to answer well and who is allowed to ask them. That list drives chunking, metadata, and the evaluation set — and it usually reveals that a third of the intended corpus is irrelevant and one ignored table is essential.
Chunking and metadata are the real model
Fixed-size splits shred tables, headings, and clause numbering — exactly the structure that made the document answerable. I chunk along the document's own boundaries, carry parent context and source metadata on every chunk, and keep the identifiers needed to cite a precise location back to the user.
Retrieval quality before generation quality
If the right passage is not in the context window, no prompt rescues the answer. Hybrid search so exact identifiers and part numbers still match, reranking on top of vector recall, and query rewriting for the real questions people type rather than the clean ones in the demo.
Grounding, citations, and knowing when to refuse
Every claim traces to a retrieved passage the user can open and check. Just as important, the system says it does not know rather than producing a fluent guess — which is the difference between a tool a domain expert keeps using and one they quietly stop trusting after the first confident error.
An evaluation set you can run on every change
A graded question set, retrieval metrics separated from answer metrics, and a regression run in CI. Without it, tuning is superstition: someone changes the chunk size, the demo feels better, and nobody can tell whether the system actually improved or three of your users' questions just broke.
Permissions enforced at retrieval, not in the prompt
In multi-tenant products and regulated data the filter belongs in the query, scoped by the same authorization rules as the rest of your API. Asking a model politely not to reveal a document it was handed is not an access-control policy.
What you get
- An ingestion and chunking pipeline shaped around your actual documents
- A retrieval layer with hybrid search, reranking, and permission filters
- Grounded answers with citations that resolve to a specific source location
- A graded evaluation set plus a regression run wired into CI
- Cost and latency instrumentation per query, so spend is visible before it surprises you
- The whole thing inside your existing API, auth, and data model
Where I have done this before
The agentic document layer reads loan files, cross-checks values against the record, and flags discrepancies and missing information — grounding extraction in the source rather than trusting a single model pass.
Not a retrieval system, but the discipline underneath one: item-level records with chain-of-custody, where a wrong number becomes a wrong regulatory disclosure.
How we would work
- Own a build end-to-end, from data model to shipped interface
- Drop into an existing team and carry a workstream
- Scope and prototype a feature before you commit a budget to it
- Review an architecture before it has to scale
I work to your timezone rather than asking you to work around mine — my hours shift to cover your team’s working day, wherever that is. Live calls where they earn the time, written updates for everything else.
Top Rated Plus · Top 3% on Toptal · 100% Job Success · 10 yrs in production.
Questions people ask before hiring
Is RAG the right approach, or should we fine-tune?
Fine-tuning teaches a model a style, a format, or a narrow task. Retrieval gives it facts it did not have. If your content changes — policies, tickets, documents, product data — retrieval is almost always the answer, because updating an index is instant and retraining is not. The two also compose: retrieval for the facts, a fine-tune for the house voice. Most teams asking about fine-tuning need retrieval and better prompting first.
How large does our corpus need to be for this to be worth it?
Volume is not the trigger — searchability is. A few hundred documents that people genuinely cannot find their way around justifies retrieval more than a hundred thousand well-indexed ones. Scale changes the engineering, though: under a few thousand chunks almost any setup looks fine, and past that, reranking, metadata filtering, and evaluation stop being optional.
Do we need a dedicated vector database?
Usually not at first. If you already run Postgres, pgvector will carry you a long way and saves you a second system to operate, back up, and keep in sync. A dedicated vector store earns its place at scale, or when you need filtering and hybrid behaviour that your primary database makes awkward. I would rather ship on your existing infrastructure and move later with evidence than add a dependency on day one.
How do you stop it from inventing answers?
Three layers. Retrieval quality first, since most hallucination is really a retrieval miss the model papered over. Then grounding: answers cite the passages they came from, and a user can open the source. Then an explicit path to refuse — when nothing relevant is retrieved, the system says so instead of guessing. The evaluation set keeps all three honest as the corpus grows.
Our data is sensitive. How do permissions work?
Access control happens in the retrieval query, not in the prompt. Chunks carry the same tenancy and role metadata as the underlying records, every search is filtered by the caller's permissions before the model ever sees a passage, and the model is never handed something the user could not open themselves. My healthtech work ran under exactly this constraint.
How do you work with a distributed team?
I work to your timezone rather than the other way round — my hours shift to cover your team's working day, whether that is London, New York, or the West Coast. Standups, review calls, and pairing happen live inside that window; the rest is async with a written update, so you always know where a piece of work stands.
Tell me what you are building.
You get an honest answer on fit within 24 hours — including when the answer is that you do not need me.