Legal RAG pipeline

The problem

A mid-size litigation firm had bought a commercial case-law search tool and stopped using it within four months. The complaint was not that it returned nothing — it was that associates could not tell whether the thing it returned was the whole answer. Every result arrived with the same confident framing, whether it had found the controlling authority or a tangentially related footnote from a different circuit.

Underneath, it was a naive embedding search over 2.4 million documents with a fixed top-k of five. Long opinions were chunked on a character count, which routinely split a holding away from the reasoning that produced it. Nothing in the pipeline distinguished a binding precedent from a case that merely cited one.

The brief was narrow: keep the corpus, replace the retrieval, and make the system’s uncertainty visible to the person reading the output.

Approach

The first decision was to stop chunking on character counts. Court opinions have structure — headnotes, syllabus, majority, concurrence, dissent — and a parser that respects that structure produces chunks that are individually answerable. This alone moved accuracy more than any model change I made later.

I rejected a pure vector search. Legal queries are full of exact tokens that embeddings flatten: statute numbers, docket numbers, party names. The pipeline runs BM25 and dense retrieval in parallel and fuses with reciprocal rank fusion, which is unglamorous and beat every learned fusion I tried.

I also rejected a re-embedding pass on every query, which was the obvious fix for the latency problem and the wrong one. The 340ms was not model time; it was a sequential scan because the pgvector HNSW index was not being used under the metadata filter. Rewriting the filter as a partial index took the p95 to 80ms without touching the model.

The last piece is a jurisdiction-aware reranker. It is a small cross-encoder fine-tuned on 12k pairs, and it carries a hard rule the model cannot override: authority from outside the querying court’s jurisdiction is surfaced but labelled, never ranked first.

Results

The three numbers in the metric row above come from three different measurements, and they are not equally strong.

Latency is the most trustworthy: p95 over 10,000 real production queries across two weeks, measured at the API boundary, so it includes the reranker. Cold-start queries against uncached embeddings are worse — around 140ms — and that is not in the number.

Accuracy is graded against 400 questions written by two associates at the firm and labelled before they saw any system output. 71% → 89% is exact-match on the controlling authority appearing in the top three results. It is a real improvement, but 400 questions from one firm is a narrow benchmark and I would not generalise it.

Cost per query fell mostly because the reranker made a smaller generation model viable, not because retrieval got cheaper.

What I’d do differently

I spent three weeks fine-tuning the reranker before I profiled the database. The single largest latency win was an index fix that took an afternoon. Profile first — I know this, and I did it in the wrong order anyway.

I would also build the labelled evaluation set before writing any retrieval code rather than after. For the first month I was tuning against my own impressions of the output, which is not measurement, and at least one change I was confident about turned out to be neutral once there were numbers.