Measuring retrieval honestly
- retrieval
- evaluation
Placeholder post. The notes collection is scaffolded and styled; this exists so the index and detail pages can be seen rendering real content.
The short version
Recall@k tells you the right chunk was somewhere in the context window. It does not tell you the model used it, and it does not tell you what happened on the queries where it was absent. Those are the two things worth knowing.
What to measure instead
Report the failure distribution, not just the hit rate. A system that is right 89% of the time and confidently wrong the other 11% is worse in practice than one that is right 80% of the time and says so on the rest.