This week, while building retrieval evaluation into Clumsies, I opened the run details screen and found that I could not explain it in plain language. It expected someone to grade retrieved candidates, search the corpus for evidence the system had missed, and understand enough of the ranking pipeline to save a useful evaluation case.
My first instinct was to remove almost all of that. If a retrieval result is wrong, why not let me report it as inaccurate with one click and ask the system to work out what happened?
That is a much better feedback flow. It is not yet a test case.
A report that says “this result was wrong” records a failure, but it does not say what the right result should have been. If I change the retrieval algorithm tomorrow, a different answer is not necessarily a better one. Information retrieval test collections need relevance judgments alongside their queries and documents. NIST describes those judgments as the “right answers”. Without one, I can collect complaints, but I cannot run a meaningful regression test.
The opposite design was technically honest and humanly unreasonable. A person should not have to search an entire corpus for missed passages just to report a bad result. That turns ordinary product feedback into a research annotation job.
I ended up separating the two moments. The first click records the failed run and preserves the query, corpus, and ranking trace. The system can then search for likely missing evidence and propose a diagnosis. The person has one narrower job: confirm whether the proposed evidence actually matches the original need. Only that confirmed case belongs in the evaluation set.
The interface became smaller once the distinction was clear. I removed the standalone details window, moved “Report inaccurate” into a contextual menu, and kept the review step for the point when there is evidence to inspect. The human judgment did not disappear. It became precise enough to be useful.
A thumbs-down can start a test case. It cannot be the test case.