Search eval sweep

Search evaluation sweep

Search changes feel better or worse depending on the last query you tried. An evaluation set makes them measurable.

The set (search-eval-set.json)

• Relevant queries: 25 to 50 phrasings real people would type, each with the results they would be happy to see. Include plain-language needs ("my build failed, what went wrong"), not just product words.
• Nonsense queries: five or more that should return nothing ("purple elephant banana keyboard").
• Add a case every time someone reports a bad search.

The script

1. Run each query through the same code path the search page uses, including any minimum score filter.
2. A hit is an expected result in the top 5. A false positive is any result for a nonsense query.
3. Print one line per query: hit or miss, result count, best similarity, and the top result.
4. Sweep the cutoff (for example 0.1 to 0.4) and print hits, misses, and false positives at each, so the cutoff is picked from the table.
5. Cache query embeddings in a file outside the repo, so reruns cost nothing and respect provider rate limits.
6. Never print API keys, database URLs, or raw vectors.

Reading the results

• A miss where the expected result is ranked first but filtered out is a score problem (cutoff or how the score blends evidence), not a ranking problem.
• A miss where the result is absent is a retrieval problem (vocabulary, embeddings, or indexing).
• Only ship a change when hits go up and false positives stay at zero.