Skip to content
SlashClone
DiscoverUse in ClaudeSubmit a link
Sign in
Made by@slashclone

Search eval sweep

@slashclone/search-eval-sweep

Measure search quality with a fixed set of real queries: hits, misses, and false positives at the current cutoff and across a sweep of cutoffs, so ranking and threshold changes are chosen from data instead of guesses. Works for keyword, semantic, or hybrid search.

SkillDatav0.1.0 (latest)
Use in Claude Save Like Clone Comment
Activity
New: be the first to try it
  1. 1---
  2. 2name: search-eval-sweep
  3. 3description: Build a small search evaluation (real queries, expected results, nonsense queries) and a script that reports hits, misses, and false positives across score cutoffs. Use before tuning search ranking or a minimum match score.
  4. 4---
  5. 5
  6. 6# Search evaluation sweep
  7. 7
  8. 8Search changes feel better or worse depending on the last query you tried. An evaluation set makes them measurable.
  9. 9
  10. 10## The set (`search-eval-set.json`)
  11. 11
  12. 12- **Relevant queries:** 25 to 50 phrasings real people would type, each with the results they would be happy to see. Include plain-language needs ("my build failed, what went wrong"), not just product words.
  13. 13- **Nonsense queries:** five or more that should return nothing ("purple elephant banana keyboard").
  14. 14- Add a case every time someone reports a bad search.
  15. 15
  16. 16## The script
  17. 17
  18. 181. Run each query through the same code path the search page uses, including any minimum score filter.
  19. 192. A **hit** is an expected result in the top 5. A **false positive** is any result for a nonsense query.
  20. 203. Print one line per query: hit or miss, result count, best similarity, and the top result.
  21. 214. **Sweep** the cutoff (for example 0.1 to 0.4) and print hits, misses, and false positives at each, so the cutoff is picked from the table.
  22. 225. Cache query embeddings in a file outside the repo, so reruns cost nothing and respect provider rate limits.
  23. 236. Never print API keys, database URLs, or raw vectors.
  24. 24
  25. 25## Reading the results
  26. 26
  27. 27- A miss where the expected result is ranked first but filtered out is a **score** problem (cutoff or how the score blends evidence), not a ranking problem.
  28. 28- A miss where the result is absent is a **retrieval** problem (vocabulary, embeddings, or indexing).
  29. 29- Only ship a change when hits go up and false positives stay at zero.
Sign in to comment on a line