Retrieval evaluation (vector-search benchmark)

A reproducible benchmark for trip2g's hybrid search (BM25 + vector, fused with RRF). It measures Recall@k, nDCG@k, and MRR over a versioned golden set of bilingual RU/EN queries, so every change to the search pipeline can be A/B-tested with numbers instead of guesses.

What's where

Path Role
docker-compose.vecbench.yml Isolated stack: one app + MinIO + a bge-m3 embedding server, on its own ports (21081/21082, MinIO 22000/22001, embedding 11436). Independent of the e2e stack and dev.
scripts/vecbench.sh Driver: up / sync / eval / down / logs.
testdata/vecbench/vault/ The corpus: 6 topics × {EN, RU} = 12 notes, so all four retrieval directions are testable. See its README.md.
testdata/eval/golden_set.json Versioned query → relevant-URL pairs (qrels).
cmd/evalretrieval/ CLI: runs the golden set against a live GraphQL endpoint, prints metrics, writes a JSON artifact, gates on a min nDCG.
internal/retrievaleval/ Pure-Go metrics + golden-set loader + GraphQL search client + report aggregation.
docs/superpowers/eval-runs/*.json Committed run artifacts — the data spine for the before/after comparison.

Prerequisites

  • Docker (the stack builds the app image and runs a local bge-m3 server).
  • The bge-m3 model (~2.3 GB) downloads on first run into the shared ~/models/embedding cache, reused across the e2e and vecbench stacks.
  • Node (for obsidian-sync, used to push the vault).

Run it

# 1. Bring up the isolated stack, push the vault, generate embeddings, reload.
./scripts/vecbench.sh up        # or: make vecbench-up

# 2. Run the benchmark; writes docs/superpowers/eval-runs/<label>.json
./scripts/vecbench.sh eval baseline-bgem3

# 3. Tear down when done (keeps the model cache).
./scripts/vecbench.sh down

After editing the vault, re-ingest with ./scripts/vecbench.sh sync (re-pushes, drains embedding jobs, reloads the app).

How it works (and two gotchas)

  1. The embeddings endpoint is required. The vector lane embeds the query live on every request, so the benchmark needs a reachable bge-m3 server. The compose file provides one.
  2. Embeddings are generated after the vault sync, by async jobs. The in-memory chunk cache (and the bleve index) load at boot, so right after a fresh sync the vector lane is empty until the app reloads. vecbench.sh up/sync handle this by waiting for the job queue to drain (/debug/wait_all_jobs) and then restarting the app. (This stale-until-reload behaviour is a known limitation of the current loader — see docs/dev/vector_search.md.)
  3. The benchmark searches as admin. Anonymous search returns only live notes; the synced vault is draft/latest. vecbench.sh eval signs in with the dev code and passes the session JWT as a bearer token so the CLI sees the full (latest) index.

Building the golden set

Public benchmarks (MIRACL, ruMTEB, BEIR) measure embedding-model quality, but no public set matches this corpus. The golden set is small and project-specific:

  • Generate candidate queries with an LLM (3 per note: synonyms, no copied phrases, plus one "need-based" phrasing), in both languages, including cross-lingual (RU query → EN note and vice versa — the topics are parallel).
  • Hand-verify every pair: run it through the live endpoint, confirm the intended note is genuinely correct, and set expected_urls to the actual permalink(s) the endpoint returns. Drop ambiguous pairs (expect ~20% loss). Only verified: true pairs count.
  • Keep the four direction buckets (ru->ru, en->en, ru->en, en->ru) roughly balanced so cross-lingual regressions surface.

For picking/justifying the embedding model itself, the relevant public benchmarks are MIRACL-ru, ruMTEB retrieval, RusBEIR, NeuCLIRBench (EN→RU cross-lingual), and small BEIR sets (SciFact/NFCorpus) as an English CI guard.

Metrics

Metric Meaning
Recall@k fraction of relevant notes found in the top k
nDCG@k rank-discounted relevance (rewards putting the right note higher)
MRR mean of 1/rank of the first relevant result

The CLI reports each overall and per direction. A run with -fail-under-ndcg X (or EVAL_MIN_NDCG=X) exits nonzero when overall nDCG@k drops below X — use this as a CI regression gate once a baseline is established.

Obsidian Help set (cross-system, en + ru)

A second, public benchmark compares trip2g with obsidian-hybrid-search and qmd on Obsidian's
own help docs (176 notes, English and the Russian translation; 58 base + ~60 tricky queries
per language, graded relevance). Kit: scripts/obsidian-help-bench/ (README there), data:
testdata/eval/obsidian-help/, committed runs: docs/superpowers/eval-runs/obsidian-help/,
write-up: docs/en/thoughts/search-benchmark-obsidian-help.md.

Run history

Corpus: 48 notes (24 topics × 2 languages = 6 themes, each a core note + 3 intra-theme distractors). Golden set: 60 hand-verified queries, balanced across the four directions; relevance = both language versions of the topic.

Run Recall@10 nDCG@10 MRR Notes
00-baseline-bgem3 0.9833 0.9157 0.9417 bge-m3, current pipeline. Weakest direction: en→ru (nDCG 0.845, MRR 0.833).

Per-direction at baseline: ru→ru nDCG 0.935 / MRR 0.972 · en→en 0.932 / 0.972 · ru→en 0.934 / 0.958 · en→ru 0.845 / 0.833.

Min match and title in the body (2026-11-04)

Kit: scripts/enterprise-rag-bench/ (README there). Memory and timing from the same run: sizing.md.

Measured on commit 9f805d36 with one change to the text lane per column. Every column has the same vectors (bge-m3 through OpenRouter). "50k" is EnterpriseRAG-Bench: 722 gold documents plus 50 000 random ones of its 511 962, all 500 questions. It reads high against its leaderboard, because the near-duplicates that make the benchmark hard mostly fell out of the sample.

Set, metric All terms, separate title Any term, title in body Half the terms, title in body
50k, Recall@10 0.687 0.830 0.838
50k, Recall@10, semantic questions 0.384 0.600 0.624
Obsidian Help en, nDCG@5 0.763 0.794 0.784
Obsidian Help ru, nDCG@5 0.689 0.748 0.751
This page's corpus, nDCG@10 0.926 0.626 0.879
This page's corpus, translation in top 10 0.983 0.483 0.983
  • What it costs. Tricky Obsidian Help queries (typos, morphology) lose: en 0.785 → 0.782, ru 0.815 → 0.779. This page's corpus keeps Recall@10 at 0.99 and loses order only. On 50k the search p50 goes from 1031 ms to 1178 ms, most of it the query's remote embedding.
  • Stop words of both languages are dropped from the query. Each analyzer knows only its own, so "is" counted as a term in the Russian field, matched almost every note and got highlighted. Dropping them took the half-terms column from 0.857 to 0.879 on this page's corpus and left 50k within noise (0.842 → 0.838, semantic 0.640 → 0.624).
  • The share was fitted on these three sets, so these numbers lean optimistic. A third of the terms is a little better on Obsidian Help en (0.796) and worse across languages (0.806); two thirds drops 50k to 0.791.
  • Not measured. A Russian question whose answer exists only in an English note: this page's corpus holds translation pairs. The full 512k corpus: the in-memory index would not fit.
  • What the text lane cannot fix. Questions needing aggregation stay low (completeness 0.56). Chunks grow past chunkTargetTokens when a block has no blank lines (linear median 3186 characters, fireflies up to 24 254), and TEI cuts them at 512 tokens.

Answer quality, same questions before and after: 129 questions sampled per category, anthropic/claude-haiku-5.5 answering from the top 5 documents (first 4000 characters each), blind grading by Claude Sonnet with the benchmark's criteria. Answer correct 45.7% → 58.1%, facts 39.2% → 52.0%, semantic 28.1% → 40.6%. 24 questions changed outcome, 20 in favor of the new search (sign test p ≈ 0.002). The judge and the sample differ from the leaderboard's, so these are not comparable with it.

Load on 50k notes, on-disk index: note page 0.5 rps at one client and 2.6 rps at 50 (p50 18 s), 96% of CPU in similarnotes.calculateScores; search 15 rps at 20 clients (p50 1.2 s); /livez 138 000 rps. Not fixed here.