Cross-encoder reranker (blend mode)

Why a reranker exists

Search is hybrid: a bi-encoder vector lane (ANN over bge-m3 embeddings) plus a
BM25 lane, fused with Reciprocal Rank Fusion (RRF). Both stages are cheap
scorers: the bi-encoder embeds query and document independently, so it can
miss fine query↔document interactions.

A cross-encoder scores a (query, document) pair jointly in one forward
pass. That is strictly more expressive than independent embeddings, which is why
retrieve-then-rerank is a standard IR pattern — a cheap recall stage proposes
candidates, an expensive precision stage reorders the top-N. It needs a
rerank-capable server (the official HuggingFace text-embeddings-inference
image, or Infinity), not the plain /embeddings endpoint: different
architecture, different API (pair scoring vs vectors).

When it helps vs when it hurts

A cross-encoder is not free precision. It helps in some regimes and hurts in
others:

Helps when:

• the first stage is weak (vector-only, or BM25-only, no fusion) — there is
  order left to fix;
• the corpus is large and diverse, so true answers and distractors are far
  apart and the CE's extra expressiveness separates them;
• inputs are short, windowed passages that fit the CE's ~512-token window,
  so the model sees the whole candidate, not a truncated head.

Hurts when:

• the first stage is already strong (our hybrid RRF is ~0.92 nDCG) — there
  is little to gain and plenty to break;
• the corpus is small and topically adjacent — near-neighbour distractors
  share query terms, and a CE that over-weights surface term overlap promotes
  them over the true answer;
• inputs are whole notes longer than the window — after truncation the
  candidates look alike and the ordering degrades to noise.

Our site-search corpus sits squarely in the hurts column on the first two
axes. That is exactly why replace mode failed and why blend mode is conservative
by construction.

The old failure: replace mode

The first attempt re-scored the fused top-N and replaced the order with the
cross-encoder's ranking. It measured strictly worse, twice
(docs/dev/search_refactoring.md, variants F3 / F3b):

Per-query, the cross-encoder over-weighted surface query↔passage term overlap
and promoted near-neighbour distractors the strong first stage had correctly
ranked below the answer:

• "медленное брожение теста" (slow dough fermentation) → promoted sauerkraut
  over sourdough;
• "каналы вместо общей памяти" (channels vs shared memory) → promoted mutexes
  over goroutines;
• "пул воркеров" (worker pool) → promoted errgroup over goroutines.

On full-note passages (F3b) it collapsed to ~0.39: passages far exceeded the
~512-token window, so after truncation the notes looked identical and the order
became noise, pushing relevant notes out of the top-10.

Two root causes: (1) overriding a good prior instead of combining with it,
and (2) feeding truncated whole notes past the model window. Blend mode fixes
both.

The fix: blend mode + window discipline

final = (1 - w) * rrf_norm + w * ce_norm

Configuration

{
  "vector_search": {
    "enabled": true,
    "model": "bge-m3",
    "base_url": "http://embedding:8000/v1",
    "reranker": {
      "enabled": true,
      "default": false,
      "base_url": "http://reranker:8000/rerank",
      "model": "BAAI/bge-reranker-v2-m3",
      "top_n": 50,
      "output_k": 20,
      "blend_weight": 0.5
    }
  }
}

The vecbench stack serves bge-reranker-v2-m3 on CPU via the official
HuggingFace text-embeddings-inference image. First boot downloads the model
(~2.2 GB). It is intentionally not in the vecbench app.depends_on, so
baseline runs never pull the model. CPU inference is slow:
a warm rerank of 20 passages takes ~6 s, of 50 passages ~25 s on a loaded box —
so raise timeout_seconds well above the 10 s default when serving on CPU, or
the lane silently degrades to the RRF order on timeout.

Asking for it per request

The second stage is linear in candidates and slow on CPU — about a second per
candidate on our own box (50 passages of ~326 tokens measured at 48.5 s on
2026-08-21). At top_n=20 that is ~20 s added to a search, which is fine for an
agent doing research and unacceptable for a human waiting on a search box. So
who pays is a per-request decision, not only a deployment-wide one:

Callers express the preference as:

• MCP — a rerank boolean on search and federated_search. On
  federated_search it rides along to the peer, which applies it against ITS
  own config: reranking happens where the corpus is, because that is the only
  place the passages exist.
• GraphQL — a rerank: Boolean field on SearchInput.

The MCP argument is advertised only when enabled is true. An argument the
instance cannot honour is worse than no argument at all: the agent spends a turn
discovering that it did nothing. Its description also carries the price in
candidates and the effective default for that instance, because neither reaches
the model otherwise — and an agent that cannot see a cost will turn the option
on by reflex.

Note for existing deployments: default is false, so an instance that
previously ran with reranker.enabled: true and expected every search to be
reranked must now set "default": true to keep that behaviour.

The second failure: a batch limit nobody saw

Blend mode measured well and shipped, and then did not run in production for
months. The serving side had --max-client-batch-size 8 while top_n was 50,
and reranker.Client.Rerank posts every candidate in a single request. The
server answered 422 batch size 50 > maximum allowed batch size 8, BlendRRF
took its graceful-degradation path, logged a warning, and returned the stage-1
order. Search kept working, so nobody looked.

The reranker's own metrics eventually told the whole story at a glance:

te_request_count{method="batch"}      61359
te_request_failure{err="batch_size"}  61354
te_request_success{method="batch"}        5

Three lessons, all cheap in hindsight:

• The benchmark stack and production must agree on serving flags. The
  vecbench compose file runs the TEI image with no batch flags at all, so it
  gets the defaults (max-client-batch-size 32) and top_n=20 fits. Production
  set 8 for memory reasons. Nothing compared the two.
• Graceful degradation needs a counter. A fallback that is indistinguishable
  from success will hide forever. If the second stage is skipped because it
  failed, that has to be visible somewhere a human looks.
• top_n is a client-side batch size. Whatever serves the cross-encoder has
  a maximum request size; top_n must fit under it, or every call fails.

Benchmark

Eval harness: cmd/evalretrieval against a running instance, golden set
testdata/eval/golden_set.json (60 hand-verified bilingual queries, four
directions). Driver: scripts/vecbench.sh on the isolated
docker-compose.vecbench.yml stack. Metrics: nDCG@10 / MRR / Recall@10, on the
passage granularity (chunks, not whole notes). Both configs run against the
same bge-m3 hybrid first stage; the only change is whether the blended
reranker second stage is applied.

Two configs, blend_weight=0.5, top_n=20 (blend run had zero rerank
timeouts):

Per-direction nDCG@10 — the blend improves every direction, most where the
first stage was weakest (cross-language en→ru):

Artifacts: docs/superpowers/eval-runs/blend-00-stage1-off.json,
docs/superpowers/eval-runs/blend-01-w050.json.

This is the mirror image of the old replace-mode result (−0.034): keeping the
RRF prior and only nudging with the cross-encoder turns a regression into a
net win, and the biggest gains land on the weak cross-language direction — the
regime where the first stage had the most room to be corrected.

Decision: blend beats stage-1 on the benchmark → recommend enabling it where
the reranker sidecar is available. The enabled flag stays default-off
because the second stage needs an extra service (the CPU cross-encoder sidecar)
and adds ~6–25 s of latency per query on CPU — it is opt-in infrastructure, not
a free default. Turn it on (with a generous timeout_seconds) when the sidecar
is deployed; the measured quality gain justifies it.