# Hippo recall evals

Measurable recall quality. Use `hippo eval <corpus.json>` to run, compare
configs before/after a scoring change, or gate CI with `--min-mrr`.

## Corpora

### `real-corpus.json`

15 hand-written queries with expected memory IDs selected by keyword match
against the current local + global store. Covers a mix of topics: project
rules, dev-environment gotchas, external project references, architecture
notes, and intentional near-duplicates. Some cases expect 1 memory, others
expect 2-9 to exercise MMR diversity.

Regenerate after the store changes significantly:

```bash
node scripts/build-eval-corpus.mjs evals/real-corpus.json
```

The script looks up expected IDs by content keywords, so if the store no
longer contains a matching memory the case will be silently dropped (with a
`WARNING: no matches for ...` line).

### Baseline results (2026-04-20, store ~1088 memories, embeddings fully populated)

Eval now routes through `searchBothHybrid` so global memories are in scope
(previously only local entries were searched, which capped recall on any
case with `g_` expected IDs).

| Config | MRR | Recall@5 | Recall@10 | NDCG@10 |
|---|---|---|---|---|
| default (lambda=0.7, localBump=1.2) | 0.967 | 0.622 | 0.682 | 0.731 |
| --equal-sources (localBump=1.0) | 0.967 | 0.632 | 0.682 | **0.735** |
| --no-mmr | 0.967 | 0.622 | 0.682 | 0.731 |
| --mmr-lambda 0.3 | 0.967 | ~0.56 | ~0.57 | ~0.65 |
| --mmr-lambda 0.9 | 0.967 | 0.622 | 0.682 | 0.731 |

**`--equal-sources` wins by +0.004 NDCG@10.** The full gain lands on
`powershell-chaining` (NDCG 0.63 -> 0.69) — with the 1.2x local bump,
the one local `mem_102793a1dfbb` dominated the six global paraphrases.
Removing the bump lets more paraphrases into top-10.

Default stays at 1.2x for now because the general case (local has fresher
project context) still benefits, but `--local-bump <f>` / `--equal-sources`
are available as tuning knobs.

**MRR = 1.0** across all configs: the first relevant hit is always ranked 1.

**MMR at lambda >= 0.7 is neutral on this corpus** because the cases with
near-duplicates (the React trio) list all three duplicates as expected, so
MMR neither helps nor hurts. MMR below 0.7 hurts because it demotes
near-duplicates that are actually all relevant.

**Embedding-weight sweep (0.3 to 0.9) is also neutral** — the set and
positional pattern of expected/non-expected in top-10 is stable regardless
of the BM25/cosine blend.

Takeaway: the default config is near the corpus ceiling. To improve
further we need harder cases (paraphrased queries, multi-hop links) where
the right answer is not BM25-obvious. That's future work.

### Methodology note

The initial version of this corpus was auto-generated by keyword match
and produced Recall@10 = 0.48. After hand-pruning false-positive expected
IDs (session summaries that mention the keyword in passing but aren't the
canonical answer) Recall@10 jumped to 0.63. **Your corpus is your eval:
noisy expected lists produce noisy numbers.**

### Parameter sweep findings (2026-04-20)

`scripts/sweep-params.mjs evals/real-corpus.json --out evals/sweep-results.md`
swept 30 combinations of (MMR lambda, embedding weight, local bump) through
the keyword corpus. Full numbers in `sweep-results.md`; the punchline:

| | MRR | R@5 | R@10 | NDCG@10 |
|---|---|---|---|---|
| default (lambda=0.7, emb=0.6, bump=1.2) | 0.967 | 66.2% | 67.1% | 0.722 |
| best (lambda=any, emb=0.4, bump=1.2) | 1.000 | 66.2% | 67.1% | 0.745 |
| delta | +0.033 | +0.0 | +0.0 | +0.023 |

**Embedding weight is the real lever.** Lowering it from 0.6 to 0.4 lifts
MRR to 1.0 and adds 2.3 pp to NDCG. **MMR lambda had no effect** on this
corpus — the handcrafted cases lack near-duplicate density for MMR to
de-cluster, so lambda=1.0 (relevance only) ties lambda=0.5 exactly.

**We did NOT change the default** because:

1. This is a single corpus of 15 handcrafted, keyword-heavy queries. BM25
   is well-suited to that style — lowering cosine weight makes BM25 score
   relatively more. Real user queries often paraphrase, where cosine wins.
2. The LLM-generated corpus (`build-eval-corpus-llm.mjs` with paraphrased
   questions) is the honest test for embedding-weight tuning. Run that
   before changing the default.
3. Overfitting to one corpus is a classic eval trap. The sweep is a
   tuning knob for users, not a new default.

Users who want the uplift today: `hippo recall <q> --embedding-weight 0.4`
or set `"embeddings": {"hybridWeight": 0.4}` in `.hippo/config.json`.

### LongMemEval results (2026-04-20, full 500 questions)

Ran v0.27 (hybrid BM25+cosine + MMR + outcome boost) against the full
LongMemEval 500-question benchmark for direct comparison with the
documented v0.11 baseline (BM25-only).

#### Initial run: budget=4000 (flawed comparison)

The first run used the production default `budget=4000` tokens. LongMemEval
memories average 14k chars (~3500 tokens), so the budget fit ~1 memory per
query. v0.11 used FTS5 with `top_k=10` and no token budget, always
returning 10 results.

| Metric | v0.27 budget=4k | v0.11 FTS5 (top_k=10) | delta |
|---|---|---|---|
| Recall@1 | 46.6% | 50.4% | -3.8 pp |
| Recall@3 | 46.6% | 66.6% | -20.0 pp |
| Recall@5 | 46.8% | 74.0% | -27.2 pp |
| Recall@10 | 46.8% | 82.6% | -35.8 pp |

The flat R@K curve (46.6 -> 46.8) confirmed budget saturation: only ~1
result per query was returned (mean 5.7, but 72/500 queries got exactly 1).

#### Root cause: budget saturation, not scoring regression

The apparent 35pp regression at R@10 was caused by the benchmark runner
comparing a budget-limited retrieval (v0.27) against an unlimited one
(v0.11). Once budget saturation is removed, the gap shrinks to 1.6pp.

#### Corrected run (full 500 questions, no budget limit)

| Metric | v0.27 budget=4k | v0.27 no-budget | v0.11 FTS5 | delta (fixed) |
|---|---|---|---|---|
| Recall@1 | 46.6% | 46.6% | 50.4% | -3.8 pp |
| Recall@3 | 46.6% | **67.0%** | 66.6% | **+0.4 pp** |
| Recall@5 | 46.8% | 73.8% | 74.0% | -0.2 pp |
| Recall@10 | 46.8% | 81.0% | 82.6% | -1.6 pp |
| answer_in_content@5 | 35.0% | **49.6%** | 46.6% | **+3.0 pp** |

**The 35pp R@10 gap was entirely budget saturation.** The real scoring
gap is 1.6pp at R@10. v0.27 wins on R@3 (+0.4pp) and answer_in_content@5
(+3.0pp), meaning the top-5 results more often contain the actual answer.

Per-type (full 500q):

| Type | v0.27 R@1 | v0.11 R@1 | v0.27 R@10 | v0.11 R@10 |
|---|---|---|---|---|
| single-session-assistant | **94.6%** | 87.5% | **100.0%** | 94.6% |
| knowledge-update | 52.6% | **70.5%** | **96.2%** | 93.6% |
| multi-session | 40.6% | **41.4%** | 82.0% | **84.2%** |
| temporal-reasoning | 42.1% | 43.6% | 78.9% | **82.7%** |
| single-session-user | 34.3% | **44.3%** | 71.4% | **77.1%** |
| single-session-preference | **16.7%** | 13.3% | 33.3% | **36.7%** |

v0.27 wins on single-session-assistant (+7pp R@1, +5pp R@10) and
knowledge-update R@10 (+2.6pp). v0.11 wins on single-session-user R@1
(-10pp). Overall, hybrid scoring trades some R@1 accuracy for better
content relevance (answer_in_content@5 +3pp).

#### v1.9.0 reranker sweep (2026-05-11, full 500 questions)

F6 reranker hardening (`docs/evals/2026-05-10-f6-reranker-result.md`,
prereg `docs/evals/2026-05-10-f6-reranker-prereg.md`). Sweep ran the
v1.9.0 hybrid path against the same 500-question LongMemEval workload
with three reranker tracks plus a no-reranker baseline. This release
does not re-assert the retracted −10pp magnitude.

| Version | R@1 | R@3 | R@5 | R@10 | answer_in_content@5 | Notes |
|---|---|---|---|---|---|---|
| v1.9.0 baseline | 50.4 | 67.6 | 75.6 | 83.6 | 49.2 | no reranker; same hybrid path as v0.27 with current ingest |
| v1.9.0 features (any topK) | 51.6 | 68.2 | 75.4 | 83.4 | 48.6 | Track 1 features reranker; topK 20/50/100 byte-identical (Gate-B FAIL) |
| v1.9.0 cross_encoder | 50.4 | 67.6 | 75.6 | 83.6 | 49.2 | Track 2; identity-fallback in test env (HF blocked); not a real cross-encoder evaluation |

See `docs/evals/2026-05-10-f6-reranker-result.md` for the full
workload-validity verdicts, per-type breakdowns, provenance, and the
roadmap-target (R@5 ≥ 85%) framing — that target is not met on the
workload tested.

#### Fix: `minResults` option

Added `minResults` parameter to `hybridSearch`, `physicsSearch`, `search`,
and `searchBothHybrid`. Guarantees at least N results regardless of
token budget. Prevents budget saturation when memories are large.

```bash
# CLI usage
hippo recall "my query" --min-results 5

# Programmatic
hybridSearch(query, entries, { budget: 4000, minResults: 5 });
```

Production default: `minResults=1` (backward compatible). Benchmark
runner default: `budget=1000000, minResults=10`.

#### Perf fixes landed while investigating

1. **MMR capped at top-100** (commit 014487f) -- was O(N^2) on large
   candidate sets. Dropped per-query time from ~50s to ~9s.
2. **`preparedCorpus` option on hybridSearch** (commit c9bf1a8) -- lets
   batch callers skip per-query tokenization. Further drop to ~6-7s/query.

### LLM-generated corpus (`llm-corpus.json`)

Harder-than-bootstrap cases with **paraphrased** queries — the right memory
is not BM25-obvious, so this is where embeddings and MMR should earn their
keep. Generated via Claude:

```bash
export ANTHROPIC_API_KEY=sk-...
node scripts/build-eval-corpus-llm.mjs --max 20 --out evals/llm-corpus.json
```

For each of `--max` random memories, Claude produces 3 realistic queries
that a developer might type where THAT memory is the ideal answer,
without reusing the memory's exact wording. Result: up to `3 * --max`
cases with 1 expected ID each.

Rate-limited at ~1 req/s. Skip memories that fail with `FAILED - <reason>`.

Compare against the keyword corpus to see which effects move:

```bash
hippo eval evals/llm-corpus.json --json > llm-baseline.json
hippo eval evals/llm-corpus.json --equal-sources --compare llm-baseline.json
```

## Model Profile Bench (null result, 2026-04-21)

Separate 20-case corpus (`model-profile-bench.json`) designed to test whether per-model context-injection profiles (budget + framing) could close behavioral gaps between Claude Opus 4.6 and 4.7. Tested four failure modes: invariant honor, hallucination guard, noise rejection, contradiction rejection.

**Result: 0 pp gap between 4.6 and 4.7 across all four types.** Both models scored 100% when hippo's default context block was injected. The planned per-model profile feature was scrapped — no measurable problem to solve.

| Model | invariant-honor | hallucination-guard | noise-rejection | contradiction-rejection |
|---|---|---|---|---|
| claude-opus-4-6 | 100% | 100% | 100%* | 100% |
| claude-opus-4-7 | 100% | 100% | 100%* | 100% |

\* A single Windows shell-escaping bug in the runner (not a model failure) caused one cell to error; it was identical for both models.

**Interpretation.** The 4.7 "regressions" circulating on social media (long-context collapse, instruction drift, gaslighting) don't appear to affect the specific failure modes hippo guards against — at least not when 1,500 tokens of curated memory are in context. Hippo's existing context injection is sufficient; no per-model tuning needed.

Corpus + runner + judge are reusable: `evals/model-profile-bench.json`, `scripts/run-model-profile-bench.mjs`, `scripts/model-profile-judge.mjs`. Uses `claude -p --model <id>` (no API key — subscription handles cost). Full investigation: `docs/plans/2026-04-21-model-aware-hippo.md` and `docs/plans/2026-04-21-phase-a-decision.md`.

## Writing new cases

A case is:

```json
{
  "id": "short-slug",
  "query": "natural-language question a user might type",
  "expectedIds": ["mem_abc123", "g_def456"],
  "description": "optional context so a failure is self-explaining"
}
```

Good cases:

- Use 2-6 expected IDs for queries where MMR matters (there are near-duplicates).
- Use 1 expected ID for "is this specific memory findable?" cases.
- Phrase queries in the user's voice, not in the memory's exact wording.

## Gate CI

```bash
# Fail the run if MRR regresses below 0.9
hippo eval evals/real-corpus.json --min-mrr 0.9
```
