# Hippo Benchmarks

Two benchmarks, two different questions.

## 1. Sequential Learning Benchmark

**Question:** Does the memory system help agents learn from mistakes over time?

No other public benchmark tests this. LongMemEval, LoCoMo, and ConvoMem all test retrieval accuracy on a fixed corpus. None of them measure whether an agent with memory performs *better over a sequence of tasks* than one without.

```bash
cd sequential-learning
node run.mjs --adapter all
```

> **v0.11.0 informal results — RETRACTED v1.7.9.** v0.11.0 reported informal numbers that did not reproduce on the formal sequential-learning harness across three pre-registered workload variants (v1.7.5 full-late SANITY_FAIL, v1.7.6 budget sweep B*=NULL, v1.7.7 `--restrict-late-to 4` SANITY_FAIL). Every C2 hippo-base late mean returned 0% across every seed. **The magnitude is RETRACTED. The mechanism is shipped; no magnitude is currently claimed.** See top-level `CHANGELOG.md` v1.7.9 entry and `docs/RETRACTION.md`.

<details>
<summary>Original v0.11.0 informal numbers (RETRACTED — preserved as audit trail in git, not reproduced here)</summary>

v0.11.0 reported a single-run informal headline citing late-phase trap-rate decline. The specific numbers are archived at git tag `v0.11.0`. Retained in version control, not reproduced here, since reproduction risks accidental re-citation.

</details>

The benchmark, harness, and adapter contract remain shipped.

**Adding your own memory system:** Implement the adapter interface in `sequential-learning/adapters/interface.mjs` and run:

```bash
node run.mjs --adapter your-adapter
```

Zero dependencies. Node.js 22.5+ only.

## 2. LongMemEval Integration

**Question:** How accurately can hippo retrieve the right memory from a large corpus?

[LongMemEval](https://arxiv.org/abs/2410.10813) (ICLR 2025) is the industry-standard benchmark for AI agent memory. 500 questions across 5 abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention.

```bash
cd longmemeval
pip install -r requirements.txt

# Download data from HuggingFace (see README)
# Then run:
python run.py --data data/longmemeval_oracle.json
```

**Hippo v0.11.0 results (BM25 only, zero dependencies):**

| Metric | Hippo | MemPalace (raw) | MemPalace (reranked) |
|--------|-------|-----------------|---------------------|
| R@1 | 50.4% | — | — |
| R@5 | 74.0% | 96.6% | 100% |
| R@10 | 82.6% | — | — |

| Question Type | R@5 |
|---------------|-----|
| single-session-assistant | 94.6% |
| knowledge-update | 88.5% |
| temporal-reasoning | 73.7% |
| multi-session | 72.2% |
| single-session-user | 65.7% |
| single-session-preference | 26.7% |

Hippo achieves 74% R@5 with BM25 keyword matching and zero runtime dependencies. MemPalace's 96.6% uses ChromaDB embeddings. Adding `hippo embed` (hybrid BM25 + cosine) should close the gap.

**Fast retrieval mode** (recommended): `retrieve_fast.py` queries SQLite FTS5 directly — 500 questions in 2 seconds. The CLI-based `retrieve.py` takes hours due to subprocess overhead.

```bash
cd longmemeval
python ingest_direct.py --data data/longmemeval_oracle.json --store-dir ./store
python retrieve_fast.py --data data/longmemeval_oracle.json --store-dir ./store --output results/retrieval.jsonl
python evaluate_retrieval.py --retrieval results/retrieval.jsonl --data data/longmemeval_oracle.json
```

For full pipeline with LLM answer generation + evaluation (requires `ANTHROPIC_API_KEY`):
```bash
python run.py --data data/longmemeval_oracle.json
```

## What each benchmark proves

| | Sequential Learning | LongMemEval |
|---|---|---|
| Tests | Agent improvement over time | Retrieval accuracy on fixed corpus |
| Unique to hippo? | Yes (no other benchmark tests this) | No (industry standard) |
| Hippo result | RETRACTED v1.7.9 — mechanism shipped, no magnitude claimed (see CHANGELOG v1.7.9) | 74.0% R@5 (BM25 only) |
| What it proves | Decay + strengthening + outcome feedback produce learning curves | BM25 keyword search competes with embedding systems at zero dependency cost |
| Metric | Trap-hit-rate decline (early vs late) | Recall@K, answer-in-content |
| Dependencies | Node.js 22.5+ | Python 3.9+ (retrieval eval needs no API key) |
