# v1.7.5 Goal-Stack Eval — Result

**Run:** 2026-05-07
**Pre-registration:** docs/evals/2026-05-07-v1.7.5-goal-stack-eval-prereg.md
**Claim inventory:** docs/evals/2026-05-07-v1.7.5-claim-inventory.md
**Decision applied:** **STOP — sanity gate failed (C2 late = 0.0%, outside pre-registered band [4%, 24%])**

## Outcome in one line

The hypothesis is **untestable on this harness** because both hippo-base (C2) and hippo+goal-stack (C3) saturate at 0% late-phase trap-hit-rate across all 20 seeds. Floor effect — no headroom for the goal-stack mechanism to demonstrate further improvement. The pre-registered sanity gate triggered before the decision rule could apply.

## Measured numbers (20 seeds, 4 conditions)

| Condition | Early mean (CI) | Mid mean (CI) | Late mean (CI) |
|-----------|-----------------|---------------|----------------|
| C0 none           | 100.0% (±0.0pp) | 100.0% (±0.0pp) | 100.0% (±0.0pp) |
| C1 static         | 33.3% (±0.0pp)  | 11.1% (±0.0pp)  | 14.3% (±0.0pp)  |
| C2 hippo-base     | 87.8% (±2.9pp)  | 11.1% (±2.4pp)  | **0.0% (±0.0pp)** |
| C3 hippo+goalstack | 87.2% (±3.1pp)  | 11.7% (±2.7pp)  | **0.0% (±0.0pp)** |

Paired Δ on late-phase (C2 − C3): **Δ = 0.0pp** (95% permutation CI [0.0pp, 0.0pp]).

Hook failures: `C3.push = 0`, `C3.complete = 0`. Eval-strict mode passed cleanly.

## Why the sanity gate fired

The pre-registered sanity gate said: *"if C2 late-phase mean is outside [0.04, 0.24] (more than ±10pp from README headline 14%), STOP — harness is broken or headline is stale."*

Measured C2 late = **0.0%**, well below the 4% lower bound. Per the mechanical rule: STOP.

**Investigation (root cause):** the divergence is real and benign. The README's 78→14% headline was measured on the canonical fixed-position task ordering, single run, pre-tag-fix. The v1.7.5 multi-seed harness changes the workload in three load-bearing ways:

1. **Category-to-slot variance** — per-seed shuffle of which category lands at which slot within each phase shape group. Even seed-1 differs from canonical.
2. **Tag-fix (Task 1 P0)** — memories now stored with `[task.trapCategory, ...category.tags, 'error']`. The `category.id` tag doesn't directly help `isRecalled` (which keys on `category.tags`), but it adds a strong searchable token to the BM25 index. Hippo recall is more reliable in the post-fix workload.
3. **Hippo CLI v1.7.4 changes** — `RecallOpts.sessionId` plumbing and the lifted `applyGoalStackBoost` may incidentally change recall ordering for unrelated reasons (audit handle threading, etc.). Not investigated in this eval.

Combined effect: hippo-base now hits 0% late on every seed. The goal-stack boost (C3) cannot demonstrate ≥10pp improvement because there is no trap rate left to reduce.

**The headline is stale, not the harness broken.** The harness, the tag-fix, and the multi-seed variance are all working as designed. The result is a different (better) workload that produces a different (lower) baseline.

## What is shipping in v1.7.5

- ✅ **Adapter contract** — `pushGoal/completeGoal` hooks exposed, integrated, tested. The mechanism is exercisable on the public benchmark; a future eval can pick up where this one stopped.
- ✅ **Multi-seed harness** — `--seed N`, `--n-seeds N`, `--eval-strict`, `aggregate.mjs` with `pairedPermutationCI`. Reusable for any future B-series eval.
- ✅ **Tag-fix** — memories stored with category.id so the goal-stack boost can match. Required for the mechanism to fire.
- ⚠️ **Eval inconclusive** — sanity gate fired due to floor effect, hypothesis untestable on this harness.

## What is NOT shipping (deferred to a future release)

- The −10pp hypothesis. Status: still untested on a discriminating workload.

## Future eval design (notes for v1.7.6 or later)

To put the workload back into the discriminating zone (C2 late > 4%), candidates:

1. **Reduce the store budget** — `hippo recall --budget 500` or smaller forces memory pressure. With only the top 2-3 results returned, marginal categories may miss.
2. **Add adversarial / harder trap categories** — current 10 categories have lessons that BM25 finds easily. Add traps where recall queries lexically diverge from stored lessons.
3. **Test on later positions only** — restrict the metric to last 4-5 trap encounters where memory has accumulated and floor effects are sharpest. The pre-reg's "late = last 8 trap encounters" was already the smallest workable bucket.
4. **Add temporal noise** — perturb stored lessons or recall queries with small text edits per encounter so exact-match recall is harder.
5. **Run on the LongMemEval public corpus** — different metric (R@5) but a known difficulty floor that doesn't saturate at 0%.

None of these blocks v1.7.5 shipping the contract + harness. They become a v1.7.6+ followup.

## Why we're shipping despite the inconclusive eval

The pre-registration discipline is doing its job. We:

- ran a 4-condition × 20-seed paired eval on a published spec,
- captured per-seed numbers, hook failures, sanity gates, and CIs,
- mechanically applied the decision rule (STOP per sanity gate),
- documented the floor-effect investigation as the root cause,
- left the −10pp hypothesis explicitly untested rather than claim or retract,
- and shipped the adapter contract + harness so the next person can finish the experiment.

Honest reporting > brand maintenance. v1.7.5 ships the infrastructure. The hypothesis remains an open question.

## Raw artifacts

- `results/v1.7.5-eval-C0-none/benchmark-*.json`
- `results/v1.7.5-eval-C1-static/benchmark-*.json`
- `results/v1.7.5-eval-C2-hippo-base/benchmark-*.json`
- `results/v1.7.5-eval-C3-hippo-goalstack/benchmark-*.json`
- Analysis script: `benchmarks/sequential-learning/analyze-v1.7.5.mjs`

Re-derive the numbers any time: `node benchmarks/sequential-learning/analyze-v1.7.5.mjs`.

## Update 2026-05-09 — v1.7.6 calibration

The "Future eval design" section above named **budget reduction** as candidate #1. v1.7.6 acted on that candidate via pre-registered calibration:

- Calibration plan: `docs/plans/2026-05-07-v1.7.6-discriminating-workload.md`
- Calibration result: `docs/evals/2026-05-09-v1.7.6-calibration-result.md`

**Verdict:** B\* = NULL. Across 5 budgets {200, 400, 600, 800, 1000} × 10 seeds, every run returned `phases.late = 0.0`. Late-phase trap suppression is total at every budget tested — the floor is structural to the 50-task workload, not a budget parameter. Per pre-registered escalation, v1.7.7 will sweep `--restrict-late-to last-4` (narrow the late metric to last 4 trap encounters). Adversarial categories deferred to v1.8.

The −10pp hypothesis remains untested on a discriminating workload. v1.7.6 ships the `--budget` infrastructure + the calibration finding + a bug fix on the starvation guard (read a non-existent JSON field; verdict unaffected because lateMean=0% was load-bearing).

## Update 2026-05-09 — v1.7.7 result

v1.7.7 acted on the pre-committed escalation: `--restrict-late-to 4`. **C2 sanity preflight FAILED** (mean=0.00% across 20 seeds; 0 of 20 seeds non-zero at the N=4 lattice gate). C3 (goal-stack ON) was NOT collected — no goal-stack data leak. Adapter was not starved (early=77.28%, mid=4.50%); the workload is structurally easy in the last 4 trap encounters regardless of window size.

Per pre-committed escalation in the v1.7.7 prereg, v1.8 retargets to adversarial trap categories with constraints named in advance (≥3 new categories, <40% Jaccard overlap, same lattice gate, same C2-before-C3 preflight, categories authored BEFORE C3). Three null/inconclusive results across distinct workload knobs is meaningful negative evidence; if v1.8 also fails, treat the −10pp magnitude as falsified.

Result: `docs/evals/2026-05-09-v1.7.7-goal-stack-eval-result.md`.

---

## Update 2026-05-09 (v1.7.9 release) — retraction (forward-pointer)

The −10pp goal-stack lift magnitude tested by this eval is **RETRACTED publicly in v1.7.9**, based on cumulative evidence from three pre-registered workload variants (v1.7.5 SANITY_FAIL, v1.7.6 B*=NULL, v1.7.7 SANITY_FAIL — every C2 hippo-base late mean returned 0% across every seed).

v1.7.9 retracts on cumulative evidence rather than waiting for v1.8. The v1.7.7 prereg's SANITY_FAIL ≠ NOT_SUPPORTED distinction was wrong: three SANITY_FAILs across distinct workload knobs is meaningful negative evidence regardless of formal verdict label. The mechanism (`pushGoal`/`completeGoal` hooks, `--use-goal-stack`, `applyGoalStackBoost`) remains shipped from v1.7.4. **The magnitude is RETRACTED. The mechanism is shipped; no magnitude is currently claimed.** v1.8.0 (queued) explores adversarial trap categories as mechanism characterisation under the magnitude-smuggling guard in `docs/RETRACTION.md`.

See:
- `CHANGELOG.md` v1.7.9 entry
- `docs/evals/2026-05-09-v1.7.9-retraction-inventory.md`
- `docs/RETRACTION.md`
- `README.md` "What's new in v1.7.9"
