# v1.7.7 Goal-Stack Eval — Result

**Run:** 2026-05-09
**Pre-registration:** `docs/evals/2026-05-09-v1.7.7-goal-stack-eval-prereg.md`
**Claim inventory:** `docs/evals/2026-05-09-v1.7.7-claim-inventory.md`
**Verdict applied:** **SANITY_FAIL — C2 preflight failed at the N=4 lattice gate. C3 (goal-stack ON) was NOT collected. Goal-stack data not produced this run.**

## SANITY_FAIL template (per pre-reg)

> **Goal-stack lift on sequential-learning benchmark — eval STOPPED per pre-registered C2 sanity gate.** v1.7.7 restricted-late workload (`--budget 2000 --restrict-late-to 4`), 20 seeds. C2 hippo-base late mean = **0.00%** (0 of 20 seeds non-zero), failing the pre-registered N=4 lattice gate (`mean ∈ [0.05, 0.50]` AND `≥3 distinct seeds non-zero`). Floor effect persists at last-4 — narrowing the metric did not produce a discriminating workload. **C3 (goal-stack ON) was NOT collected** — no goal-stack data was produced under SANITY_FAIL. Hypothesis remains untested on a benchmark variant. Per pre-committed escalation (v1.7.6 plan P2-4 + v1.7.7 prereg): v1.8 will explore adversarial trap categories with the constraints named in the v1.7.7 prereg's "Pre-committed v1.8 escalation constraints" section. Calibration: `docs/evals/2026-05-09-v1.7.6-calibration-result.md`. Pre-registration: `docs/evals/2026-05-09-v1.7.7-goal-stack-eval-prereg.md`.

## Reproducibility anchors

- `hippo --version` = `1.7.6`
- `which hippo` = `/c/Users/skf_s/AppData/Roaming/npm/hippo`
- `git rev-parse HEAD` = `fe74f130a03e6cefe27c0c4ebac7838f22c6bcde`
- `node --version` = `v24.13.0`
- `npx vitest --version` = `vitest/1.6.1 win32-x64 node-v24.13.0`
- `run.mjs` blob = `8d98ff0104a2f6d42a5116aa8f6a7b85e0328e37`
- `aggregate.mjs` blob = `286b811ca818366fb2b920fae31adf631f59a023`
- `analyze-v1.7.7.mjs` blob = `5983086794cf79e9c5f4b8791e376b932eb527a9`
- workload knob = `--restrict-late-to 4`
- budget = 2000 (default)
- run start = `2026-05-09T07:56:37Z`
- run end (C2 only) = `2026-05-09T08:04:31Z`
- wall-time (C2 only) = ~7m 54s

## Measured numbers (C2 only, 20 seeds)

| Condition | Early mean | Mid mean | **Late mean (last-4)** | Per-seed late distribution |
|-----------|------------|----------|------------------------|----------------------------|
| C2 hippo-base | 77.3% | 4.5% | **0.00%** | All 20 seeds at 0% |

**Lattice math:** at N=4 the per-seed late rate ∈ {0, 0.25, 0.5, 0.75, 1.0}; aggregate `lateMean = k/(20×4) = k/80`. Observed k = 0 hits across all 80 seed-slots in the last 4 trap encounters per seed. Distinct lattice values non-zero = 0.

**Adapter starvation check (post-review P1-6):** NO. Early mean 77.3% indicates active recall-and-miss in the first 11 trap encounters; mid mean 4.5% indicates rapid learning in the middle 10. The adapter is NOT starved — the workload is structurally easy in the last 4 encounters regardless of metric definition.

> **Display rounding note (v1.7.9, P2-2):** display rounding standardised to one decimal place. Underlying values unchanged in raw JSON at `results/v1.7.7-eval-C2-hippo-base/benchmark-1778313821987.json`. The retraction release does not modify any computed numbers.

## Why the gate fired

By the late phase (last 4 of 25 trap encounters), hippo has accumulated enough lesson memories from the prior 21 encounters that recall is reliable for every category in every seed. Narrowing the metric window does NOT produce headroom for the goal-stack mechanism to demonstrate a ≥10pp drop, because there is no trap-rate left to drop.

This matches the structural pattern from v1.7.5 (full-late=last 7, all C2 + C3 seeds at 0%) and v1.7.6 (budget reduction, all 5 budgets × 10 seeds at 0%). Three pre-registered workload variants tested; none produced a discriminating workload for the −10pp goal-stack hypothesis on this benchmark.

## Cumulative evidence (3 pre-registered variants — informational, not retraction)

| Release | Workload knob | Result | C3 collected? |
|---------|---------------|--------|---------------|
| v1.7.5 | full-late = last 7 (chronological-third) | C2 = 0.0% — SANITY_FAIL | YES (also 0.0%) — eval ran but stopped at sanity gate post-hoc |
| v1.7.6 | `--budget` sweep {200, 400, 600, 800, 1000} × 10 seeds | B*=NULL (lateMean=0% on all 5) | NO (no B* to use) |
| v1.7.7 | `--restrict-late-to 4` × 20 seeds | C2 = 0.0% — SANITY_FAIL | NO (sanity preflight blocked C3) |

The hard-stop clause in the v1.7.7 prereg fires on **NOT_SUPPORTED**, not SANITY_FAIL. v1.7.7 is SANITY_FAIL — the magnitude is not retracted automatically. The mechanism (B3 dlPFC goal-stack boost, shipped v1.7.4) remains shipped without a magnitude attached. The hypothesis remains formally untested on a discriminating workload.

**However:** three consecutive null/inconclusive results across distinct workload knobs is meaningful negative evidence. v1.8 (adversarial trap categories per pre-committed constraints) is the last pre-registered escalation. If v1.8 also SANITY_FAILs OR returns NOT_SUPPORTED, the magnitude claim should be treated as falsified pending a fundamentally different benchmark.

## What ships in v1.7.7

1. **`--restrict-late-to <int>` flag** end-to-end through `run.mjs` → `simulate()` → `hitRateByPhase(results, restrictLateTo)`. Reusable for any future window-restriction experiment. Default null preserves chronological-third behavior.
2. **`run.mjs` made import-safe** (`invokedAsScript` guard) so future tests can import `hitRateByPhase` and `parseRestrictLateTo` without spawning the benchmark. Same pattern applied to `calibrate.mjs` (latent shebang bug from v1.7.6 surfaced and fixed).
3. **`analyze-v1.7.7.mjs`** with extracted pure `computeVerdict({...tiePass})` (post-review P1-2 tie-degeneracy guard) and `exactPairedPermutationCI` helper (post-review P1-3 sensitivity check). Reusable for v1.8.
4. **17 new tests** (11 slice-math + 6 verdict). 1476 total passing.
5. **This SANITY_FAIL finding** documenting that window restriction is not a discriminating workload knob.

## What v1.7.7 does NOT ship

- The −10pp goal-stack hypothesis test on a discriminating workload. **Untested for the third time.** Per pre-reg, v1.8 retargets to adversarial trap categories with these PRE-COMMITTED constraints (named here BEFORE v1.8 implementation):
  - At least 3 new trap categories beyond the existing 10.
  - Lessons must use vocabulary with <40% Jaccard overlap with v1.7.5 lessons.
  - Same N=4 lattice gate (`mean ∈ [0.05, 0.50]` AND `≥3 distinct seeds non-zero`), OR a fresh gate justified by lattice math at v1.8's metric definition.
  - Same C2-before-C3 sanity preflight rule.
  - Categories authored BEFORE running C3.

## Why we're shipping despite the SANITY_FAIL (same as v1.7.5 and v1.7.6)

The pre-registration discipline is doing its job. We:

- pre-registered the workload knob (`--restrict-late-to 4`) BEFORE running v1.7.6 calibration,
- ran a 20-seed C2 preflight at the pre-registered metric,
- mechanically applied the N=4 lattice gate (FAIL: 0 of 20 seeds non-zero),
- documented the floor-effect investigation as the structural reason,
- did NOT collect C3 — no goal-stack data leak,
- pre-committed v1.8 escalation constraints BEFORE v1.7.7 ran,
- shipped the `--restrict-late-to` infrastructure and analyzer for v1.8 reuse.

Three null results across distinct knobs is a stronger signal than one. Honest reporting > brand maintenance.

## Raw artifacts

- `benchmarks/sequential-learning/results/v1.7.7-eval-C2-hippo-base/benchmark-1778313821987.json` (C2-only result)
- Plan: `docs/plans/2026-05-09-v1.7.7-restricted-late-window.md`
- Pre-registration: `docs/evals/2026-05-09-v1.7.7-goal-stack-eval-prereg.md`
- Claim inventory: `docs/evals/2026-05-09-v1.7.7-claim-inventory.md`
- v1.7.6 calibration result: `docs/evals/2026-05-09-v1.7.6-calibration-result.md`
- v1.7.5 prereg + result: `docs/evals/2026-05-07-v1.7.5-goal-stack-eval-{prereg,result}.md`

To re-derive C2 numbers: `node benchmarks/sequential-learning/run.mjs --adapter hippo --n-seeds 20 --eval-strict --budget 2000 --restrict-late-to 4 --output benchmarks/sequential-learning/results/v1.7.7-eval-C2-hippo-base/`. Same hash-derived seeds (1000+i for i in 0..19); deterministic.

---

## Update 2026-05-09 (v1.7.9 release) — retraction (forward-pointer)

The −10pp goal-stack lift magnitude tested by this eval is **RETRACTED publicly in v1.7.9**, based on cumulative evidence from three pre-registered workload variants (v1.7.5 SANITY_FAIL, v1.7.6 B*=NULL, v1.7.7 SANITY_FAIL — every C2 hippo-base late mean returned 0% across every seed).

v1.7.9 retracts on cumulative evidence rather than waiting for v1.8. The v1.7.7 prereg's SANITY_FAIL ≠ NOT_SUPPORTED distinction was wrong: three SANITY_FAILs across distinct workload knobs is meaningful negative evidence regardless of formal verdict label. The mechanism (`pushGoal`/`completeGoal` hooks, `--use-goal-stack`, `applyGoalStackBoost`) remains shipped from v1.7.4. **The magnitude is RETRACTED. The mechanism is shipped; no magnitude is currently claimed.** v1.8.0 (queued) explores adversarial trap categories as mechanism characterisation under the magnitude-smuggling guard in `docs/RETRACTION.md`.

See:
- `CHANGELOG.md` v1.7.9 entry
- `docs/evals/2026-05-09-v1.7.9-retraction-inventory.md`
- `docs/RETRACTION.md`
- `README.md` "What's new in v1.7.9"
