# v1.7.5 Goal-Stack Eval — Pre-Registration

**Pre-registered:** 2026-05-07, BEFORE running the eval.
**Author:** Keith So
**Mechanism under test:** B3 dlPFC goal-stack boost — `applyGoalStackBoost` helper shipped v1.7.4 (lifted from v0.38 CLI-only path).
**Public-surface status of the claim:** the −10pp lift has NEVER been confidently asserted in shipped public docs. The v0.38 dlPFC plan v2 explicitly DROPPED the trap-rate −10pp claim ("v2 ... drop trap-rate −10pp claim; success becomes a controlled paired test on a new B3 micro-fixture"). The −10pp lift lives only as a v0.39 STRETCH hypothesis ("Stretch (v0.39 target, NOT v0.38 success): ... (b) sequential-learning trap-rate −10pp lift"). v1.7.5 is the FIRST attempt to confirm or fail-to-confirm this hypothesis on the public benchmark.
**This therefore is a hypothesis test, not a claim retraction.** "Retracted" framing in v1 of the v1.7.5 plan was over-strong — the eval RESULT is "supported / not supported", and either way it gets published with measured numbers + CIs. No silent retraction and no soft-framed survival of a failed hypothesis.

## Hypothesis

H₁: Activating the dlPFC goal-stack boost on top of base hippo reduces late-phase trap-hit-rate by ≥10 percentage points on the sequential-learning benchmark, with 95% CI lower bound > 0pp.

H₀: Activation produces no detectable improvement (95% CI for Δ overlaps 0).

## Protocol

- Benchmark: `benchmarks/sequential-learning` (v1.7.5 multi-seed harness with category-to-slot variance).
- **Conditions (4):**
  - **C0 — none:** `baseline` adapter, no memory.
  - **C1 — static:** `static` adapter, all lessons pre-loaded.
  - **C2 — hippo-base:** `hippo` adapter, `--use-goal-stack` OFF.
  - **C3 — hippo+goalstack:** `hippo` adapter, `--use-goal-stack` ON.
- **Seeds: 20.** Hash-derived in `run.mjs` (`Math.imul(0x9E3779B9, 1000+i) >>> 0` for i in 0..19) to avoid correlated mulberry32 streams.
- **Eval mode:** `--eval-strict` flag → simulator hard-fails on any pushGoal/completeGoal hook error. Hook-failure counts written to result JSON; non-zero counts disqualify the run.
- **Primary metric (PRE-LOCKED):** late-phase trap-hit-rate as reported by the existing `hitRateByPhase` function in `run.mjs`, which thirds the trap encounters chronologically (not the 50-task position thirds). Trap encounters per seed = 25; "late" = last 8 trap encounters.
- **Top-5 cap interaction (PRE-LOCKED):** a task is "trap hit" iff `isRecalled(top5, category) === false`. The top-5 cap is part of the metric definition; right-lesson-at-rank-6 counts as a hit, by design.
- **Reporting:** mean ± 95% CI per phase, per condition.

## Statistical method (PRE-LOCKED)

- **Estimand:** mean late-phase Δ in percentage points = `mean_late(C2) − mean_late(C3)` over the 20 paired seeds (positive Δ = goal-stack helped).
- **CI method:** exact paired sign-flip permutation, 10,000 resamples (`pairedPermutationCI` in `aggregate.mjs`). NOT t-test — phase rates are bounded discrete and t-test is fragile at n=20.
- **Robustness check (informational):** Wilcoxon signed-rank not implemented in v1.7.5; permutation CI is the primary and only test. Future eval may add Wilcoxon.

## Decision rule (mechanical)

Compute Δ and 95% permutation CI on Δ.

- **Δ ≥ 0.10 AND lower-CI bound > 0** → **HYPOTHESIS SUPPORTED.** Ship "SUPPORTED" template citing the measured Δ, CI, and per-condition phase rates.
- **Δ < 0.10 OR CI overlaps 0** → **HYPOTHESIS NOT SUPPORTED.** Ship "NOT SUPPORTED" template. The −10pp specific magnitude was not demonstrated; the mechanism's effect (positive, null, or negative) is reported with measured numbers regardless.
- **Δ < 0 with lower-CI < 0 < upper-CI is harmless null.** **Δ < 0 with upper-CI < 0** = goal-stack measurably HARMS performance → flag for v1.8 design review.

## Stop conditions (mid-run)

- **Sanity gate** — after C2 completes: if C2 late-phase mean is outside [0.04, 0.24] (more than ±10pp from README headline 14%), STOP — harness is broken or headline is stale. Investigate before scoring C3.
- **Hook failure gate** — after C3 completes: if `hookFailures.push > 0` OR `hookFailures.complete > 0`, the run is contaminated. STOP, fix, restart.
- **Seed crash policy** — if any seed crashes mid-run, drop that seed across ALL conditions (paired) and run a replacement seed with index 1020+. Do NOT cherry-pick.
- **Outlier policy** — if any single seed's late-rate is >3 stddev from the rest in either condition, investigate (FS error, network, etc.). Drop or include based on root cause; document the choice in the result doc.

## Pre-committed report templates

### Template "HYPOTHESIS SUPPORTED" (use IFF the decision rule selects it)

> **Goal-stack lift on sequential-learning benchmark — HYPOTHESIS SUPPORTED.** 20 paired seeds, exact permutation CI. Late-phase trap-hit-rate dropped by Δ = {DELTA}pp (95% CI [{CI_LOW}pp, {CI_HIGH}pp]). Goal-stack ON: late mean = {C3_LATE} (CI ±{C3_CI}pp). Goal-stack OFF: late mean = {C2_LATE} (CI ±{C2_CI}pp). Sanity anchors — none: {C0_LATE}, static: {C1_LATE}. Pre-registration: docs/evals/2026-05-07-v1.7.5-goal-stack-eval-prereg.md.

### Template "HYPOTHESIS NOT SUPPORTED" (use IFF the decision rule selects it)

> **Goal-stack lift on sequential-learning benchmark — HYPOTHESIS NOT SUPPORTED.** 20 paired seeds, exact permutation CI. Late-phase Δ = {DELTA}pp (95% CI [{CI_LOW}pp, {CI_HIGH}pp]) does not meet the pre-registered ≥10pp threshold with lower-CI bound above zero. Goal-stack ON: late mean = {C3_LATE} (CI ±{C3_CI}pp). Goal-stack OFF: late mean = {C2_LATE} (CI ±{C2_CI}pp). Sanity anchors — none: {C0_LATE}, static: {C1_LATE}. Mechanism shipped, claim not demonstrated. Future eval may revisit. Pre-registration: docs/evals/2026-05-07-v1.7.5-goal-stack-eval-prereg.md.

## Audit trail

- Plan: `docs/plans/2026-05-07-v1.7.5-sequential-learning-adapter.md`
- Code: branch `feat/v1.7.5-sl-adapter` commits `64ad38b` (Task 1 adapter contract) + `e2e565e` (Task 2 multi-seed harness)
- Outside-voice review: `senior-code-reviewer` agent + `codex exec --model-reasoning-effort=high` ran 2026-05-07; both flagged the dominant tag-mismatch P0 that v2 plan addresses.
- Claim inventory: `docs/evals/2026-05-07-v1.7.5-claim-inventory.md` (committed alongside this prereg).
