# v1.7.7 Goal-Stack Eval — Pre-Registration

**Pre-registered:** 2026-05-09, BEFORE running the eval.
**Author:** Keith So
**Mechanism under test:** B3 dlPFC goal-stack boost (`applyGoalStackBoost`), shipped v1.7.4.
**Workload variant:** default budget (2000) with `--restrict-late-to 4` — late-phase metric narrowed from "last third (7 of 25)" to "last 4 of 25 trap encounters". Single-line workload tweak preserving the metric definition shift as the only knob.
**Public-surface status of the claim:** the −10pp magnitude was untestable in v1.7.5 (floor effect on full late = last 7) and untested in v1.7.6 (budget reduction confirmed not discriminating). v1.7.7 tests on the restricted late window; result is "supported / not supported", published with measured numbers + CIs either way.

## Hypothesis

H₁: Activating the dlPFC goal-stack boost on top of base hippo reduces late-phase trap-hit-rate by ≥10 percentage points on the v1.7.7 restricted-late workload (`--budget 2000 --restrict-late-to 4`), with 95% CI lower bound > 0pp.

H₀: Activation produces no detectable improvement (95% CI for Δ overlaps 0).

## Why this metric switch is a pre-committed adaptive follow-up (post-review P0-4)

The escalation was pre-committed in the v1.7.6 plan (commit `c670ac9`, "Plan v2 — Why budget reduction, not the other 4 v1.7.5 candidates"):

> Pre-committed escalation order:
> - If v1.7.6 SANITY_FAILs → v1.7.7 will be `--restrict-late-to last-4` (deterministic, single-line workload tweak).
> - If v1.7.7 also fails → v1.8 explores adversarial trap categories (requires new lesson authoring).
> - LongMemEval R@5 corpus is a separate eval line, not a substitute.

v1.7.6 SANITY_FAILed on calibration. v1.7.7 acts on the pre-committed next step. The metric value `4` was named in the v1.7.6 escalation prose BEFORE running the v1.7.6 calibration — so this is not a result-conditional choice.

**This is a pre-committed adaptive follow-up, not p-hacking.** The chain (v1.7.5 → v1.7.6 → v1.7.7) is an adaptive sequence over workload knobs, with each individual knob pre-registered before its eval ran. The decision rule (Δ ≥ 10pp AND lower-CI > 0pp) is unchanged from v1.7.5 and v1.7.6. The seed range (0..19) is unchanged. The four-condition design is unchanged. The metric switch is the only knob in v1.7.7. The C2 sanity band changes because the N=4 lattice required recalibration (see "Sanity gate" below).

## Hard-stop clause (post-review P0-4 — retraction at NOT_SUPPORTED)

If v1.7.7 returns NOT_SUPPORTED, **the −10pp magnitude is RETRACTED from CHANGELOG / README headlines**. v1.8 (adversarial trap categories) ships as a *new hypothesis*, not a continuation of this one. The cumulative evidence after v1.7.5 saturated, v1.7.6 found B*=NULL, and v1.7.7 found Δ < 10pp is sufficient to retire the magnitude claim. The mechanism (B3 dlPFC goal-stack boost) remains shipped from v1.7.4 with no specific magnitude attached — that does not change.

## Reporting scope (post-review P1-1 — cross-version comparability lock)

v1.7.7 reports **late-phase only as the comparable metric**. Early/mid are recorded for audit but are NOT comparable to v1.7.5/v1.7.6 due to the slice re-split:

- v1.7.5 chronological-third: early=9 / mid=9 / late=7 trap encounters.
- v1.7.7 restrict-late-to=4: early=11 / mid=10 / late=4 trap encounters.

Early-phase memory load is 22% larger at N=4 than at the default. **Do not cross-cite early/mid across versions.** "Learns?" boolean is also informational only.

## Protocol

- Benchmark: `benchmarks/sequential-learning` (v1.7.5 multi-seed harness).
- **Workload knob:** `--budget 2000` (default), `--restrict-late-to 4`.
- **Conditions (4):** C0 none, C1 static, C2 hippo-base (`--use-goal-stack` OFF), C3 hippo+goalstack (`--use-goal-stack` ON).
- **Hypothesis seeds: 20.** Hash-derived in `run.mjs` (`Math.imul(0x9E3779B9, 1000+i) >>> 0` for i in 0..19) — same as v1.7.5/v1.7.6.
- **Eval mode:** `--eval-strict`.
- **Primary metric:** late-phase trap-hit-rate over the LAST 4 trap encounters per seed. Slice math (Option A from the plan): late = trapTasks.slice(21..25), early = trapTasks.slice(0..11), mid = trapTasks.slice(11..21). Disjoint and exhaustive.
- **Top-5 cap interaction:** unchanged from v1.7.5 — a task is "trap hit" iff `isRecalled(top5, category) === false`.

## Statistical method (PRE-LOCKED, unchanged primary + new sensitivity check)

- Estimand: mean late-phase Δ in pp = `mean_late(C2) − mean_late(C3)` over 20 paired seeds (positive Δ = goal-stack helped).
- **Primary CI:** 10,000 Monte Carlo resamples of paired sign-flip permutation (`pairedPermutationCI` in `aggregate.mjs`). NOT t-test — phase rates are bounded discrete.
- **Sensitivity check (POST-REVIEW P1-3, PRE-COMMITTED):** If `|Δ - 0.10| < 0.02` OR `|ciLow| < 0.02`, ALSO run **exact enumeration over 2^20 ≈ 1M sign-flip combinations** (computationally cheap; preserves cliff-edge analysis). If exact and Monte Carlo disagree on the verdict, the result doc reports BOTH and treats the result as borderline.
- **Tie-degeneracy guard (POST-REVIEW P1-2, PRE-COMMITTED):** Compute `tiePass = (count of zero paired diffs) / 20 ≤ 0.70`. If `tiePass=false` (>14 of 20 paired diffs are zero), `computeVerdict` returns `NOT_SUPPORTED` with `degenerate: true` in the structured JSON summary. The result doc flags this with: "CI is degenerate due to ties; permutation test underpowered at N=4 lattice."
- **Discrete-distribution caveat:** at N=4 trap encounters per seed, per-seed late rate ∈ {0, 0.25, 0.5, 0.75, 1.0}. Aggregate `lateMean = k/(20×4) = k/80 ∈ {0, 1.25%, 2.5%, ..., 100%}`. Result doc reports a 2-row markdown histogram (rows: C2, C3; columns: 0%, 25%, 50%, 75%, 100%; cells: count of 20 seeds at that lattice value).

## Decision rule (mechanical)

Compute Δ and 95% permutation CI on Δ.

- **Δ ≥ 0.10 AND lower-CI > 0** → **HYPOTHESIS SUPPORTED.** Ship "SUPPORTED" template citing the measured Δ, CI, and per-condition phase rates.
- **Δ < 0.10 OR CI overlaps 0** → **HYPOTHESIS NOT SUPPORTED.** Ship "NOT SUPPORTED" template. The −10pp magnitude was not demonstrated; the mechanism is shipped (v1.7.4); the hypothesis is unsupported.
- **Δ < 0 with upper-CI < 0** = goal-stack measurably HARMS performance → flag for v1.8 design review.

## Stop conditions (mid-run)

- **C2 sanity gate (post-v1.7.6 P0-2 + post-review P0-3 recalibrated for N=4 lattice):** after C2 completes, apply this stricter rule: `mean ∈ [0.05, 0.50]` AND `≥3 distinct seeds non-zero`. If FAIL, **STOP**. Do NOT run C3. Apply SANITY_FAIL template. Retarget to v1.8 adversarial categories per pre-committed escalation. C3 (goal-stack ON) is NOT collected on SANITY_FAIL — no goal-stack data leak.

  *Lattice math:* per-seed late rate at N=4 lives on `{0, 0.25, 0.5, 0.75, 1.0}`. Aggregate over 20 seeds: `lateMean = k/(20×4) = k/80`. The inherited `[4%, 24%]` band (calibrated from a README 14% headline measured at full-late=last 7) is replaced because (a) the band is meaningless on the N=4 lattice (the 4% lower bound corresponds to ~3 hits across all 80 seed-slots, easily satisfied by a single high-rate seed); (b) `[0.05, 0.50]` is a wider headroom band that ensures discrimination room for a ≥10pp drop; (c) the `≥3 distinct seeds non-zero` requirement rules out the lone-outlier failure mode.

- **Hook failure gate (post-C3):** if `C3.hookFailures.push > 0` OR `.complete > 0`, run is contaminated. STOP, fix root cause, restart C3 with fresh result dir.
- **Seed crash policy:** drop the seed across ALL conditions (paired); replacement seed indexed 1020+. No cherry-picking.
- **Outlier policy:** any single seed >3σ from the rest in either condition → investigate and document in result doc.

## Pre-committed v1.8 escalation constraints (POST-REVIEW P1-8)

If v1.7.7 returns SANITY_FAIL or NOT_SUPPORTED, v1.8 ships adversarial trap categories with these PRE-COMMITTED constraints (named here BEFORE running v1.7.7):

- **At least 3 new trap categories** beyond the existing 10 (TRAP_CATEGORIES in `traps.mjs`).
- **Lessons must use vocabulary that has <40% Jaccard overlap with v1.7.5 lessons** (BM25 should NOT trivially match).
- **Target C2 band: `mean ∈ [0.05, 0.50]` AND `≥3 distinct seeds non-zero`** (same rule as v1.7.7 — pre-committed discrimination check, not a README-headline match).
- **Same C2-before-C3 sanity preflight rule** (no goal-stack data collected on SANITY_FAIL).
- **Categories must be authored BEFORE running C3** (no choosing-after-seeing-results).
- **All 4 conditions × 20 seeds**, hash-derived as before.

This pre-commitment exists so v1.8 cannot be retroactively justified by v1.7.7's results.

## Pre-committed report templates

### Template "HYPOTHESIS SUPPORTED" (use IFF the decision rule selects it)

> **Goal-stack lift on sequential-learning benchmark — HYPOTHESIS SUPPORTED.** v1.7.7 restricted-late workload (`--budget 2000 --restrict-late-to 4`), 20 paired seeds, 10k Monte Carlo permutation CI. Late-phase trap-hit-rate dropped by Δ = {DELTA}pp (95% CI [{CI_LOW}pp, {CI_HIGH}pp]). Goal-stack ON: late mean = {C3_LATE} (CI ±{C3_CI}pp). Goal-stack OFF: late mean = {C2_LATE} (CI ±{C2_CI}pp). Sanity anchors — none: {C0_LATE}, static: {C1_LATE}. C2 sanity preflight: {C2_PREFLIGHT_PASS_FAIL} (C2 late mean = {C2_LATE} ∈ [5%, 50%], {C2_SEEDS_NON_ZERO} of 20 seeds non-zero ≥ 3 — pre-registered N=4 lattice rule per "Stop conditions"; band corrected post-audit v1.7.8). Pre-registration: docs/evals/2026-05-09-v1.7.7-goal-stack-eval-prereg.md.

### Template "HYPOTHESIS NOT SUPPORTED" (use IFF the decision rule selects it — NO SOFT FRAMING + cumulative evidence + cliff humility)

> **Goal-stack lift on sequential-learning benchmark — HYPOTHESIS NOT SUPPORTED.** v1.7.7 restricted-late workload (`--budget 2000 --restrict-late-to 4`), 20 paired seeds, 10k Monte Carlo permutation CI {plus exact enumeration if `|Δ - 0.10| < 0.02` or `|ciLow| < 0.02`}. Late-phase Δ = {DELTA}pp (95% CI [{CI_LOW}pp, {CI_HIGH}pp]) does not meet the pre-registered ≥10pp threshold with lower-CI bound above zero. Goal-stack ON: late mean = {C3_LATE} (CI ±{C3_CI}pp). Goal-stack OFF: late mean = {C2_LATE} (CI ±{C2_CI}pp). The −10pp magnitude is not demonstrated. Tie-degeneracy: {tiePass=true|false}. Paired-diff histogram and exact-enumeration result (if applicable) in the result doc.
>
> **{include if `|Δ - 10pp| < 1pp`:}** Δ landed within 1pp of the threshold ({DELTA}pp vs 10pp). Per the pre-reg cliff rule, treated as null. Result is on the threshold; readers should weight the discrete-distribution sample noise accordingly.
>
> **Cumulative evidence (post-review P0-4):** This is the third pre-registered workload variant in which the −10pp magnitude was not demonstrated (v1.7.5: SANITY_FAIL on full-late saturation; v1.7.6: B*=NULL across 5 budgets × 10 seeds; v1.7.7: Δ={DELTA}pp on `--restrict-late-to 4`). Per the v1.7.7 prereg's hard-stop clause, the magnitude is now retracted from public-surface claims; the goal-stack mechanism remains shipped from v1.7.4 with no specific magnitude attached. v1.8 (adversarial trap categories) ships as a new hypothesis, not a continuation of this one.
>
> Pre-registration: docs/evals/2026-05-09-v1.7.7-goal-stack-eval-prereg.md.

### Template "SANITY_FAIL" (use IFF C2 preflight fails)

> **Goal-stack lift on sequential-learning benchmark — eval STOPPED per pre-registered C2 sanity gate.** v1.7.7 restricted-late workload (`--budget 2000 --restrict-late-to 4`), 20 seeds. C2 hippo-base late mean = {C2_LATE} ({k of 20 seeds non-zero}), failing the pre-registered N=4 lattice gate (`mean ∈ [0.05, 0.50]` AND `≥3 distinct seeds non-zero`). Floor effect persists at last-4 — narrowing the metric did not produce a discriminating workload. **C3 (goal-stack ON) was NOT collected** — no goal-stack data was produced under SANITY_FAIL. Hypothesis remains untested on a benchmark variant. Per pre-committed escalation (v1.7.6 plan P2-4 + v1.7.7 prereg): v1.8 will explore adversarial trap categories with the constraints named in the v1.7.7 prereg's "Pre-committed v1.8 escalation constraints" section. Calibration: docs/evals/2026-05-09-v1.7.6-calibration-result.md. Pre-registration: docs/evals/2026-05-09-v1.7.7-goal-stack-eval-prereg.md.

## Audit trail

- Plan: `docs/plans/2026-05-09-v1.7.7-restricted-late-window.md`
- Code: branch `feat/v1.7.7-restricted-late-window` commits to be filled in.
- Outside-voice review on plan: senior-code-reviewer + codex (gpt-5.5 thinking high) — to run before /full-power execute.
- v1.7.6 forward-pointer: `docs/evals/2026-05-09-v1.7.6-calibration-result.md` (Update appended).
- v1.7.5 forward-pointer (already present): `docs/evals/2026-05-07-v1.7.5-goal-stack-eval-result.md`.
- Claim inventory: `docs/evals/2026-05-09-v1.7.7-claim-inventory.md` (committed alongside this prereg).
