# v1.8.0 Adversarial-Categories Eval — Pre-Registration

**Pre-registered:** 2026-05-09, BEFORE running C2 sanity preflight or C3.
**Author:** Keith So
**Mechanism under test:** B3 dlPFC goal-stack boost (`applyGoalStackBoost`), shipped v1.7.4.
**Workload variant:** v1.8.0 adversarial trap categories. 13 categories (10 v1.7.x + 3 adversarial: timezone_naive, idempotency_retry, float_accumulation), 31 trap encounters across 62 tasks. `--budget 2000`, `--restrict-late-to 4`.

**This release does not re-assert the retracted −10pp magnitude.** Per `docs/RETRACTION.md`, this is mechanism characterisation only.

## Operational definition of "characterise" (per RETRACTION.md guard)

For v1.8 specifically, "characterise the mechanism" means producing exactly these three reports:

1. **Sanity verdict (PASS/FAIL).** From the C2 sanity preflight at `mean ∈ [0.05, 0.50] AND ≥3 distinct seeds non-zero`. This is a **workload-validity / non-saturation check**. NOT a magnitude criterion.
2. **C3 per-seed lattice histogram.** For the 20 seeds, count how many landed at each lattice value `{0, 0.25, 0.5, 0.75, 1.0}`.
3. **Sign-only seed-pair direction count vs C2.** For each of 20 paired seeds, classify (C3 vs C2) as one of: `STRICTLY_LOWER` (C3 < C2), `STRICTLY_HIGHER` (C3 > C2), `TIED` (C3 = C2).

ANY other report (pp values, deltas, medians, means juxtaposed, "improvement", "reduction") is forbidden by RETRACTION.md.

## Categories authored BEFORE C3 ran (audit anchor)

The 3 adversarial categories are committed at:
- Branch: `feat/v1.8.0-adversarial-categories`
- Task 1 commit SHA: `f295e53` (`feat(benchmark): add 3 adversarial trap categories with uniform positions (v1.8.0 pre-reg lock)`)
- Pilot smoke test commit SHA: `b2a6ad5` (`docs(benchmark): v1.8 pilot smoke test (seed=42, late=25%) + category authoring iteration log`)
- Jaccard verification commit SHA: `c7e9e0d` (`tools(benchmark): jaccard-overlap with Porter-stem, threshold <0.30 (max=0.033 PASS, v1.8.0)`)
- BM25-mismatch test commit SHA: `2cad9c0` (`tests(benchmark): bm25-mismatch sim for adversarial categories (v1.8.0)`)

This prereg references those SHAs explicitly. C2 preflight + C3 run will happen AFTER this prereg is committed.

## Category authoring iteration log

See `docs/evals/2026-05-09-v1.8.0-category-authoring-iteration-log.md`. Iteration 1 ACCEPTED on the basis of out-of-band pilot (seed=42, hippo only): late = 25%. Non-saturated. No replacements needed.

## Hypothesis (NO MAGNITUDE)

**This release does not test a magnitude hypothesis.** Per `docs/RETRACTION.md`, the −10pp magnitude is RETRACTED. v1.8 reports descriptive characterisation only.

The two questions v1.8 answers:
1. Does the v1.8 adversarial workload satisfy the workload-validity / non-saturation gate (`mean ∈ [0.05, 0.50] AND ≥3 distinct seeds non-zero`)?
2. If YES, what is the **directional** comparison of C3 (goal-stack ON) vs C2 (goal-stack OFF) on per-seed adversarial-late rates? (Reported as sign-only direction counts + C3 lattice histogram. NOT magnitude.)

If sanity FAILs, only question 1 is reported. C3 not collected. No comparison possible. Per the **Pre-committed v1.9 direction** below, v1.9 retargets to LongMemEval R@5 cross-validation.

## Pre-committed constraints (inherited from v1.7.7 prereg lines 79-87)

| # | Constraint | Status |
|---|------------|--------|
| 1 | At least 3 new trap categories beyond the existing 10 | ✓ 3 categories added (timezone_naive, idempotency_retry, float_accumulation) at Task 1 SHA `f295e53` |
| 2 | Lessons use vocabulary with **<0.30 Jaccard** (TIGHTENED from 0.40) | ✓ Verified: max Jaccard = 0.033 (timezone_naive ↔ overwrite_production), see `docs/evals/2026-05-09-v1.8.0-jaccard-verification.txt` |
| 3 | Target C2 band: `mean ∈ [0.05, 0.50] AND ≥3 distinct seeds non-zero` | TBD: tested at Task 7 below |
| 4 | Same C2-before-C3 sanity preflight (no goal-stack data on SANITY_FAIL) | ✓ Hard-stop in plan |
| 5 | Categories authored BEFORE running C3 | ✓ Task 1 SHA precedes C2/C3 runs |
| 6 | All 4 conditions × 20 seeds, hash-derived (1000+i for i in 0..19) | ✓ Same `Math.imul(0x9E3779B9, 1000+i) >>> 0` |

**No constraint relaxation. No category swap-in / swap-out after seeing C2 results. No vocabulary tuning after seeing recall behaviour.**

## Workload definition

- Total tasks: **62** (`N_TASKS` in `traps.mjs`).
- Trap categories: **13** (10 v1.7.x + 3 adversarial).
- Total trap encounters: **31** (25 v1.7.x + 6 adversarial).
- Adversarial positions (FIXED, no shape-group shuffle): timezone_naive [11, 41], idempotency_retry [21, 51], float_accumulation [33, 59].
- Existing positions (shape-group-shuffled by seed): unchanged from v1.7.x.
- **Last 4 trap encounters by chronology: positions 48, 50, 51, 59 — 2 existing + 2 adversarial. MIXED slice, NOT pure-adversarial.**

## Position-distribution rationale (per outside voice B2)

New positions {11, 21, 33, 41, 51, 59} are distributed UNIFORMLY across early/mid/late regions of the 62-task sequence:
- Early (1-22): 2 new traps (positions 11, 21)
- Mid (22-42): 2 new traps (positions 33, 41)
- Late (42-62): 2 new traps (positions 51, 59)

**NOT clustered in late.** Late-region density: 1 existing + 2 new = 3 traps in 13 slots = 23%, lower than the existing-region density of 50%. This avoids optimising the workload for magnitude visibility.

## Subset reporting lock (per outside voice A5)

**Pre-registered:** the v1.8 result reports **late-phase only** (`--restrict-late-to 4`). NO subset slicing (e.g. existing-10 only / new-3 only). NO per-version cross-cite to v1.7.5/6/7 late-phase rates (different workload; not comparable). Reporting locks before the eval runs to prevent post-hoc cherry-picking.

## Protocol

- Benchmark: `benchmarks/sequential-learning` (v1.7.5+ multi-seed harness).
- **Workload knob:** `--budget 2000`, `--restrict-late-to 4`.
- **Conditions (4):** C0 none, C1 static, C2 hippo-base (`--use-goal-stack` OFF), C3 hippo+goalstack (`--use-goal-stack` ON).
- **Hypothesis seeds: 20.** Hash-derived in `run.mjs` (`Math.imul(0x9E3779B9, 1000+i) >>> 0` for i in 0..19) — same as v1.7.5/6/7.
- **Eval mode:** `--eval-strict`.
- **Primary metric:** late-phase trap-hit-rate over the LAST 4 trap encounters per seed. Slice math (Option A): late = trapTasks.slice(-4), early/mid re-split. With 31 trap encounters at N=4: `early = first ceil((31-4)/2) = 14`, `mid = next 13`, `late = last 4`. Disjoint and exhaustive.
- **Top-5 cap interaction:** unchanged from v1.7.5 — a task is "trap hit" iff `isRecalled(top5, category) === false`.

## Sanity gate (workload-validity / non-saturation check, NOT magnitude)

Per the v1.7.7 N=4 lattice rule, applied here as a workload-validity check (not a magnitude claim):

- Sanity PASS: `lateMean ∈ [0.05, 0.50]` AND `≥3 distinct seeds non-zero`.
- Sanity FAIL: outside the band OR <3 distinct seeds non-zero. STOP. Do NOT run C3. No goal-stack data leak.

*Lattice math:* per-seed late rate at N=4 lives on `{0, 0.25, 0.5, 0.75, 1.0}`. Aggregate over 20 seeds: `lateMean = k/(20×4) = k/80`.

## Stop conditions (mid-run)

- **C2 sanity gate** as above. Hard-stop on FAIL. Apply SANITY_FAIL template + retarget to v1.9 LongMemEval per pre-committed direction.
- **Hook failure gate (post-C3):** if `C3.hookFailures.push > 0` OR `.complete > 0`, run is contaminated. STOP, fix root cause, restart C3 with fresh result dir.
- **Seed crash policy:** drop the seed across ALL conditions (paired); replacement seed indexed 1020+. No cherry-picking.
- **Outlier policy:** any single seed >3σ from the rest in either condition → investigate and document in result doc.

## Pre-committed v1.9 direction (named BEFORE v1.8 runs — per outside voice C1)

If v1.8 SANITY_FAILs, the pre-committed escalation chain over **workload knobs in the sequential-learning benchmark** is exhausted. v1.9+ requires a fundamentally different direction.

**Named NOW, before v1.8 results exist:** v1.9 will run the dlPFC goal-stack mechanism on the **LongMemEval R@5 corpus** (already named in v1.7.7 prereg line 23 as a separate eval line), as a cross-validation of the mechanism on a fundamentally different benchmark:
- Different metric: R@5 not trap-rate.
- Different corpus: 500-question public vs internal sequential learning.
- Different mechanism stress: retrieval-on-fixed-corpus vs agent-improvement-over-time.

This pre-commitment exists so the v1.8 SANITY_FAIL outcome cannot be retroactively justified by choosing a convenient v1.9 direction afterwards. v1.9 PRE-REG will be drafted before v1.9 implementation begins (separate plan).

## Pre-committed result template (use verbatim, fill in the {SLOTS})

The result doc MUST follow this structure verbatim. NO magnitude framing in any non-allowlisted line.

```text
> **Mechanism characterisation on v1.8 adversarial trap categories — descriptive report.** v1.8 workload (`--budget 2000 --restrict-late-to 4`, 13 categories including 3 new adversarial), 4 conditions × 20 paired seeds, `--eval-strict`, hash-derived seeds (1000+i for i in 0..19). **This release does not re-assert the retracted −10pp magnitude.** Per `docs/RETRACTION.md`, this is mechanism characterisation only.

## Workload-validity sanity verdict

**{PASS or FAIL}** — C2 hippo-base lateMean = {C2_LATE} (lattice rate), {C2_SEEDS_NON_ZERO} of 20 seeds non-zero. Pre-registered N=4 lattice rule from v1.7.7: `mean ∈ [0.05, 0.50] AND ≥3 distinct seeds non-zero`. Framed as workload-validity / non-saturation check per `docs/RETRACTION.md`, NOT a magnitude criterion.

## Workload baseline (C0, C1, C2)

- C0 none: late mean = {C0_LATE} (95% CI ± {C0_CI}).
- C1 static: late mean = {C1_LATE} (95% CI ± {C1_CI}).
- C2 hippo-base: late mean = {C2_LATE} (95% CI ± {C2_CI}).

> **Do not subtract across sections.** Per `docs/RETRACTION.md`, magnitude differences between conditions are not reported in this release.

## C3 mechanism characterisation (only if sanity verdict = PASS)

C3 per-seed lattice histogram (over 20 seeds at adversarial-late N=4):

| 0/4 | 1/4 | 2/4 | 3/4 | 4/4 |
|-----|-----|-----|-----|-----|
| {k0} | {k1} | {k2} | {k3} | {k4} |

Sign-only seed-pair direction count (vs C2):
- STRICTLY_LOWER (C3 < C2): {N_LOWER}
- STRICTLY_HIGHER (C3 > C2): {N_HIGHER}
- TIED (C3 = C2): {N_TIED}

Hook failures: push={N_PUSH}, complete={N_COMPLETE}. Tie-degeneracy: tiePass={true|false}.

> **Do not subtract across sections.** Per `docs/RETRACTION.md`, magnitude differences between conditions are not reported in this release.

## Outside-voice questions answered

[Verbatim answers to the 5 outside-voice questions named below in the prereg's "Outside voice questions" subsection. Each question gets 1-2 sentences. NO magnitude framing.]

## Reproducibility anchors

[hippo --version, git rev-parse HEAD, node --version, run.mjs blob SHA, traps.mjs blob SHA (Task 1 commit), tools/jaccard-overlap.mjs blob SHA, workload knobs, run timestamps, etc.]
```

## Outside voice questions (must answer verbatim in result.md per outside voice D3)

1. Are the 3 chosen categories sufficiently adversarial (per pilot smoke test outcome)?
2. Did uniform position distribution actually hold in the executed task generation?
3. Did the descriptive characterisation framing hold across all artifacts (prereg + inventory + result + CHANGELOG + README + RETRACTION update)?
4. Does the workload-validity verdict (PASS or FAIL) inform v1.9 direction (is LongMemEval still the right pre-committed next step)?
5. Did Task 13 staged-diff outside voice find any RETRACTION.md compliance leaks?

## SANITY_FAIL template (use IFF C2 sanity gate fires)

```text
> **Mechanism characterisation on v1.8 adversarial trap categories — eval STOPPED per pre-registered C2 sanity gate.** v1.8 workload (`--budget 2000 --restrict-late-to 4`, 13 categories), 20 seeds. C2 hippo-base lateMean = {C2_LATE} (lattice rate), {C2_SEEDS_NON_ZERO} of 20 seeds non-zero. Pre-registered N=4 lattice rule failed: NOT in `[0.05, 0.50]` OR <3 distinct seeds non-zero. **Workload-validity / non-saturation FAIL.** **C3 (goal-stack ON) was NOT collected.** No goal-stack data was produced under SANITY_FAIL. Per pre-committed v1.9 direction (this prereg's "Pre-committed v1.9 direction" subsection): v1.9 will run the goal-stack mechanism on the LongMemEval R@5 corpus as a fundamentally different benchmark cross-validation. **This release does not re-assert the retracted −10pp magnitude.** Per `docs/RETRACTION.md`, mechanism remains shipped; no magnitude is currently claimed. Pre-registration: `docs/evals/2026-05-09-v1.8.0-adversarial-eval-prereg.md`.
```

## Audit trail

- Plan: `docs/plans/2026-05-09-v1.8.0-adversarial-categories.md`
- Code: branch `feat/v1.8.0-adversarial-categories` commits `f295e53`, `b2a6ad5`, `c7e9e0d`, `2cad9c0`, plus this prereg commit.
- Outside-voice review on plan: senior-code-reviewer × 2 (in-house arch + adversarial RETRACTION-compliance proxy) — completed 2026-05-09. 22 findings, all accepted.
- Iteration log: `docs/evals/2026-05-09-v1.8.0-category-authoring-iteration-log.md`.
- Jaccard verification output: `docs/evals/2026-05-09-v1.8.0-jaccard-verification.txt`.
- BM25-mismatch sim test: `tests/sl-bm25-mismatch.test.ts`.
- Result (post-eval): `docs/evals/2026-05-09-v1.8.0-adversarial-eval-result.md`.

## RETRACTION.md compliance declaration

This prereg, the result doc, the CHANGELOG entry, the README "What's new in v1.8.0" block, and the `docs/RETRACTION.md` v1.8.0 outcome subsection ALL contain the verbatim sentence: **"This release does not re-assert the retracted −10pp magnitude."** All artifacts reference `docs/RETRACTION.md`. Magnitude grep with retraction-citation allowlist will run as Task 13 staged-diff outside voice gate.
