# v1.8.0 Adversarial-Categories Eval — Result (Descriptive Characterisation)

**Run:** 2026-05-09
**Pre-registration:** `docs/evals/2026-05-09-v1.8.0-adversarial-eval-prereg.md`
**Claim inventory:** `docs/evals/2026-05-09-v1.8.0-claim-inventory.md`
**Categories authored at commit:** `f295e53` (BEFORE C2/C3 ran).

> **Mechanism characterisation on v1.8 adversarial trap categories — descriptive report.** v1.8 workload (`--budget 2000 --restrict-late-to 4`, 13 categories including 3 new adversarial), 4 conditions × 20 paired seeds, `--eval-strict`, hash-derived seeds (1000+i for i in 0..19). **This release does not re-assert the retracted −10pp magnitude.** Per `docs/RETRACTION.md`, this is mechanism characterisation only.

## Workload-validity sanity verdict

**PASS** — C2 hippo-base lateMean = 0.25 (lattice rate; 25% of late-4 trap-hits aggregated across 20 seeds), 20 of 20 seeds non-zero. Pre-registered N=4 lattice rule from v1.7.7: `mean ∈ [0.05, 0.50] AND ≥3 distinct seeds non-zero` — both conditions satisfied. Framed as workload-validity / non-saturation check per `docs/RETRACTION.md`, NOT a magnitude criterion.

The v1.8 adversarial workload is non-saturated (every seed produces 1/4 late trap-hits, not 0/4). This is the FIRST time across v1.7.5/6/7/8 that the C2 sanity gate has passed — confirming that adversarial categories produce a discriminating workload at the metric level (workload-validity sense, not magnitude).

## Workload baseline (C0, C1, C2)

- C0 none: late mean = 1.000 (95% CI ± 0.000) — workload no-memory baseline.
- C1 static: late mean = 0.350 (95% CI ± 0.0588) — pre-loaded memory baseline.
- C2 hippo-base: late mean = 0.250 (95% CI ± 0.000) — sanity-gate input.

> **Do not subtract across sections.** Per `docs/RETRACTION.md`, magnitude differences between conditions are not reported in this release.

## C3 mechanism characterisation (sanity = PASS, so C3 reported)

### C3 per-seed lattice histogram (over 20 seeds at adversarial-late N=4)

| 0/4 | 1/4 | 2/4 | 3/4 | 4/4 |
|-----|-----|-----|-----|-----|
| 0 | 20 | 0 | 0 | 0 |

All 20 seeds produced exactly 1 trap-hit out of the last 4 trap encounters (0.25 lattice rate). Zero variance across seeds in the C3 condition.

### Sign-only seed-pair direction count (vs C2)

- STRICTLY_LOWER (C3 < C2): **0**
- STRICTLY_HIGHER (C3 > C2): **0**
- TIED (C3 = C2): **20**

Hook failures: push=0, complete=0.

Tie-degeneracy: tiePass = false (all 20 paired diffs are zero — degenerate).

> **Do not subtract across sections.** Per `docs/RETRACTION.md`, magnitude differences between conditions are not reported in this release.

## Outside-voice questions answered

**1. Are the 3 chosen categories sufficiently adversarial (per pilot smoke test outcome)?**
Yes. Pilot at seed=42 produced late=25% (out-of-band, hippo-only). Formal 20-seed run reproduced 25% across every seed. Categories are non-saturated against hippo-base; the workload produces meaningful late-phase trap-hit signal that the v1.7.x workload could not produce.

**2. Did uniform position distribution actually hold in the executed task generation?**
Yes. New positions {11, 21, 33, 41, 51, 59} land 2 in early region (1-22), 2 in mid (23-42), 2 in late (43-62). Adversarial categories use fixed positions (no shape-group shuffle), preserving v1.7.x existing-10 placements unchanged. Last 4 trap encounters by chronology: positions 48, 50, 51, 59 — 2 existing + 2 adversarial (mixed, not pure-adversarial).

**3. Did the descriptive characterisation framing hold across all artifacts?**
Pending Task 13 staged-diff outside voice (post-publish gate). Pre-publish self-grep (this result doc + prereg + claim inventory + plan + traps.mjs + tools/ + tests/) returned only retraction-citation lines — all allowlisted per the "retracted within 10 chars" rule. CHANGELOG / README / RETRACTION.md updates (Task 10) will be re-checked.

**4. Does the workload-validity verdict (PASS or FAIL) inform v1.9 direction (is LongMemEval still the right pre-committed next step)?**
The verdict PASSED on workload-validity — adversarial categories produce a non-saturated late-phase rate. That is informative for v1.9 design: the sequential-learning benchmark CAN be made discriminating; the question for v1.9 is whether the same applies on a different metric / corpus. LongMemEval R@5 cross-validation remains the pre-committed v1.9 direction; the v1.8 PASS verdict does not change that pre-commitment.

**5. Did Task 13 staged-diff outside voice find any RETRACTION.md compliance leaks?**
Pending — to be answered after Task 13 runs (post-Task 12 test suite, pre-Task 14 publish).

## Reproducibility anchors

- `hippo --version` = `1.7.9` (v1.8.0 not yet bumped at run time)
- `git rev-parse HEAD (start)` = post-`494e056` (Task 6 commit)
- `node --version` = `v24.13.0`
- `npx vitest --version` = `vitest/1.6.1 win32-x64 node-v24.13.0`
- `traps.mjs` blob SHA = part of commit `f295e53` (Task 1: categories authored)
- `aggregate.mjs` blob SHA = post-v1.7.9 P2-3 docstring commit
- `tools/jaccard-overlap.mjs` blob SHA = part of commit `c7e9e0d`
- Workload knobs: `--budget 2000 --restrict-late-to 4 --eval-strict`
- Run timestamps:
  - C2 start = 2026-05-09T~17:18Z, end = ~17:25Z, wall-time = ~7m 32s (452s)
  - C0 = ~5s
  - C1 = ~5s
  - C3 start = ~17:25Z, end = ~17:36Z, wall-time = ~10m 40s (640s)

## Cumulative evidence chain across v1.7.5/6/7/8

| Release | Workload knob | Outcome (workload-validity) | Mechanism comparison (sign-only) |
|---------|---------------|------------------------------|----------------------------------|
| v1.7.5 | full-late = last 7 of 25 (chronological-third) | C2 SANITY_FAIL — saturation | N/A (gate failed) |
| v1.7.6 | budget sweep {200, 400, 600, 800, 1000} | B*=NULL — workload floor | N/A (no B* to use) |
| v1.7.7 | `--restrict-late-to 4` (last 4 of 25) | C2 SANITY_FAIL — N=4 lattice gate | N/A (sanity preflight blocked C3) |
| v1.7.9 | (retraction release) | −10pp magnitude **RETRACTED** publicly | N/A (no eval) |
| **v1.8.0** | **adversarial categories (last 4 of 31)** | **C2 PASS — first non-saturated workload** | **C3 = C2 on all 20 seeds (0/0/20 lower/higher/tied)** |

This release does not re-assert the retracted −10pp magnitude. The dlPFC goal-stack mechanism (`pushGoal`/`completeGoal`, `--use-goal-stack`, `applyGoalStackBoost`) remains shipped from v1.7.4. v1.8 reports descriptive characterisation only.

The C3 = C2 sign-only result (0/0/20) is descriptive evidence that the goal-stack boost does not detectably change the per-seed late-4 lattice rate on this workload at this scale. This is workload-and-metric-specific, not a general claim about the mechanism's effect.

## Per pre-committed v1.9 direction

The pre-committed v1.9 direction (named in this v1.8 prereg's "Pre-committed v1.9 direction" subsection BEFORE v1.8 ran) is **LongMemEval R@5 cross-validation**. The v1.8 PASS verdict does not change that pre-commitment.

## Audit trail

- Plan: `docs/plans/2026-05-09-v1.8.0-adversarial-categories.md`
- Branch: `feat/v1.8.0-adversarial-categories`
- Commits: `88cccf4` (plan), `f295e53` (categories), `b2a6ad5` (pilot + iteration log), `c7e9e0d` (Jaccard tool + verification), `2cad9c0` (BM25-mismatch test), `fe4a53a` (prereg), `13d2d5f` (claim inventory), `494e056` (workload tests).
- C2 result JSON (gitignored): `benchmarks/sequential-learning/results/v1.8-eval-C2-hippo-base/benchmark-1778343702387.json`
- C0 result JSON: `benchmarks/sequential-learning/results/v1.8-eval-C0-none/benchmark-1778343804131.json`
- C1 result JSON: `benchmarks/sequential-learning/results/v1.8-eval-C1-static/benchmark-1778343809166.json`
- C3 result JSON: `benchmarks/sequential-learning/results/v1.8-eval-C3-hippo-goalstack/benchmark-1778344454767.json`
- To re-derive: re-run with the pre-registered workload knobs and same hash-derived seeds (1000+i for i in 0..19); deterministic.
