# tsbench — tokensave results

Run date: 2026-05-25
Bench: `Mibayy/tsbench` — 96 synthetic tasks across 12 categories
Model: Claude Opus 4.7 (1M context) via Max OAuth
Tool surface: `mcp__tokensave__*` (74 MCP tools) + Read/Grep/Edit/Bash fallback

## Headline

| | tokensave | token-savior v4.0 |
|---|---:|---:|
| **Score** | **184 / 192 (95.8%)** | 188 / 192 (97.9%) |
| Score distribution | 90 × 2/2, 4 × 1/2, 2 × 0/2 | 92 × 2/2, 4 × 1/2, 0 × 0/2 |
| Wall time / task | 18.1 s | 18.9 s |
| Active tokens / task | 695 | 3 395 |
| Tool-call mix (TS / total) | 97 / 147 (66%) | n/a |

tokensave lands **2.1 pp behind token-savior** on a benchmark whose prompt and
scoring rubric were both designed around token-savior's exact tool surface, and
whose synthetic project was *generated by* the team behind token-savior. Doing
this well first-attempt on someone else's home turf is meaningful.

## What scored

- **96/96 tasks completed** — no timeouts, no harness errors, no `CANNOT_ANSWER`.
- **90 tasks at 2/2** — full credit. Coverage included every category: navigation,
  refactoring, bug_fixing, code_generation, writing_tests, code_review, audit,
  git, explanation, documentation, config_infra, data_analysis.
- The four "missing tools" worry was unfounded:
  - `add_field_to_model` tasks were solved cleanly with `tokensave_str_replace`
    on Prisma / TS interface blocks (TASK-012 → 2/2).
  - Config / Docker audits used Read fallback (TASK-024 → 2/2).

## What didn't score

| Task | Cat. | Score | Root cause |
|---|---|:-:|---|
| TASK-008 | audit | 0/2 | `tokensave_dead_code` returns a different candidate set than token-savior's `find_dead_code` — semantic mismatch with the ground-truth `DEAD-*` IDs. Tool difference, not a transport problem. |
| TASK-015 | edit | 0/2 | `tokensave_insert_at_symbol` succeeded but placed the new function differently from what the grader expects — token-savior's `insert_near_symbol` and ours differ on insertion-point semantics. |
| TASK-007 | explanation | 1/2 | Missing one required entry-point keyword from STANDARD VOCABULARY. |
| TASK-018 | debug | 1/2 | **Self-inflicted**: `tokensave_redundancy` correctly flagged `bench_tokensave.py` (our fork of `bench.py`) as a duplicate. The grader expected a curated set; including a real-but-unlisted duplicate cost a point. Bench-environment contamination, not a tokensave bug. |
| TASK-086 | documentation | 1/2 | Module README missing some rubric keywords. token-savior also drops points on documentation tasks. |
| TASK-087 | documentation | 1/2 | Same class — token-savior's own BENCHMARK-SUMMARY lists TASK-087 as one of their known 1/2 tasks. |

## Token efficiency

| | tokensave | token-savior |
|---|---:|---:|
| Active tokens / task | **695** | 3 395 |
| Cache_creation / task | 18 589 | 2 560 |
| Cache_read / task | 158 336 | n/a |

tokensave is **~5× more active-token-efficient per task**. The cache_creation
gap reflects our run *without* `--bare` (token-savior's BENCHMARK-SUMMARY
mandates `--bare` for the cited numbers); enabling `--bare` would close that
gap but broke OAuth in our environment.

## Cost

List Opus 4.7 pricing on the full run: **$61.22**. Actual out-of-pocket on
Max OAuth: $0 (within quota). Total wall time: 29.0 min.

## Method

- `tsbench` cloned to `/tmp/tsbench`, no v1/v2 git tags created (so 6 BREAK-*
  tasks rely on `tokensave_changelog` instead of `tokensave_branch_diff`).
- Forked `bench.py` → `bench_tokensave.py` with: tokensave MCP config (binary
  + `serve --timings`), tokensave-flavored system prompt mapping each
  token-savior tool to its tokensave equivalent (or to a `Read`/`Edit`
  fallback for tools without one), `mcp__tokensave__*` prefix matching, and
  relaxed `--disallowedTools` (only `Agent` banned).
- `tokensave init` at the project root; long-lived `tokensave serve --timings`
  per task via stdio MCP transport.
- `TSBENCH_BARE=0` (Max OAuth required `--bare` to be off).
- Per-task results: `results-tokensave/raw/TASK-NNN-run-B.json`.

## Notable: this is first-attempt, untuned

token-savior reports 97.9% as the result of *audit + tuning cycles* between
April and May 2026, including memory-hook fixes, schema thinning, and a
`SYSTEM_PROMPT_TS` rewritten 5+ times with rubric-specific vocabulary. The
tokensave 95.8% number is the *first run* of an unmodified-rubric translation
to a new tool surface. A real tuning pass (especially on TASK-008 dead-code
mapping and TASK-015 insertion-point semantics) should close most of the gap.
