kb-arena开源AI工作流

Prerequisites

1. Python 3.11+ and pip
2. Docker (for Neo4j — the knowledge graph strategy needs it)
3. API keys for your LLM provider (Anthropic, OpenAI, or Ollama) and OpenAI (embeddings)

That's it. No Neo4j expertise needed. No graph database experience required. KB Arena handles the schema, extraction, and querying.

Install with dev dependencies

pip install -e '.[dev]'

Faster QnA Index Building

Q&A pair generation during build-vectors is now parallelized with asyncio.gather() (5 concurrent). Building QnA indexes on large corpora is up to 5x faster.

Step 1: Install

```bash pip install kb-arena

Optional: install format-specific parsers

pip install kb-arena[pdf] # PDF support (PyMuPDF) pip install kb-arena[docx] # Word documents (mammoth) pip install kb-arena[web] # Web scraping (httpx) pip install kb-arena[all-formats] # All of the above ```

Build the knowledge graph in Neo4j (entities + relationships)

kb-arena build-graph --corpus my-docs

Build vector indexes in ChromaDB (local, no server needed)

kb-arena build-vectors --corpus my-docs

Full Stack with Docker Compose

Run everything — Neo4j, the API server, and the frontend — in one command:

```bash

No-API-Keys Quick Start (Ollama)

```bash

Quick Start -- I Just Have My Docs

You have documentation files (markdown, HTML, text, PDFs). You want to know which retrieval strategy works best. Here's everything from zero.

Using the Built-in AWS Example

The AWS Compute corpus ships ready to use (75 questions across 5 difficulty tiers):

kb-arena ingest ./datasets/aws-compute/raw/ --corpus aws-compute
kb-arena build-graph --corpus aws-compute
kb-arena build-vectors --corpus aws-compute
kb-arena benchmark --corpus aws-compute
kb-arena label-chunks --corpus aws-compute        # v0.5.0: ground truth for IR metrics
kb-arena retriever-lab --corpus aws-compute       # v0.5.0: classical IR metrics, no LLM cost
kb-arena serve

---

Demo polish

• kb-arena demo truly zero-config — LLMClient init is tolerant of missing keys, demo_mode auto-enables, dashboard loads instantly
• aws-compute_bm25.json is bundled (was missing in v0.5.0 — the 8th strategy showed empty in fresh installs)
• README hero rewritten with the question-frame pitch, plus a No-API-Keys Quick Start using Ollama
• Re-recorded hero GIF + UI walkthrough GIF + retriever-lab CLI GIF, all driven by checked-in vhs tape scripts in docs/tapes/
• kb-arena --version flag

Screenshots

Home — Overview of the 9 strategies, difficulty tiers, and evaluation methodology.

Strategy comparison — Ask the same question to all 9 strategies simultaneously. Compare answers, sources, latency, and cost side-by-side.

Benchmark results — Accuracy table by tier with grouped bar chart.

Knowledge graph — Interactive force-directed visualization of entities extracted from your docs.

Live graph build — Watch entities and relationships stream in as the extractor runs.

---

Environment Variables

All prefixed with KB_ARENA_. Loaded from .env or environment.

---

Try It in 10 Seconds (no API keys)

pip install kb-arena
kb-arena demo

This launches the dashboard with pre-computed results from the AWS Compute corpus (75 questions, 9 strategies, 5 difficulty tiers). The demo runs in read-only mode — chat, arena, and tools endpoints stay disabled until you set an API key. No Docker, no Neo4j, no surprises.

To enable live chat / arena voting / tools, set KB_ARENA_ANTHROPIC_API_KEY (or KB_ARENA_OPENAI_API_KEY, or use KB_ARENA_LLM_PROVIDER=ollama for free local inference).

Preview the search space and cost — no API keys, no corpus prep needed

kb-arena optimize --corpus my-docs \ --strategies naive_vector,rerank_vector \ --top-ks 3,5,10 --chunk-sizes 256,512,1024 \ --embedding-providers openai,bge --dry-run

Reference-Free Evaluation

Benchmark without pre-written ground truth -- useful for quick evaluation of new corpora before investing in question generation.

Scores on faithfulness and answer relevancy only (no accuracy/completeness since there's no reference to compare against).

Debug Endpoint

Trace the full retrieval pipeline -- intent classification, retrieved sources, latency breakdown, and cost -- without generating a final answer.

Step 2: Set API keys

Create a .env file or export directly:

export KB_ARENA_ANTHROPIC_API_KEY=sk-ant-...    # Claude for generation + evaluation
export KB_ARENA_OPENAI_API_KEY=sk-...           # OpenAI for text-embedding-3-large

Set your API keys

export ANTHROPIC_API_KEY=sk-ant-... export OPENAI_API_KEY=sk-...

CLI Reference

All commands are independently re-runnable. Each stage writes to disk so you can re-run any step without repeating earlier ones.

CLI Features

Dry run — Preview what a command will do before committing to expensive LLM calls:

```bash kb-arena ingest datasets/my-docs/raw/ --corpus my-docs --dry-run

Strategy Plugin System

Bring your own retrieval strategy without forking. Your module exports a single Strategy subclass with build_index() and query() methods.

CI/CD Integration

Fail your pipeline if retrieval quality drops:

```bash kb-arena benchmark --corpus my-docs --fail-below 0.7

Step 4: Run the pipeline

```bash

Shows: file count by extension, parser assignment, output path

kb-arena benchmark --corpus my-docs --dry-run

Side-by-Side Strategy Comparison

New "Compare" view in the benchmark UI lets you pick two strategies and see tier-by-tier accuracy, latency, and cost differences side by side.

Strategy Arena - Blind A/B Comparison

A new Arena mode for blind head-to-head strategy battles. Ask a question, two random strategies answer it, you vote for the better response. ELO ratings emerge over time.

kb-arena serve  # then open /arena in your browser

Run Comparison

Benchmark runs now have unique IDs and timestamps. Results are preserved across runs instead of overwritten:

```bash kb-arena benchmark --corpus my-docs

Auto-generate benchmark questions from your docs (10 per difficulty tier)

kb-arena generate-questions --corpus my-docs --count 50

Run the benchmark (each question x 9 strategies, 4-pass evaluation)

kb-arena benchmark --corpus my-docs

Q&A Generator

Generate Q&A pairs from your docs — use them for chatbot training, FAQ pages, or search indexes. Only needs an Anthropic key (no embeddings, no vector DB).

```bash kb-arena generate-qa --corpus my-docs

Outputs: datasets/my-docs/qa-pairs/qa_pairs.jsonl

```

CLI

Web UI

Question Tiers

Questions are organized into 5 difficulty tiers:

Use kb-arena generate-questions to auto-generate questions from your docs, or write them by hand in YAML.

---

Shows: question count, strategy list, total queries, concurrency settings

![Dry Run Preview](docs/demo-dry-run.gif)

**JSON output** — Pipe structured data to `jq`, scripts, or CI pipelines:

![JSON Output](docs/demo-format-json.gif)

**Pipeline hints** — After every command, see what to run next:

Next: kb-arena build-graph --corpus my-docs && kb-arena build-vectors --corpus my-docs

**Progress bars** — Every long-running command shows real-time progress (extraction sections, Neo4j batch loading, vector index building, question generation tiers).

**Cost tracking** — Benchmark runs display cumulative API cost in the progress bar and print per-strategy cost/accuracy summaries after completion.

**Verbose mode** — Add `--verbose` / `-v` to any command for debug logging:

---