ChronoMiner

System Requirements

• Python 3.12+
• At least one API key (see provider table above)

All Python dependencies are declared in pyproject.toml.

Install runtime dependencies (creates .venv automatically)

uv sync

Installation

```bash git clone https://github.com/Paullllllllllllllllll/ChronoMiner.git cd ChronoMiner

Quick Start

Configuration

ChronoMiner uses five YAML files in config/.

1. Model Configuration (`model_config.yaml`)

extraction_model:
  provider: openai       # openai | anthropic | google | openrouter
                         # | custom (auto-detected if omitted)
  name: gpt-5-mini
  max_output_tokens: 128000
  reasoning:
    effort: medium       # Cross-provider (low | medium | high)
  temperature: 0.01      # Disabled automatically for reasoning
  top_p: 1.0             #   models

Key parameters: provider (auto-detected from model name if omitted), name (model identifier), max_output_tokens (must cover reasoning tokens on reasoning models), reasoning.effort (automatically translated per provider), temperature/top_p (applied only when the model supports them).

2. Paths Configuration (`paths_config.yaml`)

general:
  interactive_mode: true
  retain_temporary_jsonl: true
  logs_dir: './logs'
schemas_paths:
  BibliographicEntries:
    input: './input/bibliography'
    output: './output/bibliography'
    csv_output: true
    docx_output: true
    txt_output: true

Controls execution mode, temporary file retention, and per-schema input/output directories with output format toggles.

3. Chunking Configuration (`chunking_and_context.yaml`)

chunking:
  default_tokens_per_chunk: 7500

Also configures matching rules (whitespace normalization, case sensitivity, diacritics) and retry behavior for the line-range readjuster (certainty threshold, max retries, context expansion).

4. Image Processing Configuration (`image_processing_config.yaml`)

Provider-specific sections configure vision preprocessing automatically based on the active model:

api_image_processing:
  llm_detail: high          # OpenAI / OpenRouter
anthropic_image_processing:
  resize_profile: auto      # Anthropic
google_image_processing:
  media_resolution: high    # Google Gemini
target_dpi: 300             # PDF-to-image rendering

5. Concurrency Configuration (`concurrency_config.yaml`)

concurrency:
  extraction:
    concurrency_limit: 20
    delay_between_tasks: 0.1
    retry:
      attempts: 150
daily_token_limit:
  enabled: true
  daily_tokens: 9000000

Controls concurrent task limits, exponential backoff retry, and daily token budgets (automatic reset at local midnight).

Custom OpenAI-Compatible Endpoint

Connect to any self-hosted or third-party endpoint implementing the OpenAI Chat Completions API. Set provider: custom in model_config.yaml and configure the custom_endpoint block:

extraction_model:
  provider: custom
  name: "org/model-name"
  custom_endpoint:
    base_url: "https://your-endpoint.example.com/v1"
    api_key_env_var: "CUSTOM_API_KEY"
  max_output_tokens: 4096
  temperature: 0.0

The endpoint must support OpenAI-compatible structured outputs (response_format with type: json_schema). Custom endpoints do not support batch processing.

CLI Reference

--schema NAME              Schema name for extraction
--input / --output         Input and output paths
--chunking STRATEGY        auto | auto-adjust | line_ranges |
                           adjust-line-ranges
--batch                    Use async batch API
--image-detail LEVEL       low | high | auto | original
--input-type TYPE          Override auto-detection: text | image | pdf
--model ID                 Override model
--reasoning-effort LEVEL   none | low | medium | high | xhigh
--chunk-size N             Override tokens per chunk
--context MODE_OR_PATH     auto | none | /path/to/context.txt
--context-image            Enable context image injection (see below)
--first-n-chunks N         Process only the first N chunks/pages
--last-n-chunks N          Process only the last N chunks/pages
--resume / --force         Skip vs overwrite existing output

Run python main/process_text_files.py --help for the full list.

Common Workflows

Large-scale batch processing:

```bash

Frequently Asked Questions

Which AI provider should I choose? OpenAI gpt-5-mini offers the best cost/quality balance with a 50% batch discount. Google Gemini Flash is fastest and cheapest but may reject deeply nested schemas. Anthropic Claude excels with complex layouts. Start with OpenAI gpt-5-mini at medium reasoning effort.

How much does extraction cost? With OpenAI gpt-5-mini: roughly $0.10--0.20 per small file (50 KB), $1--2 per medium file (500 KB). Batch processing halves these costs.

Batch or synchronous? Use batch for large-scale jobs where you can wait up to 24 hours. Use synchronous for immediate results, small jobs, or testing.

Which chunking strategy should I use? Start with automatic for quick runs. For production workflows, generate line ranges, adjust with semantic boundaries, then extract using the adjusted ranges.

How do I add a custom schema? Create a JSON schema in schemas/, register it in modules/extract/schema_handlers.py, add context guidance, and configure paths in paths_config.yaml.

How do I switch providers? Edit config/model_config.yaml and set the appropriate environment variable. Provider can also be auto-detected from the model name.

What happens when extraction fails? Failed chunks are logged. Use repair_extractions.py to recover partial results from batch jobs. For synchronous jobs, re-run with --resume to skip already-processed chunks.

Can I process password-protected PDFs? No. Decrypt them first using external tools.

How do I integrate into existing pipelines? Use CLI mode (interactive_mode: false). All scripts return proper exit codes suitable for shell scripting and CI/CD.

I'm experiencing issues not covered here. Check logs in the configured logs_dir, validate configuration files, and review pyproject.toml for version mismatches. For persistent issues, open a GitHub issue with error details and relevant configuration sections.