AIfred智能多代理助手

Prerequisites

• Python 3.10+
• LLM Backend (choose one):
• llama.cpp via llama-swap (GGUF models) - best performance, full GPU control (setup guide)
• Ollama (easy, GGUF models) - recommended for getting started
• vLLM (fast, AWQ models) - best performance for AWQ (requires Compute Capability 7.5+)
• TabbyAPI (ExLlamaV2/V3, EXL2 models) - experimental

Zero-Config Model Management (llama.cpp backend): After the initia

🚀 Installation

Example Usage

```bash

Use Cases

• Cloud Control: Operate AIfred from anywhere via HTTPS/API
• Home Automation: Integration with Home Assistant, Node-RED, etc.
• Voice Assistants: Alexa/Google Home can send AIfred queries
• Batch Processing: Automated queries via scripts
• Mobile Apps: Custom apps can use the API
• Remote Maintenance: Test and monitor AIfred on headless systems

---

Per-Session Config SSOT

Agent selection, discussion mode, and research mode are persisted per session, not globally. Every chat session has its own config block stored in its session file:

{
  "data": {
    "config": {
      "active_agent": "aifred",
      "multi_agent_mode": "standard",
      "symposion_agents": [],
      "research_mode": "automatik"
    }
  }
}

Clean default on new session: every new chat starts with aifred + standard + automatik — never inheriting from the previous session.

Multi-tab / cross-channel sync via session file mtime-watching: whenever any writer (browser tab, API, email channel, voice puck) modifies the session file, all other tabs that have this session open detect the change within 1 second and reload — without polling, without events, without race conditions. This replaces the legacy update_flag mechanism entirely.

Get current global settings

curl http://localhost:8002/api/settings

Change model (global setting)

curl -X PATCH http://localhost:8002/api/settings \ -H "Content-Type: application/json" \ -d '{"aifred_model": "qwen3:14b"}'

Switch a session to Tribunal mode (per-session config)

🎤 Voice & Vision Interface

• Voice Interface: STT via Whisper Docker container (dual-device: CPU permanent + GPU with TTL auto-unload, Web-UI for model/settings management). TTS engines: Edge TTS, XTTS v2 Voice Cloning, MOSS-TTS 1.7B, DashScope Qwen3-TTS Cloud Streaming, Piper, espeak. Per-agent TTS configuration (voice, speed, pitch, on/off per agent), gapless realtime audio playback
• FreeEcho.2 Voice Terminal: Dedicated voice interface for Echo Dot 2 hardware (custom firmware). Wake word detection, immediate browser flush (user question visible within 500ms after STT), deferred TTS container management (parallel GPU cleanup during LLM inference)
• Vision/OCR: Image analysis with multimodal LLMs (DeepSeek-OCR, Qwen3-VL, Ministral-3), VL Follow-Up, interactive image crop, 2-model architecture (Vision-LLM + Main-LLM)

🔊 Voice Interface (TTS Engines)

AIfred supports 6 TTS engines with different trade-offs between quality, latency, and resource usage. Each engine was chosen for a specific use case after extensive experimentation.

Why multiple engines?

The search for the perfect TTS experience led through several iterations:

• Edge TTS was the first engine -- free, fast, decent quality, but limited voices and no voice cloning.
• XTTS v2 added high-quality voice cloning with multilingual support. Sentence-level streaming works well: while the LLM generates the next sentence, XTTS synthesizes the current one. However, it requires a Docker container and ~2 GB VRAM.
• MOSS-TTS 1.7B delivers the best speech quality of all open-source models (SIM 73-79%), but at a cost: ~18-22 seconds per sentence makes it unsuitable for streaming. Audio is generated as a batch after the complete response, which is acceptable for short answers but frustrating for longer ones.
• DashScope Qwen3-TTS adds cloud-based voice cloning via Alibaba Cloud's API. By default it uses sentence-level streaming (same as XTTS) for better intonation. A realtime WebSocket mode (word-level chunks, ~200ms first audio) is also implemented but disabled by default -- it trades slightly worse prosody for faster first-audio. To re-enable it, uncomment the WebSocket block in state.py:_init_streaming_tts() (see code comment there).
• Piper TTS and eSpeak serve as lightweight offline alternatives that work without Docker, GPU, or internet connection.

Playback Architecture: - Visible HTML5 <audio> widget with blob-URL prefetching (next 2 chunks pre-fetched into memory) - preservesPitch: true for speed adjustments without chipmunk effect - Per-agent voice/pitch/speed settings (AIfred, Sokrates, Salomo can each have distinct voices) - SSE-based audio streaming from backend to browser (persistent connection, 15s keepalive)

☁️ Cloud API Support

AIfred supports cloud LLM providers via OpenAI-compatible APIs:

Features: - Dynamic model fetching (models loaded from provider's /models endpoint) - Token usage tracking (prompt + completion tokens displayed in debug console) - Per-provider model memory (each provider remembers its last used model) - Vision model filtering (excludes -vl variants from main LLM dropdown) - Streaming support with real-time output

Note: Cloud APIs don't require local GPU resources - ideal for: - Testing larger models without hardware investment - Mobile/laptop usage without dedicated GPU - Comparing cloud vs local model quality

📁 Code Structure Reference

Core Entry Points: - aifred/state.py - Main state management, send_message()

Automatik Mode: - aifred/lib/conversation_handler.py - Decision logic, RAG context

Web Research Pipeline: - aifred/lib/research/orchestrator.py - Top-level orchestration (incl. URL ranking) - aifred/lib/research/cache_handler.py - Session cache - aifred/lib/research/query_processor.py - Query optimization + search - aifred/lib/research/url_ranker.py - LLM-based URL relevance ranking (NEW) - aifred/lib/research/scraper_orchestrator.py - Parallel scraping - aifred/lib/research/context_builder.py - Context building + LLM

Document RAG Pipeline: - aifred/lib/document_store.py - ChromaDB Documents collection — token-accurate chunking (Qwen3 tokenizer, char fallback), delete + upsert for clean re-indexing, dual embedding functions (index/query mode), folder filter + chunk-neighbor retrieval in search() - aifred/lib/file_manager.py - Single source of truth for file-system + ChromaDB operations (used by Document UI and Workspace plugin): list/create/delete/rename/index/deindex/search/list_orphaned

Supporting Modules: - aifred/lib/vector_cache.py - ChromaDB semantic cache for web research, includes OllamaEmbeddingFunction with mode-switch (index→GPU+warm, query→CPU) - aifred/lib/agent_memory.py - Per-agent ChromaDB memory collections - aifred/lib/tool_output_cap.py - Token budget for tool results (75% input ratio, JSON-aware truncation, ContextVar-based) - aifred/lib/debug_format.py - Tool-call/result formatting for the debug panel (key=value rendering, agent prefix, token count) - aifred/lib/intent_detector.py - Temperature selection - aifred/lib/agent_tools.py - Web search, scraping, context building

📝 Automatik-LLM Prompts Reference

The Automatik-LLM uses dedicated prompts in prompts/{de,en}/automatik/ for various decisions:

Language Rules: - EN only: Output is structured/numeric (parseable), language doesn't affect result - DE + EN: Output depends on user's language or requires semantic understanding in that language

Prompt Directory Structure:

prompts/
├── de/
│   └── automatik/
│       ├── research_decision.txt      # German queries for German users
│       └── followup_intent_detection.txt
└── en/
    └── automatik/
        ├── intent_detection.txt       # Universal intent detection
        ├── research_decision.txt      # English queries (Query 1 always EN)
        ├── followup_intent_detection.txt
        └── url_ranking.txt            # Numeric output (indices)

---

🌐 REST API (Browser Remote Control)

AIfred provides a complete REST API for programmatic control - enabling remote operation via Cloud, automation systems, and third-party integrations.

API Endpoints

The API enables pure remote control - messages are injected into browser sessions, the browser performs the full processing (Intent Detection, Multi-Agent, Research, etc.). The user sees everything live in the browser.

Global vs per-session: /api/settings covers truly global settings (backend, models, TTS voices, language, sampling). Anything that belongs to a specific conversation — agent, multi-agent mode, research mode, symposion participants — goes through /api/session/config and is stored in the session file as SSOT.

🔄 Research Mode Workflows

AIfred offers 4 different research modes, each using different strategies depending on requirements. Here's the detailed workflow for each mode:

Inject a message (browser runs full pipeline)

curl -X POST http://localhost:8002/api/chat/inject \ -H "Content-Type: application/json" \ -d '{"message": "What is Python?", "device_id": "abc123..."}'