# SuperLocalMemory V2.2.0 - Optional Search Components

**Created by:** Varun Pratap Bhardwaj
**Version:** 2.2.0
**Date:** February 7, 2026

---

## Overview

SuperLocalMemory V2.2.0 introduces two powerful optional search components that significantly enhance memory retrieval performance and accuracy:

1. **HNSW Index** (`src/hnsw_index.py`) - Fast approximate nearest neighbor search
2. **Embedding Engine** (`src/embedding_engine.py`) - Local semantic embedding generation

Both components are **completely optional** with graceful fallback to existing TF-IDF methods if dependencies are not installed.

---

## Component #1: HNSW Index

### What is HNSW?

HNSW (Hierarchical Navigable Small World) is a state-of-the-art algorithm for approximate nearest neighbor search. It provides:

- **Sub-10ms search** for 10,000 memories
- **Sub-50ms search** for 100,000 memories
- **Incremental updates** without full rebuild
- **Disk persistence** for instant startup

### Installation

```bash
# Optional - only install if you want HNSW acceleration
pip install hnswlib
```

### Usage

```python
from hnsw_index import HNSWIndex
import numpy as np

# Initialize index
index = HNSWIndex(dimension=384)  # Match your embedding dimension

# Build from vectors
vectors = np.random.randn(1000, 384).astype('float32')
memory_ids = list(range(1000))
index.build(vectors, memory_ids)

# Search for similar vectors
query = np.random.randn(384).astype('float32')
results = index.search(query, k=5)
# Returns: [(memory_id, similarity_score), ...]

# Add single vector (incremental)
new_vector = np.random.randn(384).astype('float32')
index.add(new_vector, memory_id=1001)

# Get statistics
stats = index.get_stats()
print(f"Indexed: {stats['indexed_count']} vectors")
print(f"Method: {'HNSW' if stats['use_hnsw'] else 'Linear fallback'}")
```

### Fallback Behavior

If `hnswlib` is not installed:
- Automatically falls back to sklearn-based linear search
- Performance: ~100ms for 10K vectors (vs <10ms with HNSW)
- No feature degradation - all functionality remains available

### CLI Commands

```bash
# Show index statistics
python src/hnsw_index.py stats

# Rebuild from database
python src/hnsw_index.py rebuild

# Run performance test
python src/hnsw_index.py test
```

### Performance Benchmarks

| Dataset Size | HNSW Search | Linear Search | Speedup |
|--------------|-------------|---------------|---------|
| 1K vectors   | <1ms        | ~10ms         | 10x     |
| 10K vectors  | <10ms       | ~100ms        | 10x     |
| 100K vectors | <50ms       | ~1000ms       | 20x     |

### Configuration

```python
index = HNSWIndex(
    dimension=384,              # Vector dimension
    max_elements=100_000,       # Maximum vectors to index
    m=16,                       # HNSW M parameter (connections)
    ef_construction=200,        # Build-time quality
    ef_search=50                # Search-time quality
)
```

**Parameters explained:**
- `m`: Higher = better accuracy, more memory (typical: 16)
- `ef_construction`: Higher = better index quality, slower build (typical: 200)
- `ef_search`: Higher = better accuracy, slower search (typical: 50)

### Security & Limits

```python
MAX_MEMORIES_FOR_HNSW = 100_000    # Prevents memory exhaustion
MAX_DIMENSION = 5000                # Typical: 384 for embeddings
```

---

## Component #2: Embedding Engine

### What is the Embedding Engine?

The Embedding Engine generates semantic embeddings locally using sentence-transformers, enabling:

- **True semantic search** (understands meaning, not just keywords)
- **100% local processing** (no API calls, no cloud dependencies)
- **GPU acceleration** (CUDA/Apple Silicon MPS support)
- **Smart caching** (LRU cache for 10K embeddings)

### Installation

```bash
# Optional - only install if you want semantic embeddings
pip install sentence-transformers

# For GPU acceleration (optional)
pip install torch  # Will auto-detect CUDA/MPS
```

### Usage

```python
from embedding_engine import EmbeddingEngine

# Initialize engine
engine = EmbeddingEngine()  # Auto-detects GPU, downloads model

# Generate single embedding
text = "SuperLocalMemory is a local memory system for AI assistants"
embedding = engine.encode(text)
# Returns: numpy array of shape (384,)

# Batch processing
texts = [
    "First memory about authentication",
    "Second memory about database design",
    "Third memory about API endpoints"
]
embeddings = engine.encode(texts, batch_size=32)
# Returns: numpy array of shape (3, 384)

# Compute similarity
emb1 = engine.encode("JWT authentication")
emb2 = engine.encode("Token-based auth")
similarity = engine.similarity(emb1, emb2)
# Returns: 0.85 (high similarity)

# Get statistics
stats = engine.get_stats()
print(f"Device: {stats['device']}")  # cuda / mps / cpu
print(f"Cache size: {stats['cache_size']}")
```

### Models Available

| Model | Dimension | Size | Speed | Quality | Use Case |
|-------|-----------|------|-------|---------|----------|
| **all-MiniLM-L6-v2** (default) | 384 | 80MB | Fast | Good | General use |
| all-mpnet-base-v2 | 768 | 420MB | Slower | Better | High accuracy |
| paraphrase-multilingual | 384 | 420MB | Medium | Good | Multiple languages |

```python
# Use different model
engine = EmbeddingEngine(model_name="all-mpnet-base-v2")
```

### Fallback Behavior

If `sentence-transformers` is not installed:
- Automatically falls back to TF-IDF vectorization
- Performance: Still fast, but less semantic understanding
- Dimension: 384 (padded/truncated to match)

### CLI Commands

```bash
# Show engine statistics
python src/embedding_engine.py stats

# Generate embeddings for all memories
python src/embedding_engine.py generate

# Clear embedding cache
python src/embedding_engine.py clear-cache

# Run performance test
python src/embedding_engine.py test
```

### Performance Benchmarks

| Device | Speed (texts/sec) | GPU Memory |
|--------|-------------------|------------|
| CPU (Intel i7) | ~100 | N/A |
| CUDA (RTX 3090) | ~1000 | ~2GB |
| Apple M1/M2 (MPS) | ~500 | Unified |
| Cache Hit | ~1,000,000 | N/A |

### GPU Acceleration

```python
# Auto-detect (recommended)
engine = EmbeddingEngine()  # Tries: CUDA → MPS → CPU

# Force specific device
engine = EmbeddingEngine(device='cuda')   # NVIDIA GPU
engine = EmbeddingEngine(device='mps')    # Apple Silicon
engine = EmbeddingEngine(device='cpu')    # CPU only
```

### Caching

The engine uses an LRU cache for repeated queries:

```python
# First call: generates embedding (~10-100ms)
emb1 = engine.encode("Same text")

# Second call: cache hit (~0.001ms)
emb2 = engine.encode("Same text")

# Save cache to disk
engine.save_cache()

# Clear cache
engine.clear_cache()
```

Cache settings:
- **Max size:** 10,000 entries
- **Eviction:** LRU (Least Recently Used)
- **Persistence:** Saved to `~/.superlocalmemory/embedding_cache.json`

### Security & Limits

```python
MAX_BATCH_SIZE = 128        # Prevents memory exhaustion
MAX_TEXT_LENGTH = 10_000    # Characters per input
CACHE_MAX_SIZE = 10_000     # Cache entries
```

---

## Integration with MemoryStoreV2

### Adding Embeddings to Database

```python
from embedding_engine import EmbeddingEngine
from pathlib import Path

engine = EmbeddingEngine()

# Add embeddings to all memories
db_path = Path.home() / ".claude-memory" / "memory.db"
engine.add_to_database(db_path)
```

This will:
1. Add `embedding` column to database (if not exists)
2. Generate embeddings for all memories
3. Store as JSON arrays in database
4. Use batch processing for efficiency

### Building HNSW Index from Database

```python
from hnsw_index import HNSWIndex
from pathlib import Path

index = HNSWIndex(dimension=384)

# Build from database embeddings
db_path = Path.home() / ".claude-memory" / "memory.db"
index.rebuild_from_db(db_path, embedding_column='embedding')
```

### Complete Search Pipeline

```python
from memory_store_v2 import MemoryStoreV2
from embedding_engine import EmbeddingEngine
from hnsw_index import HNSWIndex

# Initialize components
store = MemoryStoreV2()
engine = EmbeddingEngine()
index = HNSWIndex(dimension=384)

# Add memory with embedding
content = "Fixed authentication bug using JWT tokens"
mem_id = store.add_memory(content)

# Generate and store embedding
embedding = engine.encode(content)
# Store embedding in database...

# Add to HNSW index
index.add(embedding, mem_id)

# Search
query = "auth token issue"
query_embedding = engine.encode(query)
similar_ids = index.search(query_embedding, k=5)

# Get full memory details
for mem_id, score in similar_ids:
    memory = store.get_by_id(mem_id)
    print(f"[{mem_id}] Score: {score:.3f}")
    print(f"  {memory['content'][:100]}...")
```

---

## Installation Guide

### Minimal Installation (TF-IDF only)

```bash
# Already included in base requirements
pip install scikit-learn numpy
```

### Standard Installation (Recommended)

```bash
# Add semantic search
pip install sentence-transformers

# This automatically installs:
# - torch (PyTorch)
# - transformers (Hugging Face)
# - tqdm (progress bars)
```

### Full Installation (Maximum Performance)

```bash
# Standard + HNSW acceleration
pip install sentence-transformers hnswlib

# For GPU acceleration (if you have NVIDIA GPU)
pip install torch --index-url https://download.pytorch.org/whl/cu118
```

### Apple Silicon (M1/M2/M3)

```bash
# MPS (Metal Performance Shaders) support included
pip install sentence-transformers hnswlib

# torch with MPS support (usually auto-installed)
pip install torch torchvision torchaudio
```

---

## Verification

Check what's available:

```bash
# Test HNSW
python src/hnsw_index.py stats

# Test Embeddings
python src/embedding_engine.py stats

# Run comprehensive test
python test_new_modules.py
```

---

## Troubleshooting

### Issue: "No module named 'hnswlib'"

**Solution:**
```bash
pip install hnswlib
```

Or just use the fallback - the system will automatically use linear search.

### Issue: "No module named 'sentence_transformers'"

**Solution:**
```bash
pip install sentence-transformers
```

Or just use the fallback - the system will automatically use TF-IDF.

### Issue: "CUDA out of memory"

**Solution:**
```python
# Reduce batch size
engine = EmbeddingEngine()
embeddings = engine.encode(texts, batch_size=16)  # Default: 32

# Or force CPU
engine = EmbeddingEngine(device='cpu')
```

### Issue: "Model download is slow"

**Solution:**
Models are cached locally after first download:
- Location: `~/.superlocalmemory/models/`
- Size: 80MB (all-MiniLM-L6-v2)
- Subsequent loads: <1 second

### Issue: "Search is slow even with HNSW"

**Possible causes:**
1. Fallback mode active (check `index.get_stats()`)
2. Need to adjust `ef_search` parameter
3. Index not built yet

**Solution:**
```python
# Rebuild index
index.rebuild_from_db(db_path)

# Increase search quality (slower but more accurate)
index.ef_search = 100  # Default: 50
```

---

## API Reference

### HNSWIndex Class

```python
class HNSWIndex:
    def __init__(
        dimension: int = 384,
        max_elements: int = 100_000,
        m: int = 16,
        ef_construction: int = 200,
        ef_search: int = 50
    )

    def build(vectors: np.ndarray, memory_ids: List[int])
    def add(vector: np.ndarray, memory_id: int)
    def search(query_vector: np.ndarray, k: int = 5) -> List[Tuple[int, float]]
    def update(memory_id: int, vector: np.ndarray)
    def delete(memory_id: int)
    def rebuild_from_db(db_path: Path, embedding_column: str = 'embedding')
    def get_stats() -> Dict[str, Any]
```

### EmbeddingEngine Class

```python
class EmbeddingEngine:
    def __init__(
        model_name: str = "all-MiniLM-L6-v2",
        device: Optional[str] = None,
        use_cache: bool = True
    )

    def encode(texts: Union[str, List[str]], batch_size: int = 32) -> np.ndarray
    def encode_batch(texts: List[str], batch_size: int = 32) -> np.ndarray
    def similarity(embedding1: np.ndarray, embedding2: np.ndarray) -> float
    def add_to_database(db_path: Path, embedding_column: str = 'embedding')
    def save_cache()
    def clear_cache()
    def get_stats() -> Dict[str, Any]
```

---

## Best Practices

### 1. Generate Embeddings Once

```python
# Do this once after installation
engine = EmbeddingEngine()
engine.add_to_database(db_path)
```

### 2. Rebuild HNSW Periodically

```python
# After adding many new memories
index = HNSWIndex()
index.rebuild_from_db(db_path)
```

### 3. Use Batch Processing

```python
# Good: process in batches
embeddings = engine.encode(texts, batch_size=32)

# Bad: process one at a time (slow)
embeddings = [engine.encode(text) for text in texts]
```

### 4. Monitor Cache Hit Rate

```python
stats = engine.get_stats()
print(f"Cache size: {stats['cache_size']}/{stats['cache_max_size']}")

# Save cache periodically
engine.save_cache()
```

### 5. Choose Right Model for Use Case

- **General use:** all-MiniLM-L6-v2 (default, fast)
- **High accuracy:** all-mpnet-base-v2 (slower, better)
- **Multilingual:** paraphrase-multilingual

---

## Performance Tuning

### HNSW Parameters

**For speed priority:**
```python
index = HNSWIndex(
    m=8,                # Fewer connections
    ef_construction=100,  # Faster build
    ef_search=25        # Faster search
)
```

**For accuracy priority:**
```python
index = HNSWIndex(
    m=32,               # More connections
    ef_construction=400,  # Better build
    ef_search=100       # More thorough search
)
```

**Balanced (recommended):**
```python
index = HNSWIndex(
    m=16,
    ef_construction=200,
    ef_search=50
)
```

### Embedding Batch Sizes

| Use Case | Batch Size | Memory | Speed |
|----------|------------|--------|-------|
| Interactive (real-time) | 1-8 | Low | Fast response |
| Standard | 32 | Medium | Balanced |
| Bulk processing | 64-128 | High | Maximum throughput |

---

## Architecture Diagram

```
┌─────────────────────────────────────────────────────────────┐
│  SuperLocalMemory V2.2.0 Search Architecture                │
├─────────────────────────────────────────────────────────────┤
│                                                               │
│  User Query: "authentication token bug"                      │
│         │                                                     │
│         ├──> [1] EmbeddingEngine.encode(query)              │
│         │        ├─> Cache check (0.001ms if hit)           │
│         │        └─> Model inference (10-100ms if miss)     │
│         │                                                     │
│         ├──> [2] HNSWIndex.search(embedding, k=5)           │
│         │        ├─> HNSW search (<10ms for 10K vectors)    │
│         │        └─> Linear fallback (~100ms if no HNSW)    │
│         │                                                     │
│         └──> [3] MemoryStoreV2.get_by_id(memory_ids)        │
│                  └─> SQLite lookup (<1ms per memory)        │
│                                                               │
│  Total latency: 10-150ms (depending on cache/GPU/HNSW)      │
│                                                               │
│  Fallback path (no optional deps):                           │
│  MemoryStoreV2.search() -> TF-IDF -> Full-text search       │
│  Total latency: ~100ms (still acceptable)                    │
└─────────────────────────────────────────────────────────────┘
```

---

## Version History

### V2.2.0 (February 7, 2026)

**Added:**
- `src/hnsw_index.py` - Fast approximate nearest neighbor search
- `src/embedding_engine.py` - Local semantic embedding generation
- Complete test suite (`test_new_modules.py`)
- Comprehensive documentation (this file)

**Features:**
- Optional dependencies with graceful fallback
- Sub-10ms search for 10K memories (with HNSW)
- GPU acceleration (CUDA/MPS)
- LRU caching for embeddings
- Disk persistence for index
- Security limits and input validation
- CLI interfaces for both components

**Performance:**
- 10-20x speedup with HNSW vs linear search
- 100-1000 texts/sec embedding generation (GPU)
- <1ms cache hit latency

---

## Credits

**Architecture & Implementation:** Varun Pratap Bhardwaj (Solution Architect)

**Technologies:**
- hnswlib: Fast approximate nearest neighbor search
- sentence-transformers: Local embedding generation
- scikit-learn: TF-IDF fallback
- PyTorch: GPU acceleration

**License:** AGPL-3.0

---

## Support

For issues, questions, or contributions:
- **GitHub Issues:** https://github.com/qualixar/superlocalmemory/issues
- **Documentation:** https://github.com/qualixar/superlocalmemory/wiki

---

**SuperLocalMemory V2.2.0** - Making AI memory faster and smarter, locally.

*Created by Varun Pratap Bhardwaj, February 2026*
