# T8.5: Multilingual Awareness

## Overview

Add deterministic query language detection using franc + language-aware prompt selection for expansion/rerank. Detection runs at query time; --lang CLI flag overrides.

**CRITICAL DISTINCTION**: Detected language affects **prompt selection only**, NOT retrieval filtering. The `--lang` CLI flag remains the only way to filter by chunk language.

## Scope

- Install franc + iso-639-3 packages
- New `src/pipeline/query-language.ts` module for detection
- Hook detection into pipeline for **prompt selection + metadata only**
- Maintain existing expansion prompt logic (EN, DE, MULTILINGUAL)
- Tests for detection + integration

**Out of scope**: Adding new language-specific prompts beyond EN/DE/MULTILINGUAL (existing prompts are sufficient).

## Approach

### 1. Dependencies

```bash
bun add franc iso-639-3
```

Use `franc` (187 languages, ~200KB) not franc-min. Both packages are ESM-only, work with Bun.

### 2. Language Detection Module

New file: `src/pipeline/query-language.ts` (NOT src/search/ - keep aligned with pipeline subsystem)

```typescript
import { franc } from "franc";
import { iso6393 } from "iso-639-3";

const MIN_RELIABLE_LENGTH = 15;

// Build ISO 639-3 → BCP-47 (ISO 639-1) mapping at module init
const ISO639_3_TO_BCP47: Record<string, string> = {};
for (const lang of iso6393) {
  if (lang.iso6391) {
    ISO639_3_TO_BCP47[lang.iso6393] = lang.iso6391;
  }
}

export interface LanguageDetection {
  bcp47: string; // 'en', 'de', 'und'
  iso639_3: string; // 'eng', 'deu', 'und'
  confident: boolean; // false if < minLength or 'und'
}

export function detectQueryLanguage(text: string): LanguageDetection {
  const trimmed = text.trim();

  if (trimmed.length < MIN_RELIABLE_LENGTH) {
    return { bcp47: "und", iso639_3: "und", confident: false };
  }

  const detected = franc(trimmed, {
    minLength: MIN_RELIABLE_LENGTH,
    only: [
      "eng",
      "deu",
      "fra",
      "ita",
      "spa",
      "por",
      "nld",
      "cmn",
      "jpn",
      "kor",
    ],
  });

  if (detected === "und") {
    return { bcp47: "und", iso639_3: "und", confident: false };
  }

  const bcp47 = ISO639_3_TO_BCP47[detected] ?? "und";
  return { bcp47, iso639_3: detected, confident: true };
}
```

**Design decisions**:

- Build ISO mapping from dataset at module init (iso-639-3 exports array, not object)
- `only` filter restricts to 10 major languages for speed + accuracy
- Returns 'und' if no iso6391 exists (consistent fallback)
- `confident: false` when text too short or undetermined

### 3. Integration Points

**CRITICAL**: Two separate concepts:

| Concept         | Purpose                            | Source                       | Used For                              |
| --------------- | ---------------------------------- | ---------------------------- | ------------------------------------- |
| `queryLanguage` | Detected/overridden query language | detectQueryLanguage() or CLI | Prompt selection, metadata, cache key |
| `options.lang`  | Explicit retrieval filter          | CLI `--lang` only            | FTS/vector chunk filtering            |

**Resolution for prompt selection** (NOT retrieval):

1. Explicit `--lang <code>` from CLI (highest priority) → use as queryLanguage too
2. Detected language from query text
3. Collection's `languageHint` (only if detection uncertain AND single collection specified)
4. Return 'und' (use MULTILINGUAL prompt)

**Hook location**: `src/pipeline/hybrid.ts` at start of `executeHybridSearch()`:

```typescript
// Detect for prompt selection ONLY - does NOT affect retrieval filters
const detection = detectQueryLanguage(query);
const queryLanguage = options.lang ?? detection.bcp47;
// Pass queryLanguage to expandQuery (for prompt selection)
// Pass original options.lang (or undefined) to store.searchFts (for filtering)
```

### 4. Files to Modify

| File                                   | Changes                                          |
| -------------------------------------- | ------------------------------------------------ |
| `src/pipeline/query-language.ts`       | **New** - detectQueryLanguage()                  |
| `src/pipeline/hybrid.ts:70-76`         | Detect for prompt selection, pass to expandQuery |
| `src/pipeline/search.ts`               | Detect for meta.lang output                      |
| `src/pipeline/vsearch.ts`              | Detect for meta.lang output                      |
| `src/cli/commands/ask.ts`              | Set AskResult.queryLanguage from detection       |
| `test/pipeline/query-language.test.ts` | **New** - detection unit tests                   |

**NOT modified for retrieval filtering**: Detection does NOT change store.searchFts language or vsearch chunk filter.

### 5. Existing Code Reuse

- **Keep** `src/ingestion/language.ts` - used for chunk-level detection at index time (different purpose)
- **Keep** `src/pipeline/expansion.ts:getPromptTemplate()` - already handles en/de/multilingual
- **Keep** `generateCacheKey()` - already includes lang, will use queryLanguage
- **Do NOT tighten** `--lang` CLI validation - allow broader identifiers for future code language tags

## Risks / Dependencies

| Risk                                   | Mitigation                                          |
| -------------------------------------- | --------------------------------------------------- |
| Short queries (<15 chars) return 'und' | Fallback to MULTILINGUAL prompt (already works)     |
| franc package size (~200KB)            | Acceptable, runs at query time not startup          |
| Mixed-language queries                 | First language signal wins; acceptable              |
| ESM-only packages                      | Bun handles ESM natively                            |
| iso-639-3 API mismatch                 | Build mapping from dataset array at init (verified) |

## Acceptance Checks

- [ ] `bun add franc iso-639-3` succeeds
- [ ] `detectQueryLanguage("wie konfiguriere ich kubernetes")` returns `{bcp47: 'de', confident: true}`
- [ ] `detectQueryLanguage("hello")` returns `{bcp47: 'und', confident: false}` (too short)
- [ ] `gno query "kubernetes deployments"` uses EN prompt (via detection)
- [ ] `gno query "kubernetes auf deutsch"` uses DE prompt (via detection)
- [ ] `gno query --lang de "kubernetes"` uses DE prompt (override)
- [ ] `gno query "kubernetes"` (short) returns results from ALL languages (detection doesn't filter)
- [ ] Existing tests pass
- [ ] Cache key includes queryLanguage
- [ ] AskResult.queryLanguage populated from detection

## Test Notes

Unit tests in `test/pipeline/query-language.test.ts`:

- Known strings: "Hello world" → en, "Guten Tag" → de, "Bonjour" → fr
- Short text (<15 chars) → und
- Determinism: same input → same output across runs
- Edge cases: empty string, whitespace only, code snippets
- ISO mapping: verify iso6393 array traversal builds correct map

Integration:

- Verify detection affects prompt selection (check expansion logs)
- Verify detection does NOT filter retrieval (short query returns all chunks)
- Verify AskResult.queryLanguage reflects detection
- Add explain line for queryLanguage: `{ stage: 'lang', message: 'queryLanguage=en (detected)' }`

## References

- Issue: gno-h7i.5
- PRD: §10.2, §12, §14.3
- franc: https://github.com/wooorm/franc
- iso-639-3: https://github.com/wooorm/iso-639-3
- Existing prompts: `src/pipeline/expansion.ts:43-99`

## Review Findings Addressed

1. ✅ **CRITICAL**: Separated queryLanguage (prompts) from options.lang (retrieval filter)
2. ✅ **MAJOR**: Moved module to `src/pipeline/query-language.ts` (not src/search/)
3. ✅ **MAJOR**: Fixed iso-639-3 API - build map from dataset array at init
4. ✅ **MAJOR**: Collection hint only applies when single collection AND detection uncertain
5. ✅ **MINOR**: Don't tighten --lang validation (allow broader identifiers)
6. ✅ **MINOR**: Added explicit explain line for queryLanguage output
