Congolese Swahili → French translation · WHO DRC Ebola interviews · 15 October 2026
Which models translate Congolese Swahili into French best?
Claude Opus 5.5 (low effort) · retrieval 5-shot translates best (61.4 chrF++), out of 44 systems on 2,000 held-out sentences. Showing each model the five most similar training pairs helps every model; glossaries hurt. A 12B open model fine-tuned on the training pairs ties Sonnet 4.6. These are clean written sentences, not interview speech: test on ebola-asr's references before choosing.
Best overall
61.4
Claude Opus 5.5 (low effort) · retrieval 5-shot
Cheapest within 6 points
$0.22
DeepSeek V3.2 · retrieval 5-shot · 55.7 per 1k sentences
Best open-weight
58.2
Gemma 4 12B fine-tuned (LoRA, ours) · retrieval 5-shot
Best dedicated MT
50.1
NLLB-200 3.3B
Recommendation
The shortlist for ebola-asr §B6. Every pick uses retrieval 5-shot: the 5 most similar train pairs as examples, which added +1.3 to +4.4 chrF++ on every model tested. Do not use: glossary injection (−0.1 to −0.8 alone; −0.9 to −1.7 on top of retrieval), or naming the dialect in the prompt (no gain).
| Role / system | chrF++ | COMET | $/1k sent. | Why |
|---|---|---|---|---|
| Quality pick (API) Claude Opus 5.5 (low effort) · retrieval 5-shot | 61.4 | 84.2 | $3.21 | Best on chrF++ and COMET. Low effort costs nothing measurable and halves the bill; set the output ceiling to ≥4,096 tokens. |
| Value pick (API) Claude Sonnet 4.6 · retrieval 5-shot | 57.9 | 82.5 | $1.57 | Matches zero-shot Opus 5.5 at about half its cost. |
| Low-cost pick (API, open weights) DeepSeek V3.2 · retrieval 5-shot | 55.7 | 81.1 | $0.22 | About 4% of the leader's cost; the weights are public, so it can also be self-hosted. |
| Self-hosted pick Gemma 4 12B fine-tuned (LoRA, ours) · retrieval 5-shot | 58.2 | 82.8 | GPU hours | Our LoRA fine-tune on the train split: ties Sonnet 4.6 + retrieval (p = 0.40) and fits one GPU. The in-domain gain will overstate the gain on interview speech, so test on ebola-asr's references first. |
All systems on the test split
44 systems × 2,000 sentences, sorted by normalised chrF++. Intervals are per system; whether two systems differ needs the paired tests in RESULTS.md, not overlapping bars.
Frontier APIOpen-weight LLMDedicated MT
Table view
| # | System | Group | chrF++ | 95% CI | COMET-22 | BLEU | Failure % |
|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 5.5 (low effort) · retrieval 5-shot | Frontier API | 61.4 | 60.4–62.3 | 84.2 | 43.3 | 0.4 |
| 2 | Claude Opus 5.5 (low effort) · random 5-shot | Frontier API | 59.7 | 58.7–60.7 | 83.6 | 41.3 | 0.3 |
| 3 | Gemma 4 12B fine-tuned (LoRA, ours) · retrieval 5-shot | Open-weight LLM | 58.2 | 57.2–59.2 | 82.8 | 39.6 | 0.0 |
| 4 | Claude Sonnet 4.6 · retrieval 5-shot | Frontier API | 57.9 | 57.0–58.9 | 82.5 | 37.8 | 0.0 |
| 5 | Claude Opus 5.5 (low effort) · zero-shot | Frontier API | 57.8 | 56.8–58.8 | 82.7 | 38.2 | 0.5 |
| 6 | Claude Opus 5.5 (low effort) · zero-shot, plain 'Swahili' | Frontier API | 57.7 | 56.7–58.7 | 82.3 | 38.4 | 0.2 |
| 7 | Claude Opus 5.5 · zero-shot | Frontier API | 57.6 | 56.6–58.5 | 82.7 | 38.0 | 0.5 |
| 8 | GPT-6 Astra · zero-shot | Frontier API | 57.3 | 56.5–58.2 | 82.8 | 33.0 | 0.5 |
| 9 | Gemma 4 12B fine-tuned (LoRA, ours) · zero-shot | Open-weight LLM | 57.0 | 56.0–57.9 | 82.4 | 38.0 | 0.0 |
| 10 | Gemma 4 31B · retrieval 5-shot | Open-weight LLM | 56.1 | 55.2–57.0 | 82.0 | 35.3 | 0.0 |
| 11 | Claude Sonnet 4.6 · random 5-shot | Frontier API | 55.8 | 54.9–56.7 | 81.3 | 35.1 | 0.1 |
| 12 | DeepSeek V3.2 · retrieval 5-shot | Open-weight LLM | 55.7 | 54.8–56.6 | 81.1 | 35.4 | 0.0 |
| 13 | Claude Sonnet 4.6 · zero-shot, plain 'Swahili' | Frontier API | 55.2 | 54.2–56.1 | 80.5 | 34.0 | 0.1 |
| 14 | Gemma 4 31B · retrieval + glossary | Open-weight LLM | 55.1 | 54.2–56.0 | 81.4 | 34.1 | 0.0 |
| 15 | Llama 4 Maverick · retrieval 5-shot | Open-weight LLM | 54.8 | 53.8–55.7 | 80.9 | 34.6 | 0.1 |
| 16 | Claude Sonnet 4.6 · zero-shot | Frontier API | 54.6 | 53.6–55.5 | 80.4 | 33.2 | 0.1 |
| 17 | DeepSeek V3.2 · retrieval + glossary | Open-weight LLM | 54.1 | 53.2–55.0 | 80.1 | 33.1 | 0.0 |
| 18 | Gemma 4 12B · retrieval 5-shot | Open-weight LLM | 53.9 | 53.0–54.8 | 79.7 | 32.7 | 0.1 |
| 19 | Gemma 4 31B · zero-shot | Open-weight LLM | 53.9 | 52.9–54.7 | 80.3 | 32.8 | 0.0 |
| 20 | Llama 4 Maverick · retrieval + glossary | Open-weight LLM | 53.8 | 52.9–54.7 | 80.4 | 32.9 | 0.0 |
| 21 | Claude Sonnet 5.5 · zero-shot | Frontier API | 53.6 | 52.6–54.5 | 80.2 | 27.0 | 3.8 |
| 22 | Gemma 4 31B · glossary | Open-weight LLM | 53.2 | 52.4–54.1 | 80.0 | 31.5 | 0.0 |
| 23 | DeepSeek V3.2 · random 5-shot | Open-weight LLM | 53.1 | 52.2–54.0 | 79.4 | 32.1 | 0.0 |
| 24 | DeepSeek V3.2 · zero-shot, plain 'Swahili' | Open-weight LLM | 52.9 | 51.9–53.7 | 78.6 | 32.7 | 0.0 |
| 25 | DeepSeek V3.2 · zero-shot | Open-weight LLM | 52.5 | 51.6–53.5 | 78.5 | 32.1 | 0.0 |
| 26 | Gemma 4 12B · retrieval + glossary | Open-weight LLM | 52.2 | 51.3–53.1 | 78.2 | 30.7 | 0.0 |
| 27 | DeepSeek V3.2 · glossary | Open-weight LLM | 51.9 | 51.1–52.8 | 77.9 | 30.6 | 0.0 |
| 28 | Gemma 4 26B-A4B · zero-shot | Open-weight LLM | 51.8 | 51.0–52.7 | 78.7 | 29.9 | 0.0 |
| 29 | Llama 4 Maverick · zero-shot | Open-weight LLM | 50.4 | 49.4–51.2 | 77.9 | 29.1 | 0.1 |
| 30 | Llama 4 Maverick · glossary | Open-weight LLM | 50.3 | 49.4–51.1 | 77.6 | 28.3 | 0.3 |
| 31 | NLLB-200 3.3B | Dedicated MT | 50.1 | 49.2–51.0 | 77.0 | 29.8 | 0.0 |
| 32 | Gemma 4 12B · zero-shot | Open-weight LLM | 50.1 | 49.2–51.0 | 76.9 | 28.3 | 0.0 |
| 33 | TranslateGemma 27B | Dedicated MT | 49.5 | 48.7–50.4 | 79.1 | 24.0 | 0.0 |
| 34 | NLLB-200 1.3B | Dedicated MT | 49.3 | 48.4–50.1 | 76.9 | 28.8 | 0.0 |
| 35 | Gemma 4 12B · glossary | Open-weight LLM | 49.2 | 48.4–50.0 | 76.0 | 27.0 | 0.0 |
| 36 | Gemma 4 E4B · zero-shot | Open-weight LLM | 48.3 | 47.5–49.2 | 75.3 | 27.2 | 0.0 |
| 37 | TranslateGemma 12B | Dedicated MT | 47.2 | 46.3–48.0 | 78.0 | 24.3 | 0.1 |
| 38 | NLLB-200 600M | Dedicated MT | 46.4 | 45.6–47.3 | 74.4 | 25.3 | 0.1 |
| 39 | MADLAD-400 10B | Dedicated MT | 45.2 | 44.1–46.2 | 72.9 | 25.3 | 0.6 |
| 40 | MADLAD-400 3B | Dedicated MT | 44.8 | 43.9–45.8 | 72.9 | 24.3 | 0.2 |
| 41 | Gemma 4 E2B · zero-shot | Open-weight LLM | 43.4 | 42.4–44.3 | 71.1 | 22.4 | 2.1 |
| 42 | TranslateGemma 4B | Dedicated MT | 40.4 | 39.6–41.2 | 73.1 | 17.7 | 0.1 |
| 43 | SeamlessM4T v2 | Dedicated MT | 37.0 | 36.3–37.8 | 57.1 | 15.4 | 0.6 |
| 44 | opus-mt swc-fr | Dedicated MT | 32.7 | 32.0–33.3 | 60.3 | 12.5 | 0.1 |
Quality against cost (Bedrock-hosted systems)
The line joins the Pareto frontier: no other system is both cheaper and better. GPU-hosted systems are left out (their cost is droplet hours, not tokens).
zero-shotretrieval 5-shotother prompting
Table view
| System | Prompting | chrF++ | $ / 1k sentences | On frontier |
|---|---|---|---|---|
| Llama 4 Maverick · zero-shot | zero-shot | 50.4 | 0.05 | yes |
| Llama 4 Maverick · glossary | other prompting | 50.3 | 0.06 | |
| DeepSeek V3.2 · zero-shot, plain 'Swahili' | other prompting | 52.9 | 0.08 | yes |
| Llama 4 Maverick · retrieval 5-shot | retrieval 5-shot | 54.8 | 0.09 | yes |
| DeepSeek V3.2 · zero-shot | zero-shot | 52.5 | 0.10 | |
| Llama 4 Maverick · retrieval + glossary | other prompting | 53.8 | 0.11 | |
| DeepSeek V3.2 · glossary | other prompting | 51.9 | 0.14 | |
| DeepSeek V3.2 · retrieval 5-shot | retrieval 5-shot | 55.7 | 0.22 | yes |
| DeepSeek V3.2 · random 5-shot | other prompting | 53.1 | 0.25 | |
| DeepSeek V3.2 · retrieval + glossary | other prompting | 54.1 | 0.26 | |
| Claude Sonnet 4.6 · zero-shot, plain 'Swahili' | other prompting | 55.2 | 0.70 | |
| Claude Sonnet 4.6 · zero-shot | zero-shot | 54.6 | 0.78 | |
| Claude Sonnet 5.5 · zero-shot | zero-shot | 53.6 | 1.33 | |
| Claude Sonnet 4.6 · retrieval 5-shot | retrieval 5-shot | 57.9 | 1.57 | yes |
| Claude Sonnet 4.6 · random 5-shot | other prompting | 55.8 | 1.75 | |
| Claude Opus 5.5 (low effort) · zero-shot, plain 'Swahili' | other prompting | 57.7 | 2.50 | |
| Claude Opus 5.5 (low effort) · zero-shot | zero-shot | 57.8 | 2.62 | |
| Claude Opus 5.5 (low effort) · retrieval 5-shot | retrieval 5-shot | 61.4 | 3.21 | yes |
| Claude Opus 5.5 (low effort) · random 5-shot | other prompting | 59.7 | 3.49 | |
| Claude Opus 5.5 · zero-shot | zero-shot | 57.6 | 4.71 | |
| GPT-6 Astra · zero-shot | zero-shot | 57.3 | 6.70 |
What each prompting technique adds
Change in normalised chrF++ against the same model's zero-shot run, paired on the same 2,000 sentences. * = the 95% interval excludes 0. Gray = within ±0.25.
lossgain
Table view
| Model | Technique | Δ chrF++ | 95% CI | p (bootstrap) |
|---|---|---|---|---|
| Claude Opus 5.5 (low effort) | retrieval 5-shot | +3.60 | +2.96 to +4.26 | 0.0000 |
| Claude Opus 5.5 (low effort) | random 5-shot | +1.87 | +1.34 to +2.44 | 0.0000 |
| Gemma 4 12B fine-tuned (LoRA, ours) | retrieval 5-shot | +1.27 | +0.69 to +1.82 | 0.0000 |
| Claude Sonnet 4.6 | retrieval 5-shot | +3.38 | +2.82 to +3.99 | 0.0000 |
| Claude Opus 5.5 (low effort) | zero-shot, plain 'Swahili' | -0.12 | -0.54 to +0.28 | 0.5220 |
| Gemma 4 31B | retrieval 5-shot | +2.25 | +1.71 to +2.74 | 0.0000 |
| Claude Sonnet 4.6 | random 5-shot | +1.26 | +0.82 to +1.71 | 0.0000 |
| DeepSeek V3.2 | retrieval 5-shot | +3.18 | +2.55 to +3.83 | 0.0000 |
| Claude Sonnet 4.6 | zero-shot, plain 'Swahili' | +0.63 | +0.31 to +0.97 | 0.0000 |
| Gemma 4 31B | retrieval + glossary | +1.25 | +0.70 to +1.82 | 0.0000 |
| Llama 4 Maverick | retrieval 5-shot | +4.42 | +3.74 to +5.09 | 0.0000 |
| DeepSeek V3.2 | retrieval + glossary | +1.59 | +0.92 to +2.29 | 0.0000 |
| Gemma 4 12B | retrieval 5-shot | +3.85 | +3.22 to +4.47 | 0.0000 |
| Llama 4 Maverick | retrieval + glossary | +3.48 | +2.80 to +4.25 | 0.0000 |
| Gemma 4 31B | glossary | -0.64 | -1.08 to -0.22 | 0.0040 |
| DeepSeek V3.2 | random 5-shot | +0.60 | +0.15 to +1.10 | 0.0100 |
| DeepSeek V3.2 | zero-shot, plain 'Swahili' | +0.35 | -0.05 to +0.72 | 0.0880 |
| Gemma 4 12B | retrieval + glossary | +2.17 | +1.50 to +2.83 | 0.0000 |
| DeepSeek V3.2 | glossary | -0.58 | -1.12 to +0.01 | 0.0520 |
| Llama 4 Maverick | glossary | -0.10 | -0.68 to +0.47 | 0.6880 |
| Gemma 4 12B | glossary | -0.82 | -1.39 to -0.27 | 0.0060 |
Where the systems differ
Corpus chrF++ (normalised) on slices of the test split. Sentences per slice: swc < 25 chars 328, 25–50 849, 50–80 471, ≥ 80 chars 352, kit5k 403, kit10k 794, kit15k 803, questions 160, with negation 429, with a number 124, with a name 294, French loanwords 200, vous 211. Long sentences are where the gap between systems is widest — and interview speech is long.
lowerhigher chrF++
Table view
| System | swc < 25 chars | 25–50 | 50–80 | ≥ 80 chars | kit5k | kit10k | kit15k | questions | with negation | with a number | with a name | French loanwords | vous |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 (low effort) · retrieval 5-shot | 55.8 | 61.4 | 62.4 | 61.8 | 63.9 | 59.8 | 61.5 | 58.3 | 62.6 | 67.9 | 65.2 | 64.8 | 57.4 |
| Gemma 4 12B fine-tuned (LoRA, ours) · retrieval 5-shot | 57.0 | 60.4 | 59.7 | 55.7 | 58.9 | 57.5 | 58.6 | 56.7 | 59.2 | 62.3 | 60.9 | 60.8 | 58.9 |
| Claude Sonnet 4.6 · retrieval 5-shot | 54.4 | 58.0 | 59.5 | 57.4 | 59.5 | 57.0 | 57.9 | 54.2 | 57.6 | 64.4 | 62.3 | 61.0 | 55.7 |
| Claude Opus 5.5 (low effort) · zero-shot | 52.2 | 56.6 | 59.8 | 58.3 | 60.8 | 55.6 | 58.1 | 53.0 | 58.6 | 65.0 | 62.9 | 61.0 | 51.8 |
| GPT-6 Astra · zero-shot | 51.7 | 56.9 | 59.6 | 57.0 | 58.6 | 55.7 | 58.2 | 51.8 | 58.2 | 63.4 | 61.2 | 59.6 | 49.9 |
| Gemma 4 12B fine-tuned (LoRA, ours) · zero-shot | 54.0 | 58.1 | 58.6 | 55.5 | 58.7 | 55.7 | 57.1 | 52.5 | 57.1 | 63.1 | 60.6 | 60.3 | 54.4 |
| Gemma 4 31B · retrieval 5-shot | 53.5 | 57.2 | 57.3 | 54.9 | 57.4 | 54.6 | 56.8 | 54.1 | 56.8 | 61.6 | 59.9 | 58.3 | 53.2 |
| DeepSeek V3.2 · retrieval 5-shot | 52.1 | 57.0 | 56.7 | 54.6 | 56.6 | 54.8 | 56.1 | 53.1 | 57.0 | 61.1 | 60.0 | 58.6 | 54.0 |
| Llama 4 Maverick · retrieval 5-shot | 51.4 | 56.2 | 55.5 | 53.8 | 55.4 | 54.4 | 54.8 | 52.9 | 55.6 | 60.6 | 58.1 | 58.4 | 53.9 |
| NLLB-200 3.3B | 44.2 | 50.1 | 51.6 | 50.1 | 52.9 | 49.3 | 48.9 | 45.5 | 50.4 | 56.4 | 55.0 | 52.8 | 54.1 |
| TranslateGemma 27B | 45.4 | 49.5 | 51.9 | 48.6 | 50.7 | 48.8 | 49.5 | 46.4 | 50.2 | 55.6 | 54.2 | 52.7 | 50.4 |
Where retrieval helps
Change in normalised chrF++ from retrieval 5-shot against the same model's zero-shot run, paired on each slice's sentences. * = the 95% interval excludes 0. For every model except Opus 5.5 the gain shrinks on long sentences (the short-vs-long difference is significant for Sonnet 4.6, DeepSeek V3.2 and Gemma 4 31B; RESULTS.md), and on sentences that mix in French it is small or nil for both Gemma 4 models. Interview speech is both long and code-switched.
Table view
| Model | swc < 25 chars | 25–50 | 50–80 | ≥ 80 chars | questions | with negation | with a number | with a name | French loanwords | vous |
|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 (low effort) | +3.64 [+1.44, +5.95] | +4.71 [+3.51, +5.85] | +2.56 [+1.45, +3.76] | +3.46 [+2.46, +4.55] | +5.31 [+2.89, +7.68] | +4.03 [+2.78, +5.42] | +2.93 [+0.87, +4.91] | +2.29 [+1.06, +3.47] | +3.84 [+2.33, +5.20] | +5.65 [+3.65, +7.89] |
| Gemma 4 12B fine-tuned (LoRA, ours) | +2.97 [+0.68, +5.47] | +2.32 [+1.12, +3.56] | +1.15 [-0.11, +2.40] | +0.26 [-0.56, +1.04] | +4.17 [+1.41, +7.05] | +2.05 [+1.01, +3.08] | -0.79 [-2.44, +0.81] | +0.35 [-0.81, +1.52] | +0.54 [-0.85, +2.10] | +4.51 [+2.53, +6.59] |
| Claude Sonnet 4.6 | +6.54 [+3.90, +9.27] | +4.23 [+3.05, +5.37] | +2.88 [+1.78, +4.17] | +2.56 [+1.73, +3.39] | +5.24 [+2.63, +7.83] | +3.25 [+2.16, +4.48] | +2.16 [+0.61, +3.63] | +2.67 [+1.56, +3.96] | +3.03 [+1.56, +4.57] | +5.78 [+3.81, +7.92] |
| Gemma 4 31B | +5.27 [+3.00, +7.50] | +3.78 [+2.63, +4.81] | +1.42 [+0.35, +2.54] | +1.16 [+0.47, +1.83] | +3.29 [+0.84, +5.79] | +2.64 [+1.62, +3.70] | +1.53 [-0.07, +3.15] | +0.87 [-0.07, +1.83] | +0.53 [-0.80, +2.00] | +5.19 [+3.38, +7.07] |
| DeepSeek V3.2 | +7.91 [+5.36, +10.73] | +5.50 [+4.13, +6.84] | +2.53 [+1.23, +3.86] | +1.09 [+0.27, +1.98] | +5.57 [+2.79, +8.56] | +3.21 [+2.03, +4.47] | +0.41 [-1.05, +1.76] | +2.45 [+1.22, +3.80] | +1.87 [+0.20, +3.62] | +5.37 [+3.21, +7.89] |
| Llama 4 Maverick | +8.69 [+6.08, +11.33] | +6.21 [+4.94, +7.39] | +3.71 [+2.45, +5.02] | +2.84 [+1.83, +3.81] | +7.89 [+5.19, +10.91] | +4.23 [+2.90, +5.60] | +3.00 [+1.13, +4.83] | +2.47 [+1.00, +3.98] | +3.96 [+2.03, +5.82] | +6.08 [+3.61, +8.53] |
| Gemma 4 12B | +8.52 [+6.04, +11.09] | +6.26 [+4.96, +7.48] | +4.07 [+2.90, +5.19] | +1.13 [+0.27, +1.94] | +5.91 [+2.91, +9.04] | +4.08 [+2.92, +5.29] | +1.39 [-0.12, +2.86] | +1.92 [+0.84, +3.06] | +2.76 [+1.21, +4.41] | +9.09 [+6.86, +11.53] |
Meaning errors, graded by an LLM judge
openai.gpt-oss-120b-1:0 graded the worst error in each output on 500 test sentences (template severity-v2#ab97e76ce0c8): critical = meaning inverted or invented, or no translation; major = meaning changed. Differences are paired against the first row. The judge is not validated against a human: read the rates as relative between systems, not as error counts. Retrieval cuts Opus 5.5's critical errors (2.2% to 0.6%), the only rate under 1%, but not its major-or-worse rate.
criticalmajor or worse
Table view
| System | Critical % | Major or worse % | Δ critical vs reference | Δ major+ vs reference |
|---|---|---|---|---|
| Claude Opus 5.5 (low effort) · retrieval 5-shot | 0.6 | 14.2 | reference | reference |
| Claude Sonnet 4.6 · retrieval 5-shot | 2.2 | 20.2 | +1.6 [+0.4, +3.0] | +6.0 [+2.8, +9.6] |
| DeepSeek V3.2 · retrieval 5-shot | 2.8 | 21.0 | +2.2 [+0.8, +3.8] | +6.8 [+3.0, +10.4] |
| Gemma 4 31B · retrieval 5-shot | 2.2 | 18.6 | +1.6 [+0.2, +2.8] | +4.4 [+1.0, +7.8] |
| Claude Opus 5.5 (low effort) · zero-shot | 2.2 | 14.6 | +1.6 [+0.4, +2.8] | +0.4 [-2.8, +3.6] |
Do the metrics agree?
Each mark is one system. Rank agreement (Kendall τ): chrF++ vs COMET-22 0.90 over 44 systems; chrF++ vs BLEU 0.88. COMET is a second opinion, not ground truth: its Swahili comes from XLM-R pretraining, not Congolese Swahili.
Frontier APIOpen-weight LLMDedicated MT
Table view
| System | chrF++ | COMET-22 × 100 | COMET 95% CI |
|---|---|---|---|
| Claude Opus 5.5 (low effort) · retrieval 5-shot | 61.4 | 84.2 | 83.7–84.7 |
| Claude Opus 5.5 (low effort) · random 5-shot | 59.7 | 83.6 | 83.1–84.1 |
| GPT-6 Astra · zero-shot | 57.3 | 82.8 | 82.3–83.4 |
| Gemma 4 12B fine-tuned (LoRA, ours) · retrieval 5-shot | 58.2 | 82.8 | 82.3–83.4 |
| Claude Opus 5.5 · zero-shot | 57.6 | 82.7 | 82.2–83.3 |
| Claude Opus 5.5 (low effort) · zero-shot | 57.8 | 82.7 | 82.1–83.2 |
| Claude Sonnet 4.6 · retrieval 5-shot | 57.9 | 82.5 | 82.0–83.1 |
| Gemma 4 12B fine-tuned (LoRA, ours) · zero-shot | 57.0 | 82.4 | 81.9–83.0 |
| Claude Opus 5.5 (low effort) · zero-shot, plain 'Swahili' | 57.7 | 82.3 | 81.8–82.9 |
| Gemma 4 31B · retrieval 5-shot | 56.1 | 82.0 | 81.4–82.5 |
| Gemma 4 31B · retrieval + glossary | 55.1 | 81.4 | 80.8–81.9 |
| Claude Sonnet 4.6 · random 5-shot | 55.8 | 81.3 | 80.8–81.9 |
| DeepSeek V3.2 · retrieval 5-shot | 55.7 | 81.1 | 80.5–81.7 |
| Llama 4 Maverick · retrieval 5-shot | 54.8 | 80.9 | 80.3–81.5 |
| Claude Sonnet 4.6 · zero-shot, plain 'Swahili' | 55.2 | 80.5 | 79.9–81.1 |
| Claude Sonnet 4.6 · zero-shot | 54.6 | 80.4 | 79.9–80.9 |
| Llama 4 Maverick · retrieval + glossary | 53.8 | 80.4 | 79.8–81.0 |
| Gemma 4 31B · zero-shot | 53.9 | 80.3 | 79.7–80.9 |
| Claude Sonnet 5.5 · zero-shot | 53.6 | 80.2 | 79.5–80.8 |
| DeepSeek V3.2 · retrieval + glossary | 54.1 | 80.1 | 79.5–80.7 |
| Gemma 4 31B · glossary | 53.2 | 80.0 | 79.4–80.5 |
| Gemma 4 12B · retrieval 5-shot | 53.9 | 79.7 | 79.2–80.4 |
| DeepSeek V3.2 · random 5-shot | 53.1 | 79.4 | 78.8–80.0 |
| TranslateGemma 27B | 49.5 | 79.1 | 78.5–79.6 |
| Gemma 4 26B-A4B · zero-shot | 51.8 | 78.7 | 78.1–79.3 |
| DeepSeek V3.2 · zero-shot, plain 'Swahili' | 52.9 | 78.6 | 77.9–79.2 |
| DeepSeek V3.2 · zero-shot | 52.5 | 78.5 | 77.9–79.1 |
| Gemma 4 12B · retrieval + glossary | 52.2 | 78.2 | 77.6–78.8 |
| TranslateGemma 12B | 47.2 | 78.0 | 77.4–78.5 |
| DeepSeek V3.2 · glossary | 51.9 | 77.9 | 77.3–78.5 |
| Llama 4 Maverick · zero-shot | 50.4 | 77.9 | 77.3–78.5 |
| Llama 4 Maverick · glossary | 50.3 | 77.6 | 77.0–78.2 |
| NLLB-200 3.3B | 50.1 | 77.0 | 76.3–77.6 |
| Gemma 4 12B · zero-shot | 50.1 | 76.9 | 76.3–77.5 |
| NLLB-200 1.3B | 49.3 | 76.9 | 76.2–77.5 |
| Gemma 4 12B · glossary | 49.2 | 76.0 | 75.4–76.7 |
| Gemma 4 E4B · zero-shot | 48.3 | 75.3 | 74.6–75.9 |
| NLLB-200 600M | 46.4 | 74.4 | 73.8–75.1 |
| TranslateGemma 4B | 40.4 | 73.1 | 72.5–73.8 |
| MADLAD-400 3B | 44.8 | 72.9 | 72.3–73.6 |
| MADLAD-400 10B | 45.2 | 72.9 | 72.2–73.7 |
| Gemma 4 E2B · zero-shot | 43.4 | 71.1 | 70.4–71.8 |
| opus-mt swc-fr | 32.7 | 60.3 | 59.6–61.0 |
| SeamlessM4T v2 | 37.0 | 57.1 | 56.2–57.9 |
What these numbers do not show
- Domain. Short, clean, general Tatoeba sentences, not long code-switched interview speech; ranks can change across that gap.
- Variety and input. CLEAR Global's written Congo Swahili, not spoken Ituri/Nord-Kivu; clean text, not ASR output (~0.74 WER in ebola-asr).
- Metrics. chrF++, COMET and the LLM judge are automatic. No human evaluation was run. Salient-error checks in RESULTS.md are noisy, entities especially.
- Retrieval helps only when the example pool resembles the input. A Tatoeba pool for interview speech may help far less.
- Data handling. Public data only. Whether interview text may go to any API is a question for Eva/Nina.
Provenance
- Data:
CLEAR-Global/Gamayun-kitsrevb80906d, swc-fra kits 5k/10k/15k (25,305 pairs; 24,858 after dedupe). Frozen splits, seedmtsweep-2026-10-15: test 2,000 · dev 1,000 · train 21,858 (manifests indata/splits/). - Prompts:
baseline-v1#c7fdd9e2225f,fewshot-v1#f6f0b02fe229+ retrieval:5 (char 3–5-gram TF-IDF over train), glossary lexicon1fdb363c(IBM-1 on train). Temperature 0 where accepted; output ceiling 512, truncated calls re-run at 4,096. - Fine-tune: Gemma 4 12B-it, LoRA r16, 1 epoch over train, early-stopped on dev loss (3.24 → 0.60), one H100, 20 min.
- Metrics: chrF++ (sacrebleu 2.6, frtypo-v1 normalisation), BLEU, COMET-22 (wmt22-comet-da); 1,000 bootstrap resamples, seed 20261015. Slices
slices-v1; LLM judgeopenai.gpt-oss-120b-1:0,severity-v2, 500-sentence sample (seedmtsweep-judge-v1). - Spend (2026-10-15): about $85 in Bedrock tokens at unconfirmed table prices, $0.94 of it the LLM judge; about $8 of own H100 time; ebola-asr's box was already billing.
- Replicate:
docs/runbook.md. Lessons:docs/lessons.md. Full record:RESULTS.md.