Congolese Swahili ASR · WHO DRC Ebola interviews · 15 October 2026
Do Gemma 4 or Whisper beat Omni on Congolese Swahili?
No. On 2,016 clips of public Congolese Swahili read speech, our current model (Meta's Omnilingual ASR 7B) makes the fewest word errors. Gemma 4 E4B is a clear second, 3.8 points behind. Whisper large-v3 is 10 points behind. Gemma 4 12B does not treat the audio as Swahili at all. We keep Omni.
Word error rate by model
Share of reference words the model got wrong. Lower is better. Dots are the measured rate; bars are the 95% interval. Hollow dots were measured on the 667-clip dev split only, because we stopped those runs once dev had settled the question.
Show as table
| Model | Clips | WER | 95% interval | Character error rate |
|---|
How far behind Omni, in words and in characters
The difference from Omni on the same clips, resampled together so clip difficulty cancels out. Every interval is clear of zero. The character gaps are much smaller than the word gaps, because about half of every model's wrong words are near-misses of the right one — Omni's included.
Show as table
| Model | WER gap | 95% interval | CER gap | 95% interval |
|---|
What each model gets wrong
The reasons matter more than the ranking, because they tell us which errors a human reviewer would catch.
Every model "corrects" Congolese spelling — Omni most
Where the speaker said a Congolese form (serkali, inchi, ao), Omni wrote the standard Swahili one 91% of the time, Gemma 4 E4B 78% and Whisper 42%. The output reads fluently, so a reviewer is unlikely to notice. This is the largest single source of Omni's errors, and the one a spelling table could fix.
The prompt is a real lever
Our verbatim prompt beat Google's suggested prompt by 1.4 points (interval 0.9–1.9) on the dev split. Omni has no equivalent setting, so this is the reason to keep E4B in reserve.
Whisper keeps the dialect but loses the words
It keeps Congolese forms far more often than the others (36% against Omni's 2%), but mishears and merges words (Maramingi, ilimu), and is worst on short clips. On clips over 30 seconds it loops: 58 of 423 came back more than twice the length of what was said.
Gemma 4 12B does not treat the audio as Swahili
It follows the sounds but writes them as English, French or Japanese script, and sometimes invents fluent content ("governments of the United Nations").
Examples
Three clips from the public test set, with every model's output. Highlighted words differ from what was read.
What this does and does not tell us
- It is a best case, not interview accuracy. The clips are professional voice talent reading prepared COVID-19 text on phones. The interviews add spontaneous speech, overlap and French code-switching. On a human-checked interview excerpt, every model scores far worse.
- It is still the right test for ruling models out. The audio is in the right variety and the right channel. A model that loses on clean speech has no clear route to winning on harder speech.
- No interview audio left Kobo's infrastructure. Every model ran self-hosted on the project's DigitalOcean GPU, on public data only.
- Gemma 4 only accepts 30 seconds per request, and cuts the rest silently. The 423 longer clips are excluded for every model, so the comparison is like for like.
Recommendation
- Keep Omni 7B as the primary model.
- Keep Gemma 4 E4B in reserve, as the only model we can steer with a prompt. Its next test is the human-checked interview excerpt, not more read speech.
- Stop spending GPU time on Gemma 4 12B and Whisper large-v3 for this language.