Gemma 4 and Whisper against Omnilingual ASR on TICO-19 Congolese Swahili

1. Setup

Gemma and Whisper ran through transformers 5.19.0 and torch 2.14.1+cu130, in a separate venv behind the same newline-delimited JSON worker protocol as Omni. Every clip was decoded by ffmpeg to 16 kHz mono and written as s16 PCM WAV before any model saw it. Hardware: one H100 80 GB, shared by up to four arms at once. So throughput figures are not comparable, but outputs are, because each arm decodes at a fixed batch size.

Gemma prompts (versioned; the results file records which)

2. Data and metric

• Corpus. CLEAR Global TICO-19 Congolese Swahili speech: dev 868 + test 2,124 clips, 15.16 h, eight DRC voice talents (4 F, 4 M), every sentence read by two voices. The dev split is two speakers only (sessions 86 and 87). CC BY-NC 4.0, evaluation only.
• Subset. Clips ≤30 s (2,569 of 2,992), because Gemma 4 accepts at most 30 s. The headline also drops the 553 clips whose reference contains a digit, leaving 2,016 clips and 1,029 sentences. Readers speak numbers as words, the references write some as digits, and the number canonicaliser does not know the Congolese number forms.
• Normalisation. unintelligible → language_tags → markers → speaker_labels → spoken_numbers → tokenise(lowercase, punctuation), identical for every arm. Parenthesised digit restatements in references are removed.
• WER and CER are ratios of sums over the corpus, not means of per-clip rates. CER aligns characters with words rejoined by single spaces, so a run-together word costs one deletion.
• Intervals. 95% percentile (nearest rank), 2,000 resamples, seed 20260913. The unit is the sentence, not the clip: the two readings of a sentence share its difficulty, so resampling clips would understate the width.
• Deltas are paired. One sentence draw goes to both arms, so between-sentence difficulty cancels. The scorer refuses (delta_unpaired) if the two arms' per-sentence denominators differ, which is what catches two arms scored on different subsets.

3. Results

Paired difference from Omni (same clips, same sentence draws)

E4B vs Whisper, paired: Δ WER −0.0632 [−0.0695, −0.0565], Δ CER −0.0091 [−0.0107, −0.0075]. Prompt A/B on dev, google-original-language/v1 minus swc-verbatim/v1: Δ WER +0.0143 [+0.0093, +0.0191] digit-free, but +0.0029 [−0.0034, +0.0091] on all clips. Google's prompt asks for digits, which the digit-bearing references reward.

By split, voice and speaker (digit-free WER)

The dev split is harder than test for Omni (0.1885 vs 0.1611), so the two dev-only arms' gaps are always quoted against Omni on dev.

4. Error decomposition

Near-miss: character edit distance ≤ max(1, ⌊|ref word| / 4⌋). The near-miss share is flat across the three working arms, so E4B's deficit is volume. It makes 27% more substitutions and 80% more insertions than Omni, with deletions level. It isn't making a different kind of error.

5. Congolese forms: kept, standardised, or otherwise wrong

The 1,081 reference occurrences of 12 Congolese forms (ao, inchi, inchini, serkali, iyi, muripuko, kuzuwia, muzima, muzuri, zarura, tafazali, mupya), by what each arm aligned to them. "Standardised" means substituted by the standard Swahili form (au, nchi, serikali, hii, mlipuko, kuzuia, mzima, …).

Show as table, with every form

Form	Occurrences	Omni kept / std	E4B kept / std	Whisper kept / std

Selection bias, stated. The 12 forms were read off Omni's most frequent substitutions in BUILD-STATE §21.4, which favours finding Omni standardising. That can't produce a 1.7% vs 35.6% retention gap on its own, but a form list built independently (from a native speaker, or from all three arms' errors) is the clean version of this measurement. Omni's lead on WER holds despite this: §21.4 put the in-sample upper bound of canonicalising such pairs at 3.7 WER points for Omni on TICO.

6. Error by clip length: Whisper's two failure regimes

All 2,992 clips, digit-free WER by duration bucket. The label on each Whisper point counts its loops: outputs more than twice the reference length.

Show as table

Duration (s)	Clips	Omni WER	Whisper WER	Whisper loops	Omni loops

Whisper is worst on short clips (0.334 at ≤10 s, against 0.248 at 20–30 s) while Omni is flat. Above 30 s, Whisper's sequential long-form decoding loops on 58 of 423 clips. The production pipeline chunks at ≤28 s, so the long-form regime would not arise there; the short-clip one would.

7. Most frequent substitutions

Reference → hypothesis, digit-free ≤30 s, with counts. ya → wa in E4B's list is a real grammatical error (noun-class agreement), not an orthographic variant, so it must never be forgiven by a variant table. Whisper's distinctive errors are phonetic or word-boundary ones (kifua → kifuwa, hatari → atari, wafanyakazi → kazi).

8. Gemma 4 12B: output pathology

It hears the audio. Its output tracks the phones (Miji za kuanza za zeni, zili tumia hisi ya tua for Miji za kwanza zenye zilitumia hizi hatua), but it renders them as English words, French sentences or katakana, and sometimes invents fluent content (serikali za Umoja wa Mataifa, "governments of the United Nations"). Same worker, processor path and audio as E4B, which writes Swahili on the same clips.

9. Threats to validity

• Domain. Professional read speech of translated COVID-19 text: no overlap, hesitation or code-switching. These are floors on interview WER: on a human-referenced interview excerpt, every arm scores far worse (BUILD-STATE §21).
• Possible text contamination. TICO-19's text has been public since 2020, and the audio since IWSLT 2021. An LLM decoder that has seen the text is advantaged, so if anything this inflates Gemma, and its loss is conservative. Omni's training data is not public enough to say either way.
• The 12B checkpoint. Google's audio guide says "Gemma 4 E2B, E4B, and 12B Unified are trained for multilingual speech recognition" and lists google/gemma-4-12B-it as the 12B ID, which is what ran. If a distinct "Unified" 12B checkpoint exists, this result does not describe it. E2B was not run.
• Dev-only arms rest on two speakers. The 12B and Google-prompt arms were stopped after dev, which is sessions 86 and 87 only. Their gaps are large relative to their intervals, but the intervals cover sentence sampling, not speaker sampling.
• Narrow search. Two prompts, greedy decoding only, no beam search, no Whisper initial_prompt, and no fine-tuning of any arm. The ranking is of models as configured, not of their ceilings.
• Batch size is part of an arm's identity for Gemma. bf16 padding changes outputs between batch 1 and 2 (puwa → puya observed), so Gemma ran at batch 1 throughout. Omni ran at the production batch 2.
• Reference defects. 6 TICO clips are length-mismatched against their references (§21.4). They are not removed here and affect all arms equally.
• Gemma's 30 s ceiling is silent. In transformers 5.19.0 the processor returned exactly 750 audio tokens (25/s × 30 s) for a 42.24 s clip, with no warning. The worker refuses longer chunks; any other Gemma evaluation should check its own input lengths.

10. Reproduce

Code at commit a600db7 and later. BUILD-STATE §22.4 has the hand-built /opt/hf venv recipe.

./scripts/swc-speech-pull tico
./scripts/transcribe-clips data/swc-speech-tico tico-gemma4-e4b-swcv1 \
    --card google/gemma-4-E4B-it --prompt-id swc-verbatim/v1 --lang swh_Latn \
    --batch-size 1 --max-seconds 30 \
    --worker-python /opt/hf/.venv/bin/python --worker-script /opt/hf/hf_worker.py
./scripts/transcribe-clips data/swc-speech-tico tico-whisper-lv3 \
    --card openai/whisper-large-v3 --dtype float16 --lang swh_Latn --batch-size 1 \
    --worker-python /opt/hf/.venv/bin/python --worker-script /opt/hf/hf_worker.py
./scripts/asr score_clips data/swc-speech-tico/references.jsonl runs/tico-gemma4-e4b-swcv1 \
    --max-seconds 30 --against runs/tico-swh