Teaching TinyAya to say "I don't know"

A small multilingual model guesses on almost every question it cannot answer. We fine-tuned it to abstain instead, with labels that need no gold answers in the target language. Here are the same 252 test questions, before and after.

Base model

R4 Probe labels

Each square is one test question. Counts are the mean rate over 3 training seeds, scaled to 252 questions; the base model is a single deterministic run.

Four recipes cut hallucination from about 0.8 to between 0.17 and 0.31

Teacher-written traps (R1) help far less: in English, Spanish and Malay they lower it only from about 0.8 to 0.7.

A probe matches gold labels in three of four languages

R4 uses no gold answers in the target language. It is not significantly different from R2 in English, Spanish and Malay.

Abstention transfers across languages only partly

English to Spanish and Malay helps. Arabic and the other pairs move much less.

Every recipe, one language at a time

Each bar is all 252 test questions. A good abstaining model turns red into grey without losing much green.

Answered correctlyAnswered wrongly (hallucination)Said "I don't know"

Over-refusal is the share of questions the base model answered correctly that the tuned model refused. Abstain recall is the share of questions the base model got wrong that the tuned model refused. Means over 3 seeds; the standard deviation is under hallucination.

A probe is as good as gold labels in three of four languages

Paired bootstrap on the test questions, outcomes averaged over seeds. A filled dot means the 95% interval excludes zero. A hollow dot means we cannot tell the two recipes apart.

R4 matches R2 in English, Spanish and Malay. In Arabic R4 is more conservative: 0.040 less hallucination, but also 0.021 fewer correct answers. Against R1, R4 cuts hallucination by 0.32 to 0.52 in every language.

Abstention transfers across languages only partly

Rows are the language an adapter was trained on, columns the language it is tested on. The outlined diagonal is same-language. Seed 1 only.

English-trained adapters help most on Spanish and Malay, about 0.35 less hallucination than the base model. Arabic and the other pairs move much less. Base-model hallucination is 0.80 to 0.85 everywhere, so values near that mean no transfer.

Probe labels rank known questions as well as consistency, or better

AUROC for telling known from unknown questions, with 95% intervals. The probe trains on English gold labels and is transferred; consistency needs at least 4 of 5 sampled answers to agree.

Probe (R4)Consistency (R3)

The intervals overlap in every language, so the fair reading is "comparable or better", not "clearly better". Precision of the known class is only 0.40 to 0.48 for both sources.

What these numbers do not show