Direct Twi selection can remove a translation layer only if it matches or beats translation-first selection on useful answers, safe refusals, and correct human escalation. For cocoa advice, a fluent response is not enough: the system must select the right reviewed answer block without adding unsafe advice or hiding uncertainty.
Define the two paths before testing
AgriVoice has two ways to route a farmer’s spoken Twi question to reviewed cocoa guidance.
The translation-first path transcribes the question, translates it to English, then selects from approved answer blocks. The direct-Twi path transcribes the question and asks the model to select those same blocks from Twi alone.
Both paths must end in the same place: reviewed content, spoken back to the farmer, with a human escalation when the answer is missing or uncertain. The model selects block IDs. It does not write agronomy advice.
This boundary matters because translation can introduce a serious domain error. A previous translation rendered `kokoo`, cocoa, as “chicken.” A sentence can sound natural and still point the system toward the wrong answer.
Build a test set that includes the hard questions
A useful evaluation needs about 40 real Twi farmer questions, labelled by a Twi-speaking agriculture reviewer. Include common questions, ambiguous wording, pesticide questions, out-of-domain requests, English-Twi code-switching, corrupted speech-recognition transcripts, and questions that need more than one approved block.
Do not fill the set with easy disease-identification prompts and call the result a safety test. Include questions where the correct outcome is, “I need to connect you to an extension officer.”
For each question, record the approved block set or the correct escalation outcome. Ask a second reviewer to resolve disputed labels where possible. Forty cases will give a directional result. A one-question difference between systems does not prove they perform the same.
Measure selection quality and safety separately
Exact-set accuracy matters when a question needs a particular combination of reviewed blocks. Any-correct-block recall also matters, because a system may identify one useful block but omit another required part of the answer.
Neither measure can stand alone. Track these safety outcomes alongside them:
- Unsafe extra-block rate: how often the system selects guidance that should not be included.
- Out-of-domain accuracy: how often it avoids pretending to answer a question outside the reviewed content.
- Escalation accuracy: how often it sends uncertainty to the named human owner.
- Confidence: whether low confidence actually lines up with cases that need escalation.
- Latency and marginal cost: how long a completed answer takes and what each component adds.
A direct-Twi system that answers more questions but adds an irrelevant pesticide block has failed the comparison. The same applies to a system that sounds certain when it should open a human review.
For the chemical blocks, the standard is stricter. Advice must retain the approved safety content, including dosage, re-entry intervals, and pre-harvest intervals where the agronomist has verified them. If those details are absent, AgriVoice should refuse the instruction and escalate. Cocoa pesticide advice: Why AgriVoice Stops When Dosage Details Are Missing explains why that refusal protects farmers.
Test the transcription farmers will actually produce
A clean written Twi question is useful for diagnosis. It does not prove the voice workflow works in the field.
Run both selection paths first on reviewer-checked text, then repeat the comparison on transcripts from real farmer voice notes. This exposes a different failure mode: the speech recognizer may mishear a crop term, product name, location, or timing detail before either selection path begins.
Collect at least 20 real farmer questions with consent, transcription, and tested deletion behavior. Keep the original audio only as long as the consent and retention policy permits. Compare how each path behaves when a transcript is incomplete, code-switched, or wrong in a way that could change the advice.
If a transcript turns “harvest Friday” into a different time reference, the right response may be escalation rather than a confident answer. What Happens When a Cocoa Question Needs a Human Answer? covers the operational side of that handoff.
Decide using a safety gate, not a preference
Remove translation from AgriVoice selection only when direct Twi matches or beats translation first on the agreed task metrics without worse unsafe selection or escalation behavior.
The decision should be written down with the test set, labels, results, reviewer notes, model settings, latency, and cost. This makes the result repeatable when content changes or a new speech-recognition model is introduced.
Translation removal has a practical upside. It can reduce delay, avoid a translation failure point, and remove a non-commercial translation model from a future paid path. It also gives up a familiar English intermediate representation that reviewers may find easier to inspect. The evaluation must show that the trade is safe.
Run the next concrete test
Start by collecting 40 questions and assigning gold block IDs before choosing a winner. Then run direct Twi and translation first against the same candidate blocks, repeat with real ASR transcripts, and review every unsafe extra block and missed escalation by hand.
A farmer should receive reviewed cocoa guidance, or a clear route to a person who can help. That is the result the test must protect.
Comments
No comments yet.