A cocoa-sector partner should continue only when the two-week pilot shows safe, understood answers that farmers use again, with accountable escalation, workable response time, and a measured cost per completed question. If any unsafe pesticide instruction reaches a farmer, the partner should stop or redesign before expanding.
At 4:40 on a Friday afternoon, Abena stands at the edge of her cocoa plot with a phone in one hand and a leaf marked by dark patches in the other. Rain has started again. She records a question in Asante Twi, then waits for the reply through the speaker.
The response needs to do one of two things: give her a reviewed, understandable next step, or send the question to the named extension officer. A confident guess could lead her to treat the wrong problem, spend money on the wrong input, or put herself at risk. Silence is no better when the crop is already under pressure.
That is why the scorecard matters. It turns a pilot from an encouraging collection of voice notes into a decision the partner can defend.
Start with safety, then count successful outcomes
The first question is simple: did every pesticide-related response remain within reviewed advice, or escalate when it could not?
AgriVoice is designed to select reviewed content rather than invent agronomy guidance. Some chemical blocks remain withheld because verified dosage, re-entry, or pre-harvest details are missing. That restraint is part of the product. A farmer should hear “this needs an extension officer” before hearing an unverified instruction.
For the pilot, the target is zero unsafe pesticide answers. One unsafe or unreviewed instruction reaching a farmer is a stop or redesign signal, even if the rest of the numbers look strong. Safety is the condition that lets every other measure matter.
Then assess the successful answer rate. A successful outcome includes a correct reviewed answer and a correct escalation. The pilot’s continue signal is at least 70% answered or correctly escalated. Questions that are unclear, outside the content pack, or uncertain after speech recognition should not be forced into a nearby answer.
Abena’s question may be answered immediately if it matches a reviewed cocoa-care block. If the wording points to a pesticide product whose name is uncertain, the safe result is an escalation. A safe farm decision must wait when a pesticide interval is missing.
Check whether farmers understand and return
A response that arrives in Twi still fails if the farmer cannot follow it. The pilot asks whether at least 80% of participants report understanding the response. This needs to be gathered plainly, in language farmers can use to say what they understood, what sounded unclear, and what they would do next.
Comprehension also includes the voice itself. The current Asante Twi voice is suitable for the pilot demo, but it sounds formal and has not met the standard for broad narration. If farmers routinely struggle to understand or accept it, the partner has learned something useful: the next iteration must address voice or language before recruitment grows.
Repeat use is the other test of practical value. The continue signal is at least 30% of activated farmers asking another question in week two. One question can be curiosity. A second question suggests the service has earned a place in a real farming decision.
On the following Tuesday, Abena sends another voice note. This time she wants to know what to look for before acting, rather than asking the neighbour who sold her an input last season. That second message carries more weight than a polite answer in a closing interview. It shows that the first response was clear enough to become part of her routine.
Treat the escalation queue as part of the service
A named extension officer must own escalations before the first farmer joins. The scorecard should show who received each escalation, when it was assigned, and when the farmer received a response.
The benchmark is a median response time under one working day. A queue without an accountable owner creates a dangerous gap: the app has correctly refused to guess, but the farmer still has no route to a person who can help. Repeated unresolved cases are a stop or redesign signal.
The partner should review the queue by category. Are pesticide questions being escalated because details were wisely withheld? Are questions missing the reviewed content? Is speech recognition turning a clear question into something ambiguous? Those are different problems and call for different changes.
The same discipline applies to latency. The target is a median end-to-end response below eight seconds, excluding human escalation. A voice conversation loses its shape when every turn feels like a long wait. Measure the full path, from the farmer’s voice note to the spoken response, rather than only measuring a single model.
Make one decision from the whole scorecard
At the end of two weeks, the partner should place the six measures beside one another: safe answers, successful outcomes, comprehension, repeat use, escalation operations, response time, and cost per completed question.
Continue when safety holds, the thresholds are met, the escalation owner responds, and the component costs can be stated clearly. Iterate once when evidence points to one contained repair, such as a misunderstood voice, a weak content area, or a slow response path. Stop or redesign when unsafe advice appears, wrong answers arrive confidently, farmers cannot understand the replies, or the human escalation route does not work.
Cost deserves the same honesty. Record WhatsApp, speech recognition, language-model selection, speech output, hosting, and human escalation separately. A partner cannot assess a plausible future price from a single blended number that hides the expensive part.
Abena’s second voice note should leave the pilot with a trace the partner can inspect: what she asked, whether the system selected reviewed advice or escalated, how long the reply took, whether she understood it, and what that completed interaction cost. That is enough to make the next decision with evidence instead of optimism.
Comments
No comments yet.