Voice AI agent that listens and speaks in Arabic and English

What this actually means

Voice is two separate problems wearing one coat. First the assistant has to work out what was said, which is speech recognition. Then it has to say something back in a voice a person is willing to listen to, which is speech synthesis. Both are hard in Arabic for different reasons, and a system that is good at one and weak at the other fails as completely as one that is bad at both.

Recognition is hard because people do not speak the way text is written. They hesitate, restart, run words together, use dialect, and do it against background noise in a car or a shop. Synthesis is hard because listeners are unforgiving of anything that sounds mechanical, and because Arabic pronunciation depends on vowel marks that written text usually omits — the same letters can be read several ways, and only meaning decides which.

The third problem is time. In text, a two-second pause is invisible. In speech it is a silence, and silence on a call reads as a dropped connection. Everything in a voice pipeline is judged against that constraint.

Why two languages makes this harder

Arabic speech carries a problem English does not: the written form usually omits the short vowels that determine pronunciation. A reader supplies them from context. A synthesiser has to do the same, which means it must understand the sentence to pronounce it, not merely transcribe it. Systems that skip that step produce output that is technically Arabic and audibly wrong, and listeners notice immediately even when they cannot articulate why.

Recognition faces the mirror image. People speak dialect — Gulf, Najdi, Hijazi, and the Egyptian and Levantine varieties of a large expatriate workforce — while most Arabic speech models are trained predominantly on broadcast-standard speech. The gap between the two is not an accent; it is vocabulary, structure and rhythm. A system tuned only to formal Arabic will transcribe a Saudi caller badly and then answer the wrong question confidently.

Code-switching finishes the job. Saudi speakers routinely mix English terms into Arabic sentences — product names, technical words, numbers. A pipeline that must choose one language per utterance will mangle every sentence containing both, which in practice is most of them.

What good looks like

The failure mode worth asking about

The failure specific to voice is compounding error. A word is misheard; the misheard word changes the apparent question; the assistant answers that different question confidently and fluently. The caller now has to work out that they were misunderstood, interrupt, and start again — and the fluency of the wrong answer makes that harder, because it sounds like the system understood.

This is why confirmation matters more in voice than in text. Where a request has consequence, reading the understanding back before acting costs two seconds and prevents the entire class of failure. Systems that skip confirmation to sound faster fail more expensively and less visibly.

The second failure is refusing to let go. A voice assistant that will not escalate, or that makes escalation deliberately hard, converts a mild annoyance into a complaint. The measure of a good voice deployment is not how few calls reach a human — it is how few callers had to fight to reach one.

How this fits industry practice

The direction of travel across the industry is consistent and worth understanding before you evaluate anyone. Assistants have moved from scripted decision trees, to retrieval over a knowledge base, to systems that can take actions against connected systems. Each step added capability and added a category of risk, and the risks did not replace each other — they accumulated.

Scripted systems were safe and useless: they could only answer what someone had anticipated, so they frustrated everyone and were quietly abandoned. Retrieval systems answer far more but introduce the possibility of answering wrongly with confidence. Action-taking systems answer and do, which raises the stakes again — an incorrect answer is embarrassing, an incorrect action is a liability.

The mature position, and the one worth insisting on, is that capability should be earned rather than assumed. Ground answers in material the organisation controls. Check the finished output rather than trusting an instruction given beforehand. Keep actions narrow, reversible and reviewable. Escalate rather than improvise. None of that is exotic; it is simply what separates a deployment that survives its first year from one that gets switched off after an incident.

The specific thing this market adds is language. Nearly every published benchmark, best practice and vendor claim was developed against English. A capability that performs well in English and adequately in Arabic is not bilingual — it is an English product with Arabic support, and the difference shows up precisely where it costs most: in the questions your customers actually ask, in the words they actually use.

Measuring whether it is actually working

Most assistant deployments are measured on the wrong number. Containment — the share of conversations resolved without a person — is easy to report and easy to improve for the wrong reasons. An assistant that makes escalation difficult will show excellent containment and a deteriorating relationship with customers, and the second effect takes months to surface while the first appears on a dashboard immediately.

The numbers worth watching are the ones that expose deferred problems. Repeat contact within forty-eight hours tells you whether a resolved conversation actually resolved anything. Abandonment mid-conversation tells you where people gave up. The share of escalations in which the customer had to repeat information already given tells you whether handover is working or merely happening. None of these flatter the system, which is precisely why they are useful.

Two more are worth instrumenting from the start. The first is grounding — what share of answers can be traced to a source document. In a regulated sector this is the number an auditor will eventually ask for, and retrofitting the ability to answer it is painful. The second is language accuracy: whether the reply came back in the language of the question. It sounds trivial and it is the single most common complaint about bilingual assistants in this market.

Set a baseline before launch. Without one, every improvement is an assertion. Take a month of existing contact, categorise it, count it, and time it. That baseline is what makes the business case defensible six months later when somebody senior asks what the assistant actually changed.

Getting this right in practice

Start with the calls you already receive. Pull a week of recordings, categorise them, and count. The top five categories are almost always factual questions with a single correct answer, and they are almost always more boring than anyone expects. That list is your deployment scope.

Set the escalation rule before you set anything else. Decide what the assistant will never attempt — complaints, distress, anything involving money moving — and make reaching a person easy and obvious rather than buried. Deployments that optimise for containment first produce a metric that improves while satisfaction falls, and the metric is the thing people notice last.

Then measure the right things. Containment matters, but abandonment, repeat calls within 48 hours and the share of escalations where the caller had to repeat themselves tell you far more about whether the deployment is working. A system with high containment and high repeat contact is not solving problems; it is deferring them.