When the model is wrong, what saves you
Hallucination is not solved
Large language models produce confident statements that are not factually correct. The frequency varies by model, by prompt design, and by retrieval architecture, but it is non-zero across the board and probably will remain so for some time. A voice agent that hallucinates an availability slot, a price, or a policy creates an obligation from the customer's point of view. The defensive posture is to ground the model in a small, controlled knowledge source and to refuse to answer outside that source. Practices that skip this step learn the lesson through customer complaints, not through reading about it.
Prompt injection and adversarial input
User input flows into the prompt context, and a sufficiently determined user can manipulate the system. A booking agent can be tricked into giving a discount it should not give, a customer-service agent into revealing the system prompt or another customer's data. The mitigations are layered: validate inputs, separate trusted instructions from untrusted user content, and design for the case where the model is briefly compromised. No single defence is sufficient; the operator should assume the model will occasionally be misled and design the wider system so that misleading it produces a survivable, recoverable outcome.
Escalation paths and the human-in-the-loop
A mature AI services product designs explicit handover points. Confidence below a threshold escalates to a human. Topics on a sensitive list — medical, legal, financial advice, complaints — escalate by default. The practice maintains a small, trained human team that handles edge cases. From the buyer's perspective this is one of the strongest selling points: not 'we have an AI' but 'we have an AI plus a clear path back to a person'. Practices that omit escalation paths typically discover they are needed during their first regulatory or reputational incident.
Audit trails and model drift
Every customer-facing interaction should be logged in retrievable form: the input, the model's reply, the version of the model, the prompt and retrieval state. Two reasons. First, when something goes wrong, the practice must reconstruct what happened to handle complaints, regulator queries, and disputes. Second, models drift — provider updates change behaviour over time, and a deployment that was acceptable in March may be unacceptable in November without anyone consciously changing it. A practice without logs cannot detect this; a practice with logs can run regular regression checks and react quickly.
Real-world parallel — Moffatt v. Air Canada (2024)
In November 2022, Jake Moffatt visited the Air Canada website to book travel to attend his grandmother's funeral. The airline's customer-service chatbot told him he could apply for a bereavement-fare discount retroactively after travelling. Air Canada employees later confirmed that this was wrong — the actual policy required the discount to be requested before travel. Moffatt brought a small claim. In February 2024 the British Columbia Civil Resolution Tribunal ruled in his favour, finding Air Canada liable for negligent misrepresentation. The airline's defence had argued that its chatbot was 'a separate legal entity that is responsible for its own actions.' The tribunal rejected this. The decision is now widely cited as the first significant finding that an organisation cannot disclaim responsibility for what its automated agents tell customers. For an AI services practice, the implication is direct: the buyer carries the liability for what your agent says — and by contract, so might you.
If your automated voice agent confidently quotes a wrong price to a customer who then arrives expecting it, who explains the situation? What policy prevents the same error twice?