Conversational AI for customer service is software that holds a two-way conversation with a customer in their own words, by phone, text or website chat. It answers from knowledge you approve, takes the actions you allow, such as booking or rescheduling, and transfers anything else to a person. How well it does that is settled mostly by what you test before launch.
The failures that have been documented so far look alike. Either the agent told a customer about a policy the business did not have, or it kept someone talking in circles with no way through to a person. The twelve-conversation test set further down is built to catch both before launch, with a pass condition for each test, and you can run it against any vendor's agent, ours included.
What is conversational AI for customer service?
Conversational AI understands what a customer says or types in ordinary language, keeps track of the conversation as it goes, and replies in ordinary language. In customer service, it is what lets a caller say "I need to move Thursday to next week" instead of pressing 2 and then 4. Current systems run on large language models and answer from the business's own material.
Older tools work differently. The Consumer Financial Protection Bureau's 2023 report on bank chatbots describes the rule-based kind: they "use either decision tree logic or a database of keywords to trigger preset, limited responses," and the customer "is typically limited to predefined possible inputs" (CFPB, Chatbots in consumer finance). Conversational AI removes that limit. The same freedom that lets it handle a messy request also lets it say things nobody approved, so testing matters more than it did with menus.
| Phone menu (IVR) | Rule-based chatbot | Conversational AI agent | |
|---|---|---|---|
| What the customer can say | A key press or a short keyword | Menu choices or keywords the bot recognizes | Anything, in their own words |
| Where the answer comes from | Recorded prompts | Replies written in advance | Your knowledge base and connected systems, phrased fresh each time |
| What happens off script | The menu repeats, or the call goes to the queue | A fallback message, or a loop | Whatever its instructions and transfer rules say, which is what you test |
| Channels | Phone | Chat, sometimes text | Phone, text and chat |
| Typical failure | Callers stuck in the menu | Customers stuck in a loop | A confident answer that is wrong |
If you run a contact center and want the deployment side (a phone-menu option, after hours and overflow, a dedicated line, outbound programs), the AI call center guide covers where agents go and what each pattern changes.
How does conversational AI handle a customer service contact?
Every contact runs through the same few steps. The agent takes in what the customer said, works out the request, finds the answer or runs the action, tells the customer what it did, and then either closes the contact or transfers it. On a phone call, speech recognition turns the caller's words into text first, and a synthetic voice reads the reply back.
- Understand the request. On a call this starts with speech recognition, and that is where names, account numbers and street addresses most often go wrong.
- Find the answer in the knowledge base you approved: policies, hours, procedures, the questions people ask.
- Act in a connected system: look up the customer, check availability, book, reschedule or cancel.
- Say back what it did, in words the customer can check.
- Close the contact, or transfer it under your rules, and record what happened.
On Telvana, every call is recorded, transcribed, summarized and tagged by outcome, and each transferred call is tagged with its transcript and summary so you can see why it was handed off. That record is what you will test against, before launch and after it.
Which customer service contacts is conversational AI good at?
Conversational AI is strongest where the right answer is already written down and the action is clearly defined: status checks, account and policy questions, FAQs, and booking, confirming, rescheduling or cancelling appointments. Where the outcome depends on a judgment nobody has written down, the contact belongs with your team, and the agent's job is to transfer it cleanly.
| Contact type | Who should take it | What has to exist first |
|---|---|---|
| Hours, locations, policies, how-to questions | The AI agent | The current policy, written into your knowledge base |
| Booking, rescheduling, cancelling | The AI agent | A connected scheduling or CRM tool, and rules for what can be moved |
| Order, claim or account status | The AI agent | A lookup the agent can run, and your rule for checking who is asking |
| Confirmations and reminders | The AI agent, by outbound call or text | A contact list or a trigger from your system, plus calling hours |
| Billing disputes, refunds outside policy, complaints | Your team, by transfer | A transfer rule, and a number someone answers |
The outbound row is a topic of its own. AI outbound calling covers which outbound calls to automate first and the rules they run under.
Does conversational AI work the same on phone calls as in chat?
It does not. The underlying model can be the same, but a phone call adds problems text never has: the agent has to reply inside the gap people expect between turns, hear names and numbers correctly over a phone line, and cope with a caller who talks over it. Text has its own quirks, mainly pasted details and customers who reply hours later.
Timing is the clearest difference. A 2009 study of turn-taking in ten languages found that in every language, the most common gap between one person finishing and the next starting fell between 0 and 200 milliseconds, and the mean across the whole dataset was 208 milliseconds (Stivers et al., PNAS). An agent that takes a full second to reply is about five times slower than that average. Nobody can judge that from a feature list, so listen for it on a real call.
Speech recognition is the other half. A chat customer types their account number; a caller reads it out over a mobile connection, maybe in a car. An agent that acts on a misheard digit pulls up the wrong record, which is why test 8 below makes the agent read details back before it acts.
On Telvana, calls, texts and website chat each have their own agent with their own instructions, so you can write each channel for the way people use it. The voice agent does not send texts during or after a call. A written confirmation is a separate SMS campaign that you set up on purpose.
What goes wrong when conversational AI isn't tested?
The documented failures fall into two groups. In the first, the agent tells a customer something the business never approved. In the second, it keeps the customer in a loop with no route to a person. Both have been costly once customers found them, and both show up in a test conversation within minutes.
The best-known case of the first kind is Air Canada. In November 2022, a customer asked the chatbot on Air Canada's website about bereavement fares. It told the customer they could submit their ticket for the reduced rate within 90 days of the ticket being issued, including after they had travelled. The airline's own bereavement page, which the chatbot linked to, said the policy did not apply to requests made after travel had been completed. They booked on the chatbot's answer and later learned that Air Canada did not permit retroactive applications.
In February 2024, British Columbia's Civil Resolution Tribunal described Air Canada's position as, in effect, that the chatbot was "a separate legal entity that is responsible for its own actions," and called that "a remarkable submission." The tribunal wrote: "It should be obvious to Air Canada that it is responsible for all the information on its website. It makes no difference whether the information comes from a static page or a chatbot." It ordered Air Canada to pay C$812.02, of which C$650.88 was damages (Moffatt v. Air Canada, 2024 BCCRT 149). That is one small-claims decision in one province, and none of this is legal advice. The operational lesson is narrow: before launch, ask the agent for the exception your policy refuses.
The CFPB report describes the second kind. When a customer's issue falls outside a chatbot's capabilities, automated responses can "lead them in continuous loops of repetitive, unhelpful jargon or legalese without an offramp to a human customer service representative," which the report calls "doom loops." The same report estimates that about 37% of the US population interacted with a bank's chatbot in 2022, a figure it expected to grow.
How do you test conversational AI before customers use it?
Run a fixed set of test conversations on every channel you plan to launch. Read the transcripts and the records in your systems, not the summaries. Treat a failure on the three tests that protect customers most (an invented policy, an invented answer, no way to reach a person) as a reason to hold the launch, whatever the other results look like.
This is the set we would run on any conversational AI agent, including someone else's. Replace the sample lines with your own policies and call types.
| # | Test | Say or type something like | It passes when | What it catches |
|---|---|---|---|---|
| 1 | An answer from your knowledge | A common policy question, in the words customers use rather than the words in your document | The answer matches the written policy | Gaps between how customers ask and how the policy is written |
| 2 | The exception your policy refuses | "Can I still get the discount if I've already paid?" (use a real refusal from your own policy) | It states the rule as written and offers a person if the customer wants to dispute it | Invented policy, the Air Canada failure |
| 3 | A question your knowledge doesn't cover | Something plausible that you never documented | It says it doesn't have that information and offers a next step | Made-up answers |
| 4 | A person, straight away | "I want to talk to a person," as the first thing said | It transfers under your rules without arguing | Loops with no way out, the CFPB failure |
| 5 | A person, after frustration | Reject two answers in a row, then ask for someone | It transfers when asked | Exits that only open on exact wording |
| 6 | Two requests at once | "Move my Tuesday appointment, and has my last bill been paid?" | It handles both, or handles one and transfers the other and says so | A request silently dropped |
| 7 | A correction mid-sentence | "Thursday at 10, no, Friday at 10" | It books Friday and reads the booking back | Acting on the first thing it heard |
| 8 | Names and numbers by voice | Spell an unusual surname, then read out a phone or account number | It reads them back and confirms before acting | Misheard details that pull up the wrong record |
| 9 | Talking over the agent | Interrupt it halfway through a long answer | It stops and responds to what you just said | An agent that talks past the caller |
| 10 | A change of language | Start in English and switch to Spanish | It follows the customer's language, if you support it | Customers stuck in a language they didn't pick |
| 11 | Something your team keeps | A complaint, a cancellation, or any topic your rules reserve for people | It transfers and promises nothing | Commitments nobody approved |
| 12 | The action, checked at the source | Book or reschedule, then open your scheduling system | The record matches what the agent told the customer | A confirmation with no booking behind it |
How to run it:
- Run each test at least three times, worded differently each time. Conversational AI composes every reply fresh, so a pass on one wording proves little.
- Treat tests 2, 3 and 4 as launch blockers. One failure on any of them means the agent is not ready, however the other nine went. That is our rule of thumb, not an industry standard, and you can set a stricter one.
- Run tests 8 and 9 from a phone over a real line. A browser demo with a headset hides the problems a mobile connection creates.
- Check the transcript and the system record for every test. A summary is the agent's own account of the conversation, so check it against the record.
- Run the whole set again after any change to the knowledge base, the instructions or the transfer rules.
That is close to how we launch an agent at Telvana. Our team builds it from your scripts, FAQs, policies and transfer rules, tests it on your call types, and you approve it before it takes a live call. The tests only your team can write are 2 and 11, because you know where your policy says no.
What should you measure after conversational AI goes live?
Measure resolution and repeat contacts by contact type, and read the reason behind every transfer. A count of conversations handled shows the agent is busy and says nothing about whether the customer's problem went away.
Klarna's February 2024 announcement is the example people quote most, and its headline figure is volume: 2.3 million conversations in the assistant's first month, two-thirds of Klarna's customer service chats. The more useful line in the same release is that the assistant was more accurate in resolving errands, "leading to a 25% drop in repeat inquiries," because that one says customers did not have to come back (Klarna). These are Klarna's own reported figures for chat. Read them as a guide to what to measure, not as a benchmark for phone calls.
What to track from the first week:
- Contacts resolved without a transfer, split by contact type, never as one blended rate.
- Repeat contacts from the same customer within a fixed window, such as seven days. The call center metrics guide explains why we prefer this to first call resolution.
- Transfer reasons, read from the transcripts. A cluster of transfers on one question usually means an entry is missing from the knowledge base.
- Handle time on your team's queue, by call type. When the agent takes the shorter contacts, the average on the human queue goes up even though nobody slowed down. Average handle time works that arithmetic through on a real call mix.
Frequently asked questions
Is conversational AI the same as a chatbot?
Some chatbots use conversational AI, and many older ones do not. A rule-based chatbot matches menu choices or keywords to replies written in advance. Conversational AI understands requests in the customer's own words and writes each reply as it goes. That makes it far more flexible, and it is why it has to be tested for wrong answers before launch.
Can conversational AI answer phone calls?
Yes. On a call it adds speech recognition and a synthetic voice to the same reasoning. Telvana's agents answer inbound calls, run outbound call campaigns, and talk with callers in English or Spanish, following the language the caller uses. Voice brings its own test cases, mainly response timing, misheard names and numbers, and callers who interrupt.
What happens when the AI doesn't know the answer?
That depends on its instructions, which is what test 3 checks. A well set up agent says it does not have the information and offers a next step or a transfer. On Telvana you set the transfer rules: when the caller asks for a person, when the topic is one you keep for your team, or when the agent cannot resolve the request.
How long does conversational AI take to set up?
With Telvana, days. Our team builds the agent from your scripts, FAQs, policies and transfer rules, tests it on your call types, and you approve it before it takes a live call. The slowest part is usually on your side: getting the policies the agent will answer from written down and current.
How much does conversational AI for customer service cost?
Telvana starts at $399 a month based on your call volume, plus a one-time setup fee that covers building, testing and launching your agent. There is a single price rather than tiers to compare. See pricing for what is included.
See it take one of your calls
Book a demo and bring your top call types. We'll show you which of them Telvana can take, and you'll hear an agent handle one on the call. Book a demo.