Why Your AI Chatbot Gives Conflicting Answers (And How to Fix It)

Your AI chatbot is not lying to you on purpose. When it gives two different answers to the same question, it almost always means your knowledge base is mixing two things that should stay apart: your real current policy, and something a person said once as an exception. Separate those two piles and the contradictions usually stop.

Why does an AI chatbot invent a policy that is not real?

In 2024, a Canadian tribunal made Air Canada honor a bereavement discount its own chatbot invented. A customer asked about bereavement fares after his grandmother died. The chatbot told him he could book a full price ticket and apply for the discount afterward. That was never the real policy, which required the discount to be requested before travel. Air Canada argued the chatbot was responsible for its own words, separate from the airline itself. The tribunal disagreed and ordered Air Canada to pay the difference.

Whatever your agent says, your customer treats it as your business talking, and the tribunal's ruling made that official. The reason it happens generalizes past one airline: an AI agent does not check a rulebook before it answers. It pulls language from whatever content you gave it and blends pieces together. If two pieces of that content disagree, the agent will not referee between them. It just answers, and sometimes the answer is a blend of both that matches neither one.

Where do the contradictions actually come from?

Most small businesses gather everything they can find to train an agent: the real policy page, plus every past support ticket, chat log, and email reply. That instinct is correct, your old messages are genuinely good training material. The trouble starts when both piles sit together with no label, because past tickets hold two very different things at once: real answers to real questions, and exceptions a person made once, on one particular day, for one particular customer.

Common places a contradiction sneaks in:

• A refund given as a favor to one upset customer, later quoted back to everyone else as if it were the standing policy.

• An old pricing page nobody removed, sitting in the same folder as the current prices.

• A support rep's personal way of phrasing an answer, which the agent copies as if it were the official wording.

How do you separate current policy from old tickets?

This is mostly a filing job, not a technical one.

1. Put your real, current policy documents (refunds, shipping, pricing, membership terms) in one folder and mark it clearly as the material the agent trusts first.

2. Put past tickets, chats, and emails in a different folder. Label it for tone and phrasing, not for rules.

3. Tell whoever builds or configures the agent which folder wins when the two disagree. Policy wins, every time. Tickets only teach it how to talk.

4. Remove anything expired, an old price, a promotion that already ended, instead of leaving it next to the current version for the agent to find later.

What do you do with the exceptions you already made?

You do not need to throw exceptions away. They are often the most useful, human writing you have. They just need a label: this was a judgment call for one person, not the rule for everyone. If the next customer asks for the same treatment, that is a decision for a person to make again, not something the agent should hand out automatically.

An exception you made for one customer is not policy until you decide to make it policy. Until then, it should never be the thing your agent repeats back to the next hundred people who ask.

How do you keep this from drifting back out of sync?

Fixing it once is not the same as keeping it fixed. Update the trusted folder the same day a policy changes, not whenever someone remembers to. Once a month, pull five real answers the agent gave and check them against the current policy page, not against what sounds right. This is the same habit worth building into checking whether any automation is still working generally. And if someone else built the agent for you, ask them directly which folder they treated as the one that wins. If they cannot answer clearly, that is the real project, more than any rebuild.

This is the kind of detail that decides whether a trained support agent actually holds up day to day. The one we built for Brock Johnson's InstaClubHub community works specifically because the material behind it stays clean: current content in one place, one time exceptions kept separate and clearly marked. It also runs on the same retrieval augmented generation approach we've written about plainly before, source material matters more than the model. If you want to see what a properly trained agent looks like on your own content, our free demo builds one from a link in about a minute at demo.ajmarketingresults.com.

---

Common questions about AI chatbots and conflicting answers

Why does my AI chatbot give two different answers to the same question?

It usually means the agent is pulling from at least two sources that disagree, most often a current policy document and an old support ticket or an outdated page. The agent has no built in way to know which source is actually true anymore, so it answers from whichever piece it retrieves, or blends both together.

Can I use old support tickets to train my AI agent?

Yes, and you should. Old tickets show real questions in real customer words and real answers that worked. Just keep them in a separate folder from your official policy documents, and label them for tone and phrasing only, never as the rule.

What does source of truth mean for an AI knowledge base?

It means one document, or one small set of documents, is the only material the agent is allowed to treat as the current, official answer. Everything else, past tickets, old emails, casual notes, can shape tone but should never override it.

Is this a RAG problem or a knowledge base problem?

It is a knowledge base problem first. Retrieval augmented generation just retrieves whatever content you gave it. If that content already disagrees with itself, no retrieval method fixes that on its own. The fix happens before the AI ever gets involved.

How do I know if my agent already has this problem?

Ask it the same real question three different ways over a few days and compare the answers side by side. If they do not match, or if one answer includes a detail that is not on your actual policy page, you likely have an unlabeled exception sitting in the training material.