How to Tell If an AI Vendor's Statistic Is Actually Real
Look at where the number actually comes from before you repeat it or use it to make a decision. If a vendor's blog says something like "AI chatbots resolve most support tickets" with no link, no date, and no named source, that's not data, that's a guess dressed up as one. A real stat traces back to an actual study, survey, or dataset you can go open and check yourself.
Why do so many AI vendor stats show up with no source?
Most of it isn't malice, it's speed. A content team needs a blog post out this week, so a stat gets pulled from a competitor's article, or from an older post nobody re-checked, and it keeps moving down the chain. By the third or fourth blog that repeats it, nobody involved has ever seen the original number.
We ran into this ourselves checking sources for a client's blog, a process we've written about before. One vendor's escalation guide gave three different figures for how many routine questions a bot can handle, plus a customer walkaway stat, and none of it pointed to anything. We rejected it as a citation and said the point plainly instead, without inventing a number to replace it.
What does a citation you can actually trust look like?
It names the study, the survey, or the company that ran the numbers, says what year the data was collected, and links to something you can open yourself, not just another blog post repeating the same line. If all you can find is "research shows" or "studies indicate" with nothing attached, there's nothing to check, which means there's nothing to trust yet either.
Does it matter how old the stat is?
A lot. What a chatbot could do in 2021 or 2022 is not what a trained agent does now. A deflection rate or accuracy number from a few years ago is describing a button menu bot, not the kind of AI agent people are actually shopping for today. If a post doesn't say when the data was collected, that's worth noticing on its own.
What's the fastest way to check a stat before you repeat it?
Click through. If "studies show" has no link, or the link goes to another blog post instead of an actual source, you have your answer.
Check when the data was collected, not just when the post went up.
Look for a ceiling dressed as an average. "Up to 70 percent" is not the same claim as "an average of 70 percent," and vendors blur that line on purpose.
Ask the vendor directly for the source. A company with a real number can send it to you in one email.
A vendor with a real number can send it to you in one email. One that can't is telling you something.
Is this actually something anyone official cares about?
Yes. The FTC opened what it calls Operation AI Comply in September 2024, specifically to go after companies that overstate what their AI actually does, and it's kept bringing cases since, including ones built on claims made to other businesses, not just consumers. A loose stat in a blog post isn't just something to double check for your own decision. Enough of them in one place is the kind of thing that gets a company investigated.
None of this means every AI vendor is lying to you. Most of these stats started somewhere real. They just picked up too many hops between the study and the blog post in front of you. Check the number before it becomes part of your decision.
We hold ourselves to the same rule writing about our own work: if we can't point to something real, we don't say it. If you want to see what an agent built that way actually looks like, there's a live demo at demo.ajmarketingresults.com.
FAQ
Is it normal for AI vendors to not cite their stats?
It's common. That doesn't make it something you should treat as reliable. Most marketing content gets written fast, and a stat that started somewhere real can get repeated for years without anyone going back to the source.
What if the vendor shows a case study instead of a stat?
A named case study beats a floating number, but check whether it names a real company you can look up, a specific before and after, and whether it's one best case customer standing in for everyone else.
Should I just ask the vendor where a number came from?
Yes, ask directly and ask for it in writing. How fast and how clearly they answer tells you almost as much as the source itself. It's one more thing worth checking alongside the other questions to ask before you buy an AI chatbot in general.
Does this apply to AJM's own numbers too?
Yes. Any stat we put in an article should trace back to something real. If it can't, we either find the real source or leave the number out and say the point plainly instead.
Comments