Back to blog

Dispatch, triage

AI Triage: Don’t Ask How Often It’s Right. Ask What Happens When It’s Wrong. 

•

3 minute read

Alan Vollmar

AI triage tools love a big number. 95% accuracy. 96%. 98%. Spend a week in demos and you’ll hear all of them. 

On paper, 95% sounds like a rounding error. Nineteen tickets right for every one wrong. Plenty of service desks would take that trade today. 

But your clients don’t experience your accuracy rate. They experience their ticket. If theirs lands in the 5%, it doesn’t matter how good the other 95% looked. 

This isn’t an argument against AI triage. It’s an argument for asking a better question before you trust any number: not how often it’s right, but what happens when it’s wrong. 

An average hides the tickets that matter 

Start with what the number even covers. Category, priority, and board are three different calls. A score might be per field or per whole ticket. Most published claims don’t say which. 

Then remember that accuracy is an average. Averages describe the typical ticket well and the rare one badly. Most of your volume is routine: password resets, new user setups, a printer that won’t print. A tool can be excellent at those, shaky on the unusual ticket, and still post a number in the high 90s. Google’s machine learning course explains why: when one kind of case is rare, a model can score high on accuracy while handling that case badly. 

You already know this from SLA reporting. Nobody judges a service desk on one blended SLA number, because 95% overall can hide a string of missed P1s. You split it by priority. Do the same with triage. 

Where the 5% lives 

Misses aren’t spread evenly. They cluster in tickets that are vague, unusual, or one of a kind. “Computer is acting weird.” “Something’s off with email.” A ticket from the client whose setup looks like nobody else’s. 

That’s also where trouble tends to start. Security responders have noted that the first sign of a ransomware attack will probably be a help desk ticket . And attackers often sit inside an environment for days or weeks before they strike. So that vague ticket might be nothing. Or it might be the earliest warning you’ll get. 

File it as a low-priority performance issue and it waits behind the routine work. Nobody did anything obviously wrong. The ticket just looked ordinary. That’s what makes the 5% dangerous: the misses that matter most rarely look important when they arrive. 

What one miss costs 

Not every miss costs the same. A wrong subtype is a reporting annoyance. A wrong board means a bounce and a delay. A wrong priority on the wrong ticket means a client’s business is stalled while the SLA clock runs. 

Put rough numbers on it. At 80 tickets a day, 5% wrong is four tickets a day, about 20 a week. Nineteen of those might be harmless. The twentieth is the one your client remembers. Downtime estimates for small businesses vary widely, but even conservative ones put an hour of a stalled office in the thousands of dollars. That cost lands on your client first, and on your relationship with them right after. 

Your clients don’t grade you on averages. They judge you by how you handled their worst day. 

Know your own number first 

Here’s the uncomfortable part: your current triage isn’t at 100% either. Every service desk misclassifies tickets. It just rarely gets measured. 

Before you compare tools, find your baseline. Once a month, pull 50 to 100 closed tickets. Compare how each was first triaged with how it was actually worked. Count the reassignments, since each one means a client explained their problem twice. Then flag which misses were high impact. 

Most teams find their triage is less consistent than they thought. That’s not a people problem. Your dispatcher makes hundreds of fast calls a week on incomplete information. It’s a process signal, and it gives you the accuracy number that matters most: yours, on your tickets, for your clients. 

Build for the misses 

No triage, human or AI, will ever be perfect. So the goal isn’t a perfect score. It’s a setup where the misses that matter get caught before they reach a client. 

When you evaluate any AI triage tool, look past the headline number and ask three questions: 

  1. What happens when it isn’t sure? Uncertain tickets should go to a person, not get a confident guess. 
  2. Can you see why it made each call? Visible reasoning lets your dispatcher spot a bad call in seconds. 
  3. Who has the final say? Your dispatcher knows which client is weeks from renewal and which “slow computer” deserves a second look. Keep them in charge. 

That’s how we built Dispatch HQ. Triage recommends a classification from your own ticket history and shows its reasoning. Your dispatcher confirms or corrects it before it reaches a technician, and every correction sharpens the next recommendation. 

The number on the slide will always look good. Ask what happens when it’s wrong.

Start your 14-day free trial of TimeZest today!