The approval gate is the product

Anyone who has shipped a robotics or machine learning system learns the same thing early: most of the hard engineering is not in the model. It is in the handoff — knowing what the system does not know, and getting a human involved before that gap becomes an action nobody can undo.

That framing transfers almost unchanged to AI software for a business.

Two failure modes, not one

When people evaluate an AI feature they usually ask one question: how often is it right? That treats every error the same, and they are not the same.

• A message sorted into the wrong category costs the user a few seconds. They re-file it and move on.
• A confidently wrong reply sent to a client can cost the relationship.

These sit at opposite ends of a cost curve, so they deserve opposite defaults. Classification can act on its own. Anything client-facing cannot.

What we actually built

Every classification in our product returns a structured result rather than a bare label:

{
  "category": "renewal",
  "priority": "high",
  "confidence": 0.82,
  "escalation": "review"
}

escalation has three values — handle it, review it, or put it in front of a person — and the user controls what happens at each. That is the dial. There is no global "trust the AI" switch, because a single switch cannot distinguish between filing a newsletter and quoting a premium.

Outbound replies then sit behind a hard stop. The model writes a draft, the draft waits in the thread, and it goes nowhere until someone reads it and approves it. Not a default that can be changed in settings — a property of the system.

The objection, and the answer

The obvious pushback is that this caps the value. If a person still reads every reply, what did the automation buy?

It bought the writing, which is the slow part. Reading a good draft and pressing approve takes a few seconds. Composing the same message from scratch — pulling up the client's history, remembering the standing policy, getting the tone right — takes minutes. Across a full inbox that is the difference between an afternoon and a coffee break.

It also bought trust, which is the part that determines whether the product is still installed in six months. An owner who has watched every draft for two weeks and found them all reasonable will start skimming instead of scrutinising. That is a decision they get to make with evidence, at their own pace. It is not one we get to make for them in a config default.

Where this generalises

If you are building anything that acts on a business's behalf, the question worth asking early is not "how accurate is the model?" It is:

1. What is the most expensive irreversible action this system can take?
2. Does a human see it first?
3. Can that gate be turned off — and if so, who turns it off, and at what hour of what bad day?

Get those three right and you can be relaxed about the accuracy number, because the failure mode is a person raising an eyebrow at a draft. Get them wrong and no accuracy number is high enough.