Context engineering isn't enough

If your own team can't agree on the answer, stop asking the model to sort it out.

I've watched teams improve retrieval and still get stuck on the same questions. The agent has the documents. The business says its answer is wrong.

Then you ask what the answer should have been. Different people give you different answers — all with a decent explanation.

You can spend another fortnight on the prompt. You'll still need someone to make the bloody decision.

Context engineering means giving the model the information it needs. I'm all for it. But a better supply of contradictory instructions won't settle which one we're meant to follow.

Experienced service reps deal with this through judgement. They know which exception matters and who to ask. That works until we try to automate a decision we've never properly agreed on.

My view is that this belongs in the AI programme. Getting a policy settled is delivery work, even if it makes a dull demo.

It also changes who needs to be in the team. We need people who can build the thing and people who can get a disputed decision resolved. The second job doesn't disappear because it isn't technical.

Once the rule is agreed, enforce permission in the system doing the work. The model can suggest a refund. It shouldn't get to decide how much authority it has to issue one.

Someone also needs permission to handle exceptions. Writing a rule doesn't make every awkward case go away — and treating everyone identically isn't always good service.

This is where content maintenance comes in. Write the decision down, keep it current and give the agent a route out when the rule doesn't fit.

There's a limit to this argument. Sometimes the policy is perfectly clear and the agent is simply wrong. Another meeting won't fix that either.

I use a simple test. Give a new starter the same material and ask them to answer the question. If they have to find someone to settle what the policy means, start there.

If they can answer it, get back to the engineering. They're not a perfect comparison — people bring knowledge the model wasn't given — but they'll tell you where to look.

Next time someone brings you a bad answer, ask them to write the right one. Can the person who owns that decision stand behind it?