Blog · AI Agents

Building reliable AI agents: best practices for your SMB

Why "it works in the demo" doesn't mean it's reliable

A demo usually tests the best case: a clear question, clean data, no unexpected input. Reality is different. Customers ask questions you didn't anticipate, systems return error messages, and data is never as clean as in your test set. An agent that works correctly 95% of the time in a demo can fail in unexpected ways on exactly the 5% that matters most in practice — the angry customer, the order that doesn't add up, the invoice with the wrong amount. You measure reliability by the worst cases, not the best ones.

Guardrails: setting limits before granting autonomy

A guardrail is a hard limit the agent cannot cross, regardless of what the model "thinks" is the right action. Concretely: an agent that drafts quotes should never send an amount without a check, a customer service agent should never authorise a refund above a fixed amount, and an agent that sends emails should never send anything that hasn't first passed a tone-and-content filter. You build these limits into the code, not into the prompt — an instruction in a prompt can be ignored or misread by a model, a hard check in the code cannot.

Compare this with building simple automation (as in my article on AI automation tools): with a fixed workflow that has no AI decision-making, this risk doesn't exist. The moment a model starts making its own choices, a guardrail becomes just as important as the functionality itself.

Human-in-the-loop: where a human must approve

Not every step needs human review — that would undermine the whole point of automation. But certain steps always deserve approval before they proceed:

• Financial transactions — payments, refunds, price changes
• External communication — the first few months, until you've proven the agent consistently gives appropriate answers
• Irreversible actions — deleting data, entering into contracts, definitively cancelling orders
• Situations outside the training data — if the agent itself signals uncertainty, that should always go to a human

The practical approach: start with an agent that makes proposals you (or your team) approve with one click. Once you've seen months of reliable proposals, you can let the lowest-risk steps proceed automatically.

Evaluation: how you measure whether an agent performs well

"It feels right" is not an evaluation method. What does work: a fixed set of test questions and situations (including odd, wrong and edge cases) that you regularly run past the agent and score manually. This is exactly the work I do daily at Outlier.ai — evaluating ChatGPT, Gemini and Grok responses for correctness, tone and completeness. I apply that same discipline to every agent project: a test set of 20-50 realistic situations, including the tricky ones, and a score for how often the agent gets it right. Without that measurement, you don't know whether a change made the agent better or worse — you're guessing.

Logging and escalation: seeing what happens, and being able to fix it

Every action an agent takes needs to be logged: what input it received, what decision it made, which tool it called, and what the outcome was. Without logging, you can't reconstruct what went wrong when a customer complains. Every agent also needs a clear escalation path: if it gets stuck, is uncertain, or moves outside its scope, it should say so explicitly and hand off to a human — not keep guessing. This is exactly what I look for in my article on custom software and AI: a system that "fails" by honestly saying "I don't know" is safer than one that always makes up an answer.

GDPR: what an agent is allowed to do with customer data

Once an agent processes customer data (names, email addresses, order history), regular GDPR rules apply in full. A few practical points that often get overlooked: (1) data you send to an AI API (OpenAI, Claude) leaves your system — check the vendor's data processing policy and formalise it in a data processing agreement, (2) don't log unnecessary personal data longer than needed, and (3) be transparent with customers when they're interacting with an AI system rather than a human. For most European AI vendors (OpenAI and Anthropic both offer EU data processing options) this is manageable, but it needs to be set up deliberately — not as an afterthought.

Risk level	Example	Required controls
Low	Internal summarisation of documents	Logging, spot-check evaluation
Medium	Answering customer questions based on FAQs	Logging, escalation path, periodic evaluation
High	Drafting quotes, processing orders	Amount guardrails, human-in-the-loop, full logging
Very high	Financial transactions, contractual decisions	Mandatory human approval, no autonomous execution

What you get with ultimAItech

I don't build agents without these controls in place from day one — not as an extra, but as the foundation. That comes directly from my daily work as an AI trainer at Outlier.ai, where I see exactly how models fail when no one has set those limits.

• Custom guardrails — hard limits in code, not just instructions in a prompt
• Evaluation setup — a test set of realistic and difficult situations, with a measurable score
• Logging and escalation path — so you can always see what happened and fix mistakes
• GDPR-aware setup — data processing agreements, data minimisation and transparency with customers
• Gradual autonomy — starting with human-in-the-loop, expanding based on proven results

Also read my article on what an AI programmer does for your business and how I approach that process.

Frequently asked questions

What is a guardrail in an AI agent?

A guardrail is a hard limit in the code (not just an instruction in a prompt) that prevents an agent from taking certain actions, such as a payment above a set amount or sending unchecked emails.

Does every action of an agent need to be reviewed by a human?

No, that would remove the value of automation. Financial, irreversible or uncertain actions should always stay with a human, while low-risk tasks can proceed automatically once reliability has been proven.

How do you evaluate whether an AI agent performs well enough?

With a fixed test set of realistic and difficult situations that you score manually and regularly for correctness, tone and completeness — not with a gut feeling, but with a measurable score you can compare over time.

Is using the OpenAI or Claude API GDPR-compliant?

It can be, provided you sign a data processing agreement with the vendor, only send necessary data, and are transparent with customers about AI use. Both major vendors offer options for European data processing.

What happens when an agent gets stuck or makes a mistake?

With a good escalation path, the agent explicitly signals uncertainty and hands off to a human, instead of continuing to guess. Full logging also makes it possible to reconstruct afterwards exactly what happened.

Conclusion

A reliable AI agent isn't about picking the best model — it's about guardrails, evaluation, logging and human oversight built in from day one. The agent that impresses most in a demo isn't automatically the one you can safely let loose on your customers or finances. Build with that order in mind and you prevent most problems before they happen.

Want to build an AI agent with these controls in place? Book a free call or send a WhatsApp message. I'm happy to think through what should and shouldn't run automatically.