Building AI Agents That Actually Work in Production
Lessons from deploying autonomous agents at scale — error recovery, guardrails, observability, and why most agent demos fail in the real world.
A demo agent runs once, on a clean input, with someone watching. A production agent runs ten thousand times on inputs nobody anticipated, with no one watching. Everything that matters lives in that gap.
Bound the loop
An agent without a step budget is an outage waiting for a trigger. I gate every loop on steps, wall-clock time, and token spend, and I make the termination reason explicit in the result.
type Stop = "done" | "max_steps" | "timeout" | "budget";
export async function runAgent(goal: string, limits = { steps: 12, ms: 60_000, tokens: 80_000 }) {
const started = Date.now();
let tokens = 0;
const trace: Step[] = [];
for (let step = 0; step < limits.steps; step++) {
if (Date.now() - started > limits.ms) return finish("timeout", trace);
if (tokens > limits.tokens) return finish("budget", trace);
const decision = await plan(goal, trace);
tokens += decision.usage.total;
if (decision.type === "answer") return finish("done", trace, decision.answer);
const result = await callTool(decision.tool, decision.args); // never throws
trace.push({ ...decision, result });
}
return finish("max_steps", trace);
}Tools fail; the agent should not
Tool errors are information, not exceptions. Return them to the model as text so it can adapt, and validate arguments before executing anything with side effects.
const schemas = { refund: z.object({ orderId: z.string().uuid(), cents: z.number().int().positive().max(50_00) }) };
async function callTool(name: string, args: unknown) {
const schema = schemas[name as keyof typeof schemas];
if (!schema) return { ok: false, error: `unknown tool ${name}` };
const parsed = schema.safeParse(args);
if (!parsed.success) return { ok: false, error: parsed.error.message };
try {
return { ok: true, data: await registry[name](parsed.data) };
} catch (e) {
return { ok: false, error: e instanceof Error ? e.message : "tool crashed" };
}
}The guardrails that earned their keep
- –Idempotency keys on every side-effecting tool, so a retried step cannot double-charge.
- –A human-approval gate for irreversible actions above a threshold.
- –Full traces persisted with prompt version, model, and token cost — you cannot debug what you did not record.
- –Replay tests: yesterday's failing traces become today's regression suite.
None of this makes the agent cleverer. It makes it survivable, which is the actual requirement.