AI Agents in Production: What Actually Works

Gartner predicts 40% of AI agent projects will fail by 2027. After building agent workflows that handle real restaurant operations — order routing, FAQ automation, menu updates — I understand why. The failure isn't usually in the AI. It's in the architecture around it. Here's what I've learned about making agents that don't break your business. A demo agent is a single prompt, a single tool, a single turn. You ask it a question, it calls an API, it responds. It works because the scope is tiny. A production agent needs to:

• Handle multi-step workflows with dependencies
• Recover from tool failures gracefully
• Know when to stop and ask a human
• Maintain consistent behavior across thousands of interactions
• Not delete your database (yes, this is a real production concern)

The jump from demo to production is not incremental. It's architectural. Every reliable agent I've built follows the same structure. I think of it as three layers. This is what the agent can actually do. Each capability is a discrete, well-tested function with a clear contract.

// A capability is just a function with a strict contract
interface AgentCapability {
  name: string;
  description: string;
  inputSchema: JSONSchema;
  execute: (input: unknown) => Promise<CapabilityResult>;
}

// Example: look up an order status
const checkOrderStatus: AgentCapability = {
  name: "check_order_status",
  description: "Look up the status of a customer's order by order ID",
  inputSchema: {
    type: "object",
    properties: {
      orderId: { type: "string", pattern: "^ORD-\\d{6}$" },
    },
    required: ["orderId"],
  },
  execute: async ({ orderId }) => {
    const order = await db.orders.findById(orderId);
    if (!order) return { success: false, error: "Order not found" };
    return { success: true, data: { status: order.status, eta: order.estimatedReady } };
  },
};

The key principle: each capability is independently testable and fails gracefully. The agent doesn't crash if a tool fails. It gets an error result and decides what to do next. This is where the LLM decides which capabilities to call and in what order. The orchestration layer is deliberately dumb — it's just a loop.

async function runAgent(
  userMessage: string,
  capabilities: AgentCapability[],
  context: AgentContext
): Promise<AgentResponse> {
  const messages = [
    { role: "system", content: buildSystemPrompt(context) },
    { role: "user", content: userMessage },
  ];

  for (let step = 0; step < MAX_STEPS; step++) {
    const decision = await llm.route(messages, capabilities);

    // The agent wants to call a tool
    if (decision.type === "tool_call") {
      const capability = capabilities.find(c => c.name === decision.tool);
      const result = await capability.execute(decision.input);
      messages.push({ role: "tool", content: JSON.stringify(result) });
      continue;
    }

    // The agent wants to escalate to a human
    if (decision.type === "escalate") {
      await notifyHuman(context, decision.reason);
      return { type: "escalated", message: "Connecting you with our team." };
    }

    // The agent has a final answer
    if (decision.type === "response") {
      return { type: "complete", message: decision.content };
    }
  }

  // Safety valve: too many steps means something is wrong
  return { type: "timeout", message: "Let me connect you with someone who can help." };
}

Two things matter here: the step limit (agents can loop forever otherwise) and the escalation path (the agent must always have a way to hand off to a human). Guardrails sit between the agent and the user. They catch things the LLM shouldn't say or do.

// Before the agent's response reaches the user
function guardResponse(response: string, context: AgentContext): GuardResult {
  // Never reveal internal order IDs or system details
  if (containsSystemTokens(response)) {
    return { blocked: true, reason: "Internal system information detected" };
  }

  // Don't make promises about refunds or compensation
  if (matchesRefundLanguage(response) && !context.permissions.canApproveRefunds) {
    return {
      blocked: true,
      replacement: "I can't process refunds, but I'll connect you with our team who can.",
    };
  }

  // Rate limit per customer
  if (context.customerMessageCount > HOURLY_LIMIT) {
    return { blocked: true, reason: "Rate limit exceeded" };
  }

  return { blocked: false };
}

Guardrails are unsexy and essential. They're the difference between an agent that's helpful and an agent that's a liability. The thing that degrades agent quality over time isn't the model. It's state. Agents need to remember context across interactions — what the customer asked before, what was already tried, what the resolution was. There are three kinds of memory, and they need different storage:

Memory Type	What It Holds	Storage
Working	Current conversation context	In-context window (ephemeral)
Episodic	Past interactions with this user	Database, retrieved on demand
Semantic	General knowledge (docs, policies)	Vector store, RAG pipeline

Most agent failures I've seen come from confusing these. Trying to stuff everything into the context window. Or retrieving irrelevant history. Or not retrieving history at all, so the agent asks a customer for information they already provided. My approach: keep the context window lean. Retrieve episodic memory only when the user references a past interaction. Use RAG for policy questions. Don't try to be clever — predictable retrieval beats smart retrieval in production. This is the most important decision, and it's made too late most of the time. Use an agent when:

• The input is unstructured (natural language, images)
• The workflow has conditional branching that's hard to enumerate
• The user doesn't know what they want until they start talking

Don't use an agent when:

• A form would work
• A deterministic API call would work
• The "AI" is just a wrapper around a database lookup

I've seen teams build agent-based interfaces for things that should be a dropdown menu. The agent adds latency, unreliability, and cost for zero user benefit. If a user knows exactly what they want, give them a button. Save the agent for genuine ambiguity. Here's a mental model that helped me: every agent feature has a reliability budget. The question is how you spend it.

• If the agent handles FAQ deflection (low stakes), a 5% failure rate is fine — the user just retries or waits for a human.
• If the agent handles order modification (medium stakes), 1% failure is the ceiling, and every failure needs a clean fallback.
• If the agent handles payments (high stakes), don't use an agent. Use a deterministic flow. Period.

Match the autonomy to the stakes. This sounds obvious until you see a startup letting an agent process refunds autonomously because the demo looked impressive. The agents that work in production are boring. They do narrow tasks, they escalate quickly, they have clear guardrails, and they never pretend to be smarter than they are. The agents that fail are the ones built to impress investors rather than serve users. Build the boring version first. Add intelligence only where determinism fails. That's the whole secret.