Structured Outputs That Don't Break in Production

The first version of our menu scanner at Menyo Pro worked perfectly. On the three menus I tested. The moment we pointed it at a real restaurant's crumpled, coffee-stained paper menu, the structured output fell apart. That gap — between the demo and production — is where most LLM features die. Here's what I learned closing it. You write a careful prompt. You ask for JSON. You describe every field. The model returns clean JSON on the first try, and you think you're done. Then a user uploads a menu written in three languages with handwritten prices. The model returns JSON with a price field containing "12 AED — ask staff". Your parser crashes. The onboarding fails. The restaurant owner closes the tab and never comes back. The core issue: prompt engineering is a suggestion, not a contract. You are asking a probabilistic system to behave deterministically. No amount of prompt refinement changes that fundamental reality. What actually works is treating the LLM as one layer in a pipeline, not the whole pipeline. Here's the structure I converged on after hundreds of broken scans. Instead of describing the output shape in prose, define it as a schema and let the model's native structured-output mode enforce it.

const menuSchema = {
  type: "object",
  properties: {
    categories: {
      type: "array",
      items: {
        type: "object",
        properties: {
          name: { type: "string" },
          items: {
            type: "array",
            items: {
              type: "object",
              properties: {
                name: { type: "string" },
                price: { type: "number" },
                currency: { type: "string", enum: ["AED", "SAR", "USD", "EUR"] },
                description: { type: "string" },
              },
              required: ["name", "price"],
            },
          },
        },
      },
    },
  },
};

const result = await model.generateStructured({
  schema: menuSchema,
  image: menuPhoto,
});

This gets you 90% of the way there. The model can't return a string where a number belongs. But it can still return 0 for a price it couldn't parse. The schema guarantees shape. It does not guarantee sense. You need a validation pass that catches semantic errors.

function validateMenuItem(item: ParsedItem): ValidationResult {
  const errors: string[] = [];

  // Price sanity
  if (item.price < 0) errors.push("Negative price");
  if (item.price > 10000) errors.push("Price unreasonably high");
  if (item.price === 0 && !item.name.toLowerCase().includes("free")) {
    errors.push("Zero price without free flag");
  }

  // Name sanity
  if (item.name.length < 2) errors.push("Name too short");
  if (item.name.length > 200) errors.push("Name too long — likely a parsing error");

  // Currency mismatch
  if (item.currency && !supportedCurrencies.has(item.currency)) {
    errors.push(`Unsupported currency: ${item.currency}`);
  }

  return { valid: errors.length === 0, errors };
}

This is the boring code that nobody writes for a hackathon. It's the code that determines whether your feature survives contact with real users. When validation fails, don't silently retry the same prompt. Feed the error back.

async function extractWithRetry(image: Buffer, maxAttempts = 3) {
  let lastErrors: string[] = [];

  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    const result = await model.generateStructured({
      schema: menuSchema,
      image,
      // On retry, tell the model what went wrong
      feedback: attempt > 0 ? `Previous attempt had errors: ${lastErrors.join(", ")}` : undefined,
    });

    const validation = result.categories
      .flatMap(c => c.items)
      .map(validateMenuItem);

    const failedItems = validation.filter(v => !v.valid);
    if (failedItems.length === 0) return result;

    lastErrors = failedItems.flatMap(v => v.errors);
  }

  // Final fallback: flag for human review instead of failing silently
  return { ...result, needsReview: true };
}

This pattern — validate, feedback, retry, escalate — is the difference between a feature that works 70% of the time and one that works 98% of the time. One thing that surprised me: menus in the Gulf often list prices without a currency symbol. "Margherita — 45" could be dirhams, riyals, or pounds depending on where the photo was taken. The fix wasn't in the prompt. It was in context. We pass the restaurant's registered location into the extraction call so the model infers currency from geography. A small thing, but it cut our price-parsing errors by roughly a third. Lesson: the prompt is not the only input. Context, user state, and business rules all shape what the right answer is. The most important design decision was this: extraction failure is not onboarding failure. When the scanner can't confidently parse an item, it doesn't crash. It creates a draft entry flagged needsReview, pre-fills what it could parse, and surfaces it to the restaurant owner in the editor. The owner fixes it in ten seconds. Onboarding continues. This is the principle that makes LLM features viable in products where the user isn't a developer. You never show them a stack trace. You show them a draft they can fix. If you're building your first production LLM feature:

1. Write the validation logic before you write the prompt. It forces you to define what "correct" means.
2. Assume the model will be wrong in creative ways. Your job is to catch it gracefully.
3. Measure failure modes, not just success rate. "95% accurate" is meaningless. "Fails on multi-currency menus with handwritten prices" is actionable.
4. Build the human-in-the-loop fallback from day one. It's not a temporary patch; it's the architecture.

The demo is the easy part. Production is the whole job.