Operations·May 2025·10 min read

The Role of AI and GenAI in Building Resilient Business Models

AI makes an operation more resilient only when it carries the same reliability engineering discipline as any other production system: evaluated before it ships, degraded gracefully when it fails, and watched closely once it's live. Bolt a model onto a workflow without those three and the model is a new failure mode, not a resilience upgrade.

The pitch and the failure mode are the same sentence

The pitch for AI-driven resilience usually sounds like this: a model predicts the disruption before it hits, so the business reacts faster than a human ever could. That's true when the model is right.

It's also exactly how AI becomes a new single point of failure. A forecasting model that silently drifts, a chatbot that hallucinates a return policy, an automation that fires on bad input, all of these fail quietly, and quiet failure is worse for resilience than no automation at all.

The gap between the pitch and the failure mode is almost always the same three things missing: no evaluation gate before deployment, no fallback path when the model is wrong or unavailable, and no observability once it's running in production.

Evaluate before it ships

An eval is a test suite for model behavior, not code paths. It runs a fixed set of inputs against the model or system, scores the outputs against expected behavior, and gives a pass/fail or a score you can track over time and across model or prompt changes.

OpenAI's own guidance on this is direct: strong evals are what makes an LLM application resilient to code and model changes, the same way a unit test suite makes a codebase resilient to refactors (OpenAI — Evaluation best practices). Treat a prompt change or a model version bump the same way you'd treat a dependency upgrade: nothing ships without the eval suite passing first.

What actually belongs in the suite

A useful eval suite for an operations-facing AI system covers three categories: known-good cases the system must handle correctly, known-hard edge cases (empty input, contradictory signals, adversarial phrasing), and regression cases captured from real production failures.

Known-good cases: the routine 80% of traffic, used to catch silent quality regressions.
Known-hard cases: nulls, empty carts, conflicting inventory signals, ambiguous customer intent.
Regression cases: every production incident becomes a permanent eval case, so the same failure can't ship twice.

An eval suite that only contains happy-path cases isn't testing resilience. It's testing that the demo still works.

Design for the model being wrong

Every AI call in a business-critical path needs a defined behavior for when it's wrong, slow, or unavailable. That's not a new idea. It's the circuit breaker pattern, applied to a model endpoint instead of a downstream service.

The circuit breaker pattern trips after repeated failures, stops sending traffic to the failing dependency, and routes to a fallback, whether that's a cached response, a simpler deterministic rule, or a queued retry, until the dependency recovers (Microsoft Learn — Circuit Breaker pattern). The same logic applies whether the failing dependency is a payment gateway or a forecasting model.

What the fallback actually looks like

For a demand-forecasting model, the fallback is the deterministic rule the business used before the model existed: reorder points based on trailing 8-week average sales, not a frozen or best-guess prediction. For a support chatbot, the fallback is a clean handoff to a human queue, not a generic error message.

The fallback path has to be built and tested before the AI path ships, not sketched out after the first incident. If nobody has run the fallback in the last quarter, it isn't a fallback, it's an assumption.

Guardrails for actions, not just answers

Anthropic's own guidance on agent design separates the model that acts from the model or check that screens the action, because a single model call handling both the task and its own safety check tends to perform worse than splitting the two (Anthropic — Building effective agents). For anything irreversible, a canceled order, a deleted record, a large discount applied, that guidance also points to a human-approval checkpoint before the action executes.

The same guidance is direct about where to build confidence before granting an agent more autonomy: extensive testing in sandboxed environments, with the appropriate guardrails already in place, because an autonomous agent's mistakes compound in a way a single bad completion doesn't. A sandbox that mirrors production data shapes, without production side effects, is what makes that testing meaningful instead of theoretical.

Cost is a resilience metric too

A model that's technically "up" can still take an operation down by burning through a rate limit or a budget in minutes, an outage that looks nothing like a typical service failure. Both OpenAI and Anthropic enforce hard limits on requests and tokens per minute, tied to account usage tier, and return a 429 response once a workload exceeds them (OpenAI — Rate limits, Anthropic — Rate limits).

A circuit breaker tuned only for outright errors misses this failure mode entirely, since a 429 is the provider working as designed, not failing. The breaker's trip condition needs to include rate-limit responses and cost-per-hour thresholds, not just 5xx errors, or a traffic spike turns into a silent request pile-up instead of a clean, tested degradation.

function callForecastModel(input) {
  if (circuitBreaker.isOpen()) {
    return deterministicReorderRule(input); // fallback, always tested
  }
  try {
    const result = model.predict(input);
    if (!evalGate.passes(result)) {
      circuitBreaker.recordFailure();
      return deterministicReorderRule(input);
    }
    circuitBreaker.recordSuccess();
    return result;
  } catch (err) {
    circuitBreaker.recordFailure();
    return deterministicReorderRule(input);
  }
}

Observe the thing once it's live

An eval suite tells you the model behaves correctly before deployment. Observability tells you whether it's still behaving correctly at 2am on a Tuesday, under real traffic, with real data drift.

The OpenTelemetry project's semantic conventions for generative AI standardize what a trace, metric, or event for an LLM call should record: the model invoked, token counts in and out, latency, and, when opted in, the actual prompt and completion content (OpenTelemetry — Generative AI semantic conventions). That standardization matters because it's what lets an AI call show up in the same trace and alerting pipeline as every other service call, instead of living in a separate, bespoke dashboard nobody checks during an incident.

The three signals and what each one catches

Signal	What it captures	What it catches that the others miss
Traces	The full path of a single request, including the model call, tool calls, and downstream services	Where in a multi-step agent flow a specific request failed or stalled
Metrics	Aggregates over time: latency percentiles, token usage, error rate, circuit-breaker trip count	Slow drift, like rising latency or a creeping error rate, that no single trace shows
Events	Discrete records of prompt and completion content, tool inputs and outputs	What the model actually said or did, needed to debug a specific bad output after the fact

A model call without a trace ID is a black box wearing an API response. The first incident review after go-live is where that gap gets discovered, which is the most expensive place to discover it.

Governance is part of the resilience story, not a separate track

NIST's Generative AI Profile, published as a companion to the broader AI Risk Management Framework, catalogs risks specific to generative AI, including confabulation (a model stating something false with confidence), and ties each risk to concrete actions across the AI lifecycle (NIST AI 600-1 — Generative Artificial Intelligence Profile). Confabulation is precisely the failure mode an eval suite and a fallback path exist to catch, so governance and reliability engineering point at the same work, not two separate compliance and engineering tracks.

The base AI Risk Management Framework itself, published in January 2023, frames risk management as a continuous lifecycle activity: govern, map, measure, manage, not a one-time sign-off before launch (NIST — AI Risk Management Framework).

Where the risk categories map onto the engineering work already described

Three of the profile's named risk categories line up directly with the mechanisms above, which is the useful part for an engineering team, not the compliance checklist part.

Confabulation is caught by the eval suite's known-hard cases and by observability events, since a confabulated output still has to pass through both before it reaches a customer.
Information security maps to the guardrail model that screens actions before they execute, particularly for agents with tool access to internal systems.
Human-AI configuration maps to the human-approval checkpoint on irreversible actions, since it covers exactly the question of when a person, not a model, should make the final call.

Treating the profile as a second, unrelated workstream is how governance becomes a rubber stamp applied after the engineering is done. Treating it as a naming convention for risks the eval suite and guardrails already need to cover is how it actually changes what gets built.

Sequencing it into an actual operating model

None of the pieces above work in isolation. The sequence matters, and skipping a step is how teams end up with an eval suite nobody runs or a dashboard nobody watches.

Build the eval suite first, before the model touches production traffic, using known-good, known-hard, and regression cases.
Run the model in shadow mode against real traffic, logging predictions without acting on them, and compare against the eval suite.
Roll out with the circuit breaker and fallback path active from day one, not added after the first incident.
Instrument every model call with traces, metrics, and events using a standard schema, not a bespoke logging format.
Feed every production incident back into the eval suite as a new regression case before closing the incident.

That loop, eval, guarded rollout, observe, feed back, is what turns "we added AI" into an operation that's actually harder to break. Skipping steps 1, 3, or 5 is how a resilience initiative becomes the outage postmortem's first bullet point.

FAQ

What's the difference between an eval and a unit test?

A unit test checks that code produces an exact expected output for a given input. An eval scores a model's output against a rubric or expected behavior, because model outputs vary even for the same input, and a pass/fail or numeric score is what makes that variation trackable over time (OpenAI — Evaluation best practices).

Does every AI feature need a circuit breaker?

Every AI call in a path that affects a customer-facing outcome or a financial transaction needs a defined failure behavior. Circuit breakers make that failure behavior automatic instead of relying on someone noticing the model is down.

What should the fallback be for a customer-facing chatbot?

A clean handoff to a human support queue, with the conversation history attached, rather than a generic error message or a repeated retry against a failing model.

Is NIST's AI Risk Management Framework mandatory?

No. It's a voluntary framework, but it's the most detailed public catalog of generative-AI-specific risks and mitigations available, and it maps cleanly onto engineering practices like evals and incident response (NIST AI 600-1).

Why does OpenTelemetry matter specifically for AI observability?

Because it standardizes attribute names and structure for model calls, which is what lets an AI call appear in the same tracing and alerting tooling as every other part of the stack, instead of requiring a separate, one-off monitoring setup per AI feature (OpenTelemetry GenAI semantic conventions).

Where should the human-approval checkpoint go in an agent workflow?

Immediately before any irreversible action, canceling an order, issuing a refund above a threshold, deleting a record, per Anthropic's own guidance on agent guardrails (Anthropic — Building effective agents).

Does a rate limit count as a resilience incident?

Yes. Both major providers enforce hard per-minute request and token caps by usage tier, and hitting one mid-traffic-spike is functionally an outage for anything depending on that call, even though the provider itself is healthy (OpenAI — Rate limits).

Why test agents in a sandbox instead of a staging environment with real traffic?

Because an autonomous agent's errors can compound across multiple tool calls before anyone notices, and a sandbox lets that failure mode play out against realistic data without touching a real order, refund, or customer record (Anthropic — Building effective agents).

References

From the Destm engineering archive. For current work on this topic, start at Solutions or the blog.