Engineering·July 2024·9 min read

Mitigating Downtime: Advanced Techniques for Managing High Traffic Events

Managing downtime during a traffic spike isn't a list of tools to buy. It's an error budget that tells you how much risk you're allowed to take, health checks that catch failure before users do, and a deploy strategy that lets you back out fast when something breaks anyway.

A traffic spike doesn't cause downtime by itself. It exposes whatever wasn't load-tested, wasn't monitored, or couldn't fail over cleanly, and it does it in front of the largest audience the site will see all quarter.

Google's Site Reliability Engineering discipline gives this problem a vocabulary that holds up better than a generic checklist: service level objectives, error budgets, and blameless postmortems. The techniques below sit inside that frame.

Service level objectives set the actual target

An SLO (service level objective) is a target value for a service level, measured by an SLI (service level indicator, like request latency or error rate). Without one, "is the site down" has no agreed answer during an incident, which is exactly when a team can least afford to argue about it.

Error budgets turn reliability into a number you can spend

An error budget is the inverse of the SLO: if the target is 99.9% availability, the budget is the remaining 0.1%, and the team can spend it on risk (a big feature launch, an aggressive traffic event) as long as it doesn't run out. Once the budget's spent, the trade shifts toward stability work over new features until it recovers.

This reframes a high-traffic launch correctly: it's not a reliability afterthought bolted onto a marketing calendar, it's a planned withdrawal from a budget the team already knows the size of.

An error budget doesn't ask "did we hit five nines." It asks whether you have enough budget left this quarter to take the risk you're about to take. That's a planning question, not a postmortem question.

Health checks that actually catch failure

A server that responds to a ping but can't serve a real request is worse than one that's visibly down, because nothing routes traffic away from it automatically. The fix is a health check that tests the thing that actually matters, not just process liveness.

Liveness, readiness, and startup probes

Kubernetes separates this into three distinct checks: a liveness probe restarts a container that's deadlocked, a readiness probe removes a pod from load balancing when it can't currently serve traffic (without killing it), and a startup probe gives slow-booting containers time before liveness checks apply.

Conflating these is a common failure mode: a liveness probe that's too aggressive during a traffic spike restarts healthy pods that are just momentarily slow, turning a load problem into a cascading restart storm.

Load balancer health checks matter just as much

Cloud load balancers run their own health checks independent of the application's internal ones. AWS Auto Scaling health checks, for example, can pull from EC2 status, an Elastic Load Balancer target check, or a custom check, and a misconfigured one either routes traffic to dead instances or kills healthy ones under load.

Deploy strategies that don't require a maintenance window

The traffic event itself is the worst possible time to discover a deploy strategy doesn't actually avoid downtime. Three strategies cover almost every real case, each with a different risk-versus-cost trade.

Strategy	How it works	Rollback speed	Infra cost
Rolling update	Replace instances gradually, a few at a time, behind the same load balancer	Slow — has to roll forward or back through remaining instances	Low — no duplicate fleet needed
Blue-green	Run two full environments; switch traffic at the router/DNS level	Fast — flip the switch back	High — two full environments running simultaneously
Canary	Route a small percentage of traffic to the new version, expand gradually	Fast — pull the canary traffic back immediately	Moderate — small extra footprint for the canary

Kubernetes implements rolling updates natively, controlling how many pods can be unavailable or created above the desired count during the rollout. Google Cloud's guidance on deployment and testing strategies covers blue-green and canary patterns for teams not running on Kubernetes natively.

Canary deployments need a real automated gate

A canary that a human has to remember to check isn't a safety mechanism, it's a delay. The gate needs to be automated: error rate and latency on the canary compared against the baseline, with an automatic rollback if the canary's metrics cross a threshold, not a Slack message asking someone to look at a dashboard.

Load and stress testing before the event, not during it

Load testing simulates expected traffic to find the breaking point before real users do; stress testing pushes past that point deliberately to see how the system fails. Both answer different questions and both matter before a known traffic event.

Apache JMeter: open-source, mature, strong for HTTP and API load testing with a large plugin ecosystem.
k6: scriptable in JavaScript, built for CI integration, popular for teams that want load tests versioned alongside application code.

What to actually look for in the results

The number that matters isn't "requests per second the system can handle" in isolation, it's where latency starts degrading relative to the SLO defined earlier. A system that handles 10x current traffic at 3x the acceptable latency hasn't actually passed the test.

Define the traffic pattern to simulate, based on the actual event (a flash sale spikes differently than steady organic growth).
Run the test against a staging environment that mirrors production infrastructure as closely as possible, not a scaled-down copy.
Identify the first component to degrade (database connection pool, a specific API dependency, cache eviction rate) rather than stopping at "the site got slow."
Fix that bottleneck and re-test, rather than assuming a fix elsewhere addressed it.

Rate limiting has an actual HTTP status code

Rate limiting isn't just an infrastructure setting, it's part of the HTTP standard. RFC 6585 defines status code 429 (Too Many Requests) specifically for this case, distinct from a generic error, so clients and monitoring tools can tell the difference between "your request was rejected because you're over your limit" and "the server is broken."

Retry-After tells the client when to come back

Pairing a 429 (or a 503) with the Retry-After header tells a well-behaved client exactly how long to wait before retrying, instead of hammering the server immediately or backing off with an arbitrary guess. Most commerce clients (browsers, mobile apps, even bots) respect this header when it's present, which turns a blunt rejection into a coordinated backoff.

Graceful degradation beats an all-or-nothing outage

When capacity runs out, the choice isn't between "fully up" and "fully down." HTTP 503 Service Unavailable exists precisely for the case where a server is temporarily unable to handle a request due to overload, and returning it deliberately for non-critical functionality is a legitimate strategy.

A checkout flow that stays up by temporarily disabling product recommendations, review widgets, and other non-essential features is a better outcome than a checkout flow that goes down entirely because those same features exhausted a shared resource pool. Deciding in advance which features are load-sheddable, and building the kill switch before the traffic event, is cheaper than deciding during one.

Status pages and the blameless postmortem

A status page (Atlassian's Statuspage and similar tools) does something monitoring dashboards don't: it tells customers what's happening before they have to ask, which measurably reduces support ticket volume during an incident even though it doesn't fix anything technical.

The postmortem is where the next incident gets prevented

Google's SRE practice treats blameless postmortems as a discipline: the goal is identifying contributing factors and system gaps, not identifying who to blame. A postmortem culture that punishes the person who made the change trains people to hide problems instead of surfacing them.

A blameless postmortem isn't about being nice. It's the only structure that gets an engineer to describe exactly what they did wrong in enough detail to actually fix the system, instead of a sanitized version that protects them.

A pre-event runbook checklist

Before a known high-traffic event (product launch, major sale, expected press coverage), this is the minimum that should already be true.

SLOs defined for the critical user journeys (checkout, search, product page load), not just overall uptime.
Load test completed against production-equivalent infrastructure, with the first bottleneck identified and fixed.
Liveness, readiness, and load-balancer health checks verified independently, not assumed to be correctly configured.
A deploy strategy chosen in advance (rolling, blue-green, or canary) with a tested rollback path, not decided mid-incident.
A status page ready to update, and an on-call rotation that knows who updates it.
Rate limiting configured at the edge for both fair usage and abuse mitigation, tested against expected legitimate peak load so it doesn't throttle real customers.

FAQ

What's the difference between an SLO and an SLA?

An SLO is an internal target a team holds itself to; an SLA is an external, often contractual, commitment to customers, usually set looser than the internal SLO to leave margin for error.

How aggressive should a liveness probe's failure threshold be?

Loose enough that a momentary slowdown under real load doesn't trigger a restart, since restarting a pod that's merely slow (not deadlocked) removes capacity at exactly the moment the system needs it most.

Is canary deployment overkill for a small team?

Not if the risk of a bad deploy during a specific high-traffic window is high enough to justify the setup cost. For steady, low-stakes traffic, a rolling update with good health checks is often sufficient.

Does a status page actually reduce support load during an incident?

It reduces the volume of "is it just me" tickets and social posts, since a visible, updated status page answers that question without a support agent needing to. It doesn't reduce the underlying incident-resolution work.

What's the single most common cause of downtime during a traffic spike that teams don't test for?

A downstream dependency (payment processor, third-party API, database connection limit) that wasn't included in the load test, because the team tested their own application's capacity and assumed everything it calls will scale the same way.

Should a 429 or a 503 be used when rate limiting a client?

Use 429 when the client itself has exceeded a defined limit (per-user or per-API-key quota), since that's specifically what the status code communicates. Use 503 when the server as a whole is overloaded and shedding load indiscriminately, which is a system-wide condition rather than a per-client one.

References

From the Destm engineering archive. For current work on this topic, start at Solutions or the blog.