Your BFCM Load Test Is Testing the Wrong Thing

Most BFCM load tests prove the CDN can serve a cached homepage, which was never in doubt. Sale day fails in the write path: hot-SKU inventory rows, checkout contention, queue backlogs, and payment-provider rate limits, and a homepage RPS number says nothing about any of them.

The load-test report lands in the pre-peak review deck every October. Green across the board, a large requests-per-second figure, read latencies well inside target, sign-off all around.

Then the sale starts, and the homepage stays fast the entire time carts are timing out. The report was true and the site still fell over, because the report measured the wrong system.

Reads are cache-shaped, writes are contention-shaped

Browse traffic scales the way load tests love. It is cacheable, stateless, and horizontally boring, so adding CDN and edge capacity pushes the RPS number up until the test budget runs out.

Checkout traffic has the opposite physics. Every order decrements inventory, creates payment state, and writes to systems that maintain invariants, and invariants are enforced by some form of serialization.

A load test weighted toward browse, and most are because browse scripts are easy to write, measures the part of the system that was never going to fail. The part that fails is the part the script skipped because it needed test cards and idempotency keys.

The funnel a browse-weighted load test never exercises: capacity is decided at the 2 pinch points on the right, not at the wide end.

BFCM is 3 different tests, and you probably ran a 4th

Grafana's k6 documentation on load test types separates average-load, stress, spike, soak, and breakpoint tests, and its framing is blunt: a good strategy requires more than executing a single script, because different traffic patterns create different risk profiles. Most pre-peak programs run 1 average-load test and call it coverage.

Map the types to the questions the weekend will actually ask.

• Spike test: the doorbuster drop. Does the system survive a sudden, massive, short surge, or does it fall over and stay down after the surge passes?
• Stress test: the sustained peak day. Where does behavior degrade once load runs beyond the expected average, and is the degradation graceful or a cliff?
• Soak test: the full Thursday-to-Monday arc. What leaks, fills, or fragments over 96 hours that a 1-hour run never touches?
• Breakpoint test: the ceiling itself. At what load do you start failing, so day-of decisions reference a number instead of a feeling?

Each profile fails differently, which is the point of running them separately. A connection-pool leak shows up in the soak and never in the spike.

The doorbuster is 1 database row

Inventory for the hero SKU is, in most systems, a single row. 10,000 concurrent buyers of that SKU are 10,000 transactions that must serialize on it, and application-tier autoscaling changes nothing about that.

PostgreSQL's explicit locking documentation spells out the mechanics: a row-level lock acquired by an UPDATE or a SELECT FOR UPDATE blocks every other writer of that row until the transaction commits or rolls back. Throughput on a hot row is therefore bounded by 1 divided by lock hold time, so a checkout transaction that holds the row for 50ms caps that SKU near 20 orders per second no matter what the hardware costs.

-- the doorbuster bottleneck, in 3 statements
BEGIN;
SELECT qty_available FROM inventory
  WHERE sku = 'HERO-001' FOR UPDATE;  -- every rival checkout waits here
UPDATE inventory SET qty_available = qty_available - 1
  WHERE sku = 'HERO-001';
COMMIT;  -- lock hold time = your per-SKU ceiling; the row does not scale out

This is not a small-shop problem. Shopify's engineering write-up on flash-sale traffic describes a single merchant's February 2016 sale taking down not only her own store but every store on the same database shard, and the fix they built was a checkout throttle at the edge, in Nginx and Lua, rather than a bigger database.

The lesson transfers directly. Decide the maximum checkout rate you can serialize, then queue or throttle above it at the edge on purpose, instead of letting lock waits queue customers for you by accident.

If your load test spreads purchases uniformly across the catalog, it never creates this contention. Skew the SKU distribution until a handful of doorbusters dominate write traffic, because that is what the merchandising calendar already promises will happen.

A load test with uniformly distributed SKUs is a test of a sale you are not running. The sale you are running concentrates its writes on the 20 products in the email subject line.

Queue depth lies, queue age does not

Async queues between checkout and everything downstream, order management, email, loyalty, the ERP sync, absorb spikes, which is why they are everywhere. They also hide overload right up until recovery becomes the incident.

The Amazon Builders' Library article on queue backlogs makes the core point: a queue-based system converts overload into latency debt, and a backlog that built up during a surge keeps punishing you long after the surge ends, because new work keeps arriving while old work drains. Its instrumentation advice is the part to steal, which is to watch the age of the oldest message rather than raw depth.

Age is the honest metric because it is the customer's experience in seconds. Depth tells you how much work is waiting; age tells you how stale the "order confirmed" email at the back of the line already is.

An order confirmation that arrives 6 hours late is an incident your load test marked as a pass, because the enqueue succeeded in 4 milliseconds.

So the test must assert on drain, not on enqueue. After the simulated spike ends, measure how long oldest-message age takes to return to baseline, and check whether consumers scale on age or on depth, because scaling on depth reacts a full backlog too late.

The same article covers the poison-pill case, a message that fails processing repeatedly and jams the pipeline, which the spike makes far more likely because malformed orders arrive at volume. If your consumers have no dead-letter path, 1 bad order placed at 6:01am can stall the confirmation stream behind it, and the soak test is the only place you will meet that before customers do.

Shed load on purpose or shed customers by accident

The companion Builders' Library article on load shedding draws the distinction that matters for peak: throughput is requests accepted, goodput is requests completed usefully, and past the overload point extra throughput reduces goodput because work times out mid-flight while still consuming resources.

The retail translation is a priority ladder. Protect payment and checkout first, then cart, then product pages, then search, then everything decorative, and hold the feature flags that let you walk down that ladder under pressure.

I made the case in the September piece on code freezes that those load-shedding flags need production flips before peak. The load test is where you rehearse them: flip the recommendations flag mid-test and confirm the saved capacity actually reaches checkout instead of vanishing into retries.

Your payment provider has a rate limit and your mock does not

Stripe documents its limits precisely: a global default of 100 requests per second per account in live mode, 25 per second on individual endpoints unless noted otherwise, and 429 responses with a reason header beyond that. Whichever provider you use, the shape is the same, and the budget is per account, not per pod, so horizontal scaling multiplies your ability to hit it.

A load test that stubs the payment API with an always-200 mock validates nothing about sale day. The failure mode that matters is a burst of 429s at the exact minute of peak revenue: does checkout retry with backoff and idempotency keys, degrade into a visible queue, or hard-fail paying customers?

Run payment traffic against the provider's sandbox and treat its throttling as the feature under test, not as an inconvenience to mock away. And if your projected order rate approaches the documented live limits, that conversation with the provider happens in October, not in a sev-1 channel in November.

What the report measures vs what the weekend exercises

Dimension	Typical BFCM load test	What sale day actually exercises
Traffic mix	Browse-weighted, scripted from the sitemap	Checkout-weighted bursts driven by the promo calendar
SKU distribution	Uniform across the catalog	Extreme skew; doorbusters serialize on single inventory rows
Payment path	Always-200 mock	Provider rate limits, 429 bursts, retry and idempotency behavior
Async work	Enqueue latency	Drain time and oldest-message age after the spike
Success metric	RPS and read latency percentiles	Goodput: completed orders per minute under contention
Duration	A 1-hour run on a weekday afternoon	A 96-hour weekend of leaks, log growth, and queue creep

The test plan worth running

1. Build the traffic model from last year's peak-hour funnel ratios, sessions to carts to checkouts to payments, not from this year's homepage traffic hopes.
2. Skew SKU demand hard, so a few doorbusters dominate write traffic the way the campaign calendar says they will.
3. Run spike, stress, and soak profiles as separate tests with separate pass criteria.
4. Point payment traffic at the provider sandbox, script 429 injection, and assert on retry behavior and duplicate-charge prevention.
5. Assert on goodput, oldest-message age after the spike, and database lock waits. RPS goes in the appendix.
6. Rehearse the degradation ladder while under load: flip the shedding flags and confirm the freed capacity shows up where you need it.

The homepage RPS number will still look excellent in the deck. Let it be the appendix, and make the cover slide the checkout goodput curve with the hot-SKU skew turned on.

FAQ

We are on a managed commerce platform. Isn't scale the vendor's problem?

The vendor scales browse and their own checkout, and their contract says so. Your customizations, middleware, ERP sync, and third-party integrations are yours, and the queue between their webhook and your warehouse is where the backlog forms.

When should the load test happen?

Finished, with fixes landed and re-tested, by the end of October. A failure found in mid-November is a fact you get to be anxious about rather than a defect you get to fix.

How do we generate checkout load without charging real cards?

Provider sandboxes exist for exactly this, with test cards and configurable failure responses. Keep the sandbox keys out of production config, and treat the sandbox's lower documented rate limits as a bonus stress on your backoff logic.

Shouldn't we just reserve inventory at add-to-cart to avoid the hot row?

Reservations move the contention rather than removing it, and expiring holds under bot pressure creates its own oversell and undersell modes. I wrote about the deeper fix, treating inventory as an event stream with per-consumer projections, in Inventory Is a Stream, Not a Table.

Is the homepage test worthless then?

No, keep it as the smoke layer that validates scripts and baseline health, which is exactly where the k6 taxonomy places it. It is necessary, cheap, and nowhere near sufficient.

References

• Grafana k6 docs — load test types
• PostgreSQL docs — explicit locking
• Shopify Engineering — surviving flashes of high-write traffic using scriptable load balancers
• Amazon Builders' Library — avoiding insurmountable queue backlogs
• Amazon Builders' Library — using load shedding to avoid overload
• Stripe docs — rate limits