The Week Before Black Friday: An Engineering Runbook
The week before Black Friday is not for building anything. It is for removing surprise: scope the freeze, flip every flag once, time a real rollback, rehearse the incident you will probably have, and decide who talks to whom at 2am before 2am arrives.
Black Friday 2026 lands on November 27, which makes the week of November 23 the last week that matters. This runbook covers that week, day by day.
It is published 3 weeks early on purpose. Half the items below have prerequisites, and the worst time to discover a prerequisite is the Monday of the week itself.
The organizing principle: by this week, capacity is what it is and the code is what it is. Everything left to gain comes from verification and rehearsal, and everything left to lose comes from surprise.
The shape of the week
Each day has 1 job. Resist the urge to compress them into a single heroic Wednesday, because each job produces findings the next day consumes.
| Day | The job | The artifact it produces |
|---|---|---|
| Monday, Nov 23 | Scope and publish the freeze | A 1-page policy: frozen categories, named exception approvers |
| Tuesday, Nov 24 | Flag inventory and production flips | The degradation ladder, tested, with owners per rung |
| Wednesday, Nov 25, morning | Rollback drill, timed, on production | A measured rollback number the on-call can rely on |
| Wednesday, Nov 25, afternoon | Incident rehearsal and comms tree | Named roles, 1 channel, a stakeholder update cadence |
| Thursday, Nov 26 | Thanksgiving: final checklist, skeleton staffing, watch begins at the evening opening | A calm start instead of a scramble |
| Friday, Nov 27 | Watch, shed, roll back. Nothing else | An honest log for the post-mortem |
The week as a sequence: 4 days of verification and rehearsal feeding a watch-only weekend, with a rehearsed incident loop as the only branch.
Monday: scope the freeze precisely
I argued in September's piece on code freezes that a blanket freeze is a symptom, and that the durable fix is deploy automation, canaries, flags, and pinned dependencies built by October. Whatever portion of that you built, Monday is when you scope the freeze you will actually run.
Freeze by category, not by calendar blanket. Schema migrations, infrastructure topology, framework upgrades, and dependency bumps stop; flag-gated application code and canaried funnel fixes keep shipping through the standard pipeline.
Publish it as 1 page, and put 2 phone numbers on it: the person who approves exceptions and the person who approves them when the first person is asleep. An exception process that resolves to "post in a channel and hope" is not a process.
Monday is also when you widen the definition of a deploy. The tag manager, the promotion engine, and the search-index configuration all change production behavior, so they get the same freeze scoping and the same named approvers as code.
Tuesday: the flag inventory and the degradation ladder
Tuesday's job is proving that every load-shedding lever works, this week, in production. A flag last flipped in staging in August is a hypothesis, and sale day is a bad place to test hypotheses.
Build the ladder explicitly, ordered by what you will sacrifice first.
- Review imports, social feeds, and anything decorative on the product page.
- Recommendations and personalization, falling back to static merchandising.
- Search suggestions and autocomplete, keeping search itself.
- Loyalty and promotional banners, keeping earned discounts honored at checkout.
- Never on the ladder: checkout, payment, cart, and auth. These are what the ladder protects.
Flip each rung on and off in production during Tuesday's low-traffic hours, and record 3 things per rung: the owner, the measured effect, and the caches that needed clearing before the flip took. That last one is the classic surprise, and I covered rehearsing flips under actual load in the load-testing piece.
Every rung of the degradation ladder gets flipped in production this week, by the person who owns it, with the effect written down. A lever nobody has pulled is scenery.
Wednesday morning: the rollback drill
The single number that changes incident behavior on Friday is the measured time from "roll it back" to healthy metrics. Teams with that number act in minutes; teams without it hold debates while the error rate compounds.
The mechanics are usually already there. Kubernetes Deployments keep a rollout history and support rolling back to a prior revision, with the retained revision count set by the resource's revision history limit. What is usually missing is evidence the path works under your RBAC, your pipeline, and your on-call's fingers.
So drill it, on production, on Wednesday morning.
- Ship a harmless change through the normal pipeline, not a shortcut.
- Have the engineer who is on call Friday, not the release owner, execute the rollback.
- Time it from decision to dashboards green, and write the number into the incident channel topic.
- Drill the non-code reverts too: a promotion rule, a tag-manager container version, a search relevance config.
If any revert path fails or takes longer than your patience on a peak day, fixing it jumps the queue immediately. Everything else on Wednesday waits behind it.
Wednesday afternoon: rehearse the incident and draw the comms tree
The Google SRE book's chapter on incident management describes what unmanaged incidents look like: everyone debugging at once, nobody communicating, decisions made by whoever is loudest. Its fix is borrowed from emergency services, a separation of responsibilities where incident command, operational work, and communications are distinct roles held by distinct people.
Wednesday afternoon is when those roles get names for the weekend. Not team names, human names, per shift, written where everyone can see them.
PagerDuty's incident response documentation, which is the public version of their internal process, is worth an hour of the team's afternoon. Two of its points earn their place here: the incident commander runs the incident without needing to be the most senior engineer present, and severity definitions are agreed before the incident so nobody negotiates them during one.
Then run a tabletop, 90 minutes, 3 scenarios: a payment-provider 429 burst at 6:05am, an order-queue backlog growing past its drain rate, and a bad change that must be rolled back under peak traffic. Each scenario ends when someone has said, out loud, who declares the incident, who mitigates, and who updates stakeholders.
The comms tree is 5 lines: who declares, in which single channel the incident lives, who owns the status page, who updates execs and on what cadence, and which vendor escalation numbers are already saved. Write it Wednesday, because none of those 5 answers improves at 2am.
Collect the vendor half deliberately. Payment provider, CDN, platform, and tax-service escalation contacts go in the incident channel topic, because filing a normal support ticket during your sev-1 is how a 20-minute problem becomes a 4-hour one.
Severity, decided in advance
Severity levels only work when they are examples, not adjectives. "Major" means nothing at 6am; "checkout is failing for a class of customers" means everything.
| Severity | Peak-weekend example | Response |
|---|---|---|
| Sev-1 | Checkout down, payment authorization failing broadly, site unreachable | Incident declared, commander named, status page updated, exec cadence starts |
| Sev-2 | One funnel step degraded, a ladder rung flipped to compensate | Incident declared, no status page yet, fix or stabilize before promoting severity |
| Sev-3 | A non-funnel feature broken: reviews, wishlist, a content block | Ticket and a note in the handoff. Nobody gets woken up |
Argue about the boundaries on Wednesday, with coffee, until everyone accepts them. The alternative is arguing about them on Friday, with an error rate climbing behind the debate.
The dashboards that matter, and the alerts that do not
The SRE book's monitoring chapter gives the 4 golden signals: latency, traffic, errors, and saturation. That is the systems layer, and for a retail peak it needs a business layer on top, because "did we break checkout" answers faster in orders per minute than in CPU.
The wall for the weekend is 6 panels.
- Orders completed per minute, against the same hour last year as context.
- Checkout funnel conversion by step, so a payment failure reads differently from a cart bug.
- Payment authorization success rate, with the provider's status page linked beside it.
- Golden signals for the checkout and cart services specifically, not a service-mesh average.
- Oldest-message age on the order-processing queues, which is drain health in customer terms.
- Error rate on the top 5 third-party calls, because their sale day is happening at the same time as yours.
Alert hygiene belongs to Wednesday too. The SRE workbook's chapter on SLO alerting makes the case for paging on error-budget burn rate rather than on raw thresholds, and its logic matters most this weekend: traffic is 5x normal, so any static threshold tuned in October is either silent or screaming.
Silence every alert that does not demand a human action within minutes. A pager that fires 40 times on Friday trains the on-call to ignore it by Saturday, which is the most expensive possible training.
One more Wednesday task on the dashboards themselves: annotate them with the campaign calendar. A traffic spike at 6:00am reads as an attack until someone remembers the email send, and the marketing schedule drawn on the graph converts that panic into context for free.
An on-call structure that survives 96 hours
Cyber weekend is a marathon staffed like a sprint. The SRE book's chapter on being on-call is direct about sustainable load, arguing for balanced rotations and capping incidents per shift so each one gets handled properly and the engineer recovers between them.
The translation for 1 retail weekend is unglamorous scheduling.
- Shifts of 6 to 8 hours, never a 24-hour hero. Heroics on Friday are debt collected on Saturday night.
- A primary and a secondary per shift, with the secondary owning comms so the primary can debug uninterrupted.
- Handoffs with a written note: current state, open weirdness, changes made, and anything watched but not yet paged.
- The incident commander rotation staffed separately from the debugging rotation, per the role separation above.
- Explicit rest rules for anyone who worked an overnight incident, decided now, not negotiated while exhausted.
Thursday: the quiet day that is not
Thursday is Thanksgiving in the US, most of the team is at a table somewhere, and most campaign calendars open the sale that evening anyway. Which means the final checklist finishes Thursday morning, on skeleton staffing, hours before the first real wave.
- Autoscaling floors raised ahead of the curve, since scaling reactively into a vertical traffic wall means serving errors while instances boot.
- Caches warm, verified by hit-rate dashboards rather than by hope.
- TLS certificates checked for expiry dates inside the window. It is a 2-minute check against a legendarily embarrassing failure mode.
- Heavy scheduled jobs, reindexing, exports, batch syncs, paused or moved out of peak hours.
- The war-room channel open, the dashboard wall up, the comms tree pinned.
The evening opening is a full rehearsal with real money. Whoever holds the Thursday-night shift watches the same 6 panels, and whatever wobbles at 7pm Thursday gets the overnight to be understood before Friday's volume arrives.
Friday: the discipline of doing nothing
Friday's job is watching, and the hard part is that watching feels passive while adrenaline says act. The only moves on the board are the rehearsed ones: walk down the degradation ladder, execute the measured rollback, or declare an incident and run Wednesday's script.
Anything else, a config tweak, a quick fix, a "harmless" cache clear that was not on Tuesday's list, is a new experiment launched at the year's worst moment. The log of what you did and when, kept honestly, is Friday's second deliverable, because December's post-mortem is only as good as it.
Keep that log as a running timeline in the war-room channel: every flag flip, every scaling change, every anomaly noticed and dismissed, timestamped as it happens. Reconstructing a timeline from memory on Sunday produces fiction, and fiction makes for confident, wrong post-mortems.
Monday night: close the window properly
Cyber Monday's evening is the end of the acute phase, and it deserves 30 deliberate minutes rather than a slow drift back to normal. Lower the scaling floors on purpose, unpause the batch jobs in a chosen order, and keep the freeze scoping in place until traffic actually normalizes, which is rarely Tuesday.
Then schedule the post-mortem for the first half of December, while the timeline log is fresh and the pain is specific. Every rehearsal gap the weekend exposed, a flag that lied, a rollback that stalled, a severity argued mid-incident, is next year's September backlog, and writing it down now is the cheapest planning work of 2027.
And say thank you like you mean it. The people who watched dashboards through Thanksgiving dinner are the capacity plan nobody budgets for.
What this week is not for
The list is short and absolute, and every item on it has a story behind it somewhere in retail engineering.
- No schema migrations, however small the table looks.
- No framework, runtime, or dependency upgrades outside security patches.
- No new third-party tags, pixels, or scripts, whatever marketing promises they cost.
- No infrastructure topology changes after Wednesday: no new regions, no load-balancer swaps, no DNS.
- No experiments, including A/B tests that touch the funnel. The variant you have is the variant you sell with.
FAQ
We are a 6-person team. Which parts of this still apply?
All of it, smaller. The freeze page becomes a paragraph, the tabletop becomes 30 minutes over lunch, and the roles collapse to 2 people, but the rollback still gets timed and the comms answer still gets written before Friday.
Isn't a rehearsal 2 days before the sale too late?
Wednesday's tabletop is the final walkthrough, not the first exposure, and its real product is the named roles and the comms tree for that specific weekend. If the team has never run an incident process at all, start that in September, not Thanksgiving week.
Who should be the incident commander?
Someone who can coordinate calmly and delegate, which is a skill, not a seniority level. The role runs the incident and keeps the state coherent while others debug, so the strongest debugger is usually the wrong choice.
Should executives be in the war room?
They should be on the comms cadence, receiving updates at an agreed interval from the communications role. An open question from a VP in the incident channel costs the team its focus at exactly the moment focus is the scarce resource.
What about the weekend after? Everyone is exhausted and traffic is still high.
Cyber Monday closes the window, and the returns surge that follows is its own capacity event with its own prep. Keep the on-call structure running through the first week of December, and schedule the post-mortem while memories are fresh.
Our platform vendor runs their own peak readiness. Do we still need this?
Yes, because their runbook covers their blast radius and stops at your customizations, integrations, and third parties. The queue between their webhook and your ERP is nobody's runbook but yours.
References
Related reading
More in OperationsThe Peak-Season Code Freeze Is a Symptom, Not a Strategy
A blanket peak-season freeze prices every deploy as equally dangerous, which is only rational when you cannot tell them apart. Harden rollback drills, canary analysis, feature flags, and dependency pinning by October, then freeze only the genuinely irreversible changes.
Returns and Refunds Are an Engineering Problem
A return is a long-running distributed transaction that moves goods one way and money the other. Policy as code, classification as a routing problem, the 5-state refund machine, and an explicit line between what to automate and what stays human.
One Event Backbone Beats Point-to-Point Integration
With 6 systems there are 15 possible pairs, and every new system you add costs another handful of point-to-point links. A backbone changes the slope: the Nth system costs 1 connection. Here are the source-of-truth rules, cadence tiers, retry and dead-letter mechanics that make that hold in production.