On November 18, 2025, Cloudflare’s own engineers spent part of the morning convinced they were under attack.
The symptoms fit. Traffic to a huge swath of the internet was throwing 5xx errors, and the failures came and went in waves — bad, then fine, then bad again — which is exactly what a large, adaptive denial-of-service attack looks like. To make it worse, Cloudflare’s public status page, which is hosted off their own infrastructure specifically so it survives their outages, happened to go down around the same time. If you were in that incident channel, every signal pointed at an adversary.
There was no adversary. Cloudflare was attacking itself, and it took hours to be sure of that, because the failure was oscillating for a reason nobody had designed and nobody could see from the outside.
I want to walk through three Cloudflare outages, because Cloudflare does something almost no company at its scale does: it publishes long, technical, genuinely honest postmortems, and then it lets you read them side by side. And when you read them side by side, the striking thing isn’t that a company this good has outages. It’s that the outages keep telling the same story.
2019: the regex
Start on July 2, 2019. Cloudflare pushed a new rule to its WAF — the managed rules that inspect traffic for attacks. One of the rules contained a regular expression, and that regex had catastrophic backtracking: on certain input it didn’t match-or-fail quickly, it wandered through an exponential number of possibilities, pinning a CPU core while it thought.
Cloudflare runs that WAF on every machine that serves HTTP traffic. So the rule went out, and CPU usage across the entire global network climbed to essentially 100% on every core doing HTTP and HTTPS. For 27 minutes, a big chunk of the internet behind Cloudflare was simply gone — not slow, gone — because every server was busy backtracking through a regex instead of answering requests.
The postmortem has a detail that matters more than the regex itself. A protection had existed that would have capped CPU spent on any single regex evaluation. It had been removed during an earlier refactor of the WAF. The guard that was supposed to catch exactly this class of mistake wasn’t there anymore, and the rule shipped to the whole planet at once with no staged rollout to catch it on the way.
June 2025: the dependency
Fast-forward to June 12, 2025. This one looks completely different on the surface. No bad regex, no CPU exhaustion. Instead, Workers KV — Cloudflare’s key-value store — lost access to its backing storage for two hours and twenty-eight minutes.
The reason KV mattered so much is that a startling number of other Cloudflare products lean on it. Access, Gateway, WARP, Images, Stream, Turnstile — they all read configuration and state out of Workers KV, and part of KV’s backing store lived on a third-party cloud. That cloud had an IAM problem: an identity-service rollout went sideways, tokens stopped being issued, and reads and writes to the store timed out. KV went down, and everything standing on KV went down with it.
Different failure, same shape. A single component that a huge blast radius quietly depended on, taken out by a change nobody expected to be load-bearing.
November 2025: the feature file
Which brings us back to the day everyone thought they were being attacked.
Bot Management — Cloudflare’s system for scoring whether a request is a human or a bot — runs on a “feature file,” a configuration blob generated by a query against a ClickHouse database and shipped out to the edge every few minutes. On November 18, someone made a permissions change to that database. The change was reasonable in isolation. Its side effect was that the query feeding the feature file started returning duplicate rows, and the file roughly doubled in size.
The new proxy engine had a hard limit: it would accept up to 200 features. In normal operation the file carried around 60, so nobody thought about the ceiling. The doubled file blew past 200. And the Rust code that hit the limit did the thing that turns a bad config into a global outage: on the error path, it called unwrap(). In Rust, unwrap() on an error panics the thread. The process serving HTTP traffic fell over, and that became 5xx errors for users.
Now the oscillation makes sense. That feature file was regenerated every few minutes, and only some of the database nodes had the permissions change at first. So the pipeline alternated between generating a good file and a bad one. Good file: things recover. Bad file: everything panics. Recover, panic, recover, panic — a heartbeat of failure that looked, from the inside, exactly like an attacker probing for weaknesses.
The same outage, three times
Line them up.
2019: a change (a WAF rule) shipped everywhere at once, hit a latent problem (a removed CPU guard), and there was no blast-radius containment (global push, no staging).
June 2025: a change (a dependency’s IAM rollout) hit a latent problem (half of Cloudflare silently standing on one key-value store), and there was no blast-radius containment (KV failed for everyone at once).
November 2025: a change (a database permission edit) hit a latent problem (a 200-feature limit and an unwrap() that fails closed), and there was no blast-radius containment (feature file pushed globally every few minutes).
Three outages, three completely different proximate causes — a regex, a cloud provider, a database permission — and underneath, one structure every single time. A change propagates everywhere fast. It meets a latent fragility that was invisible until the exact input that triggers it showed up. And nothing in the middle is built to make the failure small: no canary, no cap on how much of the fleet a single bad artifact can take down, no error path that degrades instead of dying.
That last part is the quiet villain in all three. The 2019 guard was removed. The 2025 unwrap() chose panic over a fallback. When your error handling fails closed — when “something is wrong” means “stop serving” instead of “serve the last known-good version” — every latent bug becomes an outage instead of a log line.
Why this is hard to fix, and not really about Cloudflare
The reflexive response is “add a staged rollout and stop pushing globally.” Cloudflare knows this; they’ve said as much, and after November they announced an internal effort literally named around failing small. But there’s a reason the pattern is sticky, and it’s the same reason CDNs concentrate risk in the first place: the whole value proposition of a global edge is that a change reaches everywhere in seconds. Configuration that propagates instantly is a feature. Bot scores that update in near-real-time across the planet are a feature. The speed and the uniformity are what you’re paying for.
Staged rollout is friction deliberately reintroduced into a system whose selling point is having removed it. Every canary you add is a few minutes of “this machine has the new rule and that one doesn’t,” which is its own source of bugs and its own operational tax. The instinct that made the platform fast is the same instinct that makes a bad artifact global before anyone can react.
So I don’t read these postmortems as “Cloudflare keeps messing up.” I read them as the recurring bill for a design choice the entire industry made — centralize, propagate fast, trust the artifact — and mostly hasn’t paid down. The honest lesson isn’t a checklist. It’s that if your platform’s superpower is doing one thing to the whole world at once, then your defining risk, forever, is doing the wrong thing to the whole world at once. Every postmortem is that same sentence, rewritten with a new proximate cause.
The next one is already being written. We just don’t know yet which change will look reasonable in isolation.