On October 11, 2018, ICANN changed a single cryptographic key and, for a few tense hours, waited to find out whether it had just broken name resolution for some fraction of the internet.
The key was the root zone’s Key Signing Key — the KSK. If you’ve ever drawn the DNSSEC chain of trust, it’s the thing at the very top, the anchor everything else hangs from. Your resolver trusts the signature on example.com because it chains up to .com, which chains up to the root, which is signed by a key your resolver already trusts before it asks a single question. That trusted-in-advance key is the trust anchor, and for the root there had only ever been one. It was generated in 2010, when the root zone was first signed, and in the eight years since it had never changed. Call it KSK-2010. Every validating resolver on earth had it baked in.
The uncomfortable thing about a key that never changes is that you can’t tell the difference between “we never needed to change it” and “we couldn’t.” And a key you can’t replace isn’t much of a security control. If it ever leaked, or the algorithm weakened, or you simply wanted to follow the basic hygiene of rotating your most important secret, you’d need a way to swap it out — across every resolver, everywhere, without a flag day where someone flips a switch and half the internet’s DNSSEC validation goes dark. So the real question in 2018 wasn’t is the new key good. It was can we change this key at all.
The mechanism that was supposed to make it boring
DNSSEC’s designers saw this coming and built an answer years earlier: RFC 5011, “Automated Updates of DNSSEC Trust Anchors.” The idea is elegant. When you want to introduce a new trust anchor, you publish it in the zone and sign it with the old key. A resolver that already trusts the old key sees the new one vouched for by something it believes, waits out a hold-down period — 30 days, to make sure the new key isn’t a brief injection by an attacker — and then quietly adds it to its set of trusted anchors. No human logs in. No config gets edited. The key rolls itself.
That’s the theory. The 2018 rollover was the first time anyone had run it for real at the root of the entire DNS. RFC 5011 had existed for a decade and had, at the scale that mattered, never actually been tested in production. The new key, KSK-2017, was generated in a ceremony, published, and signed by KSK-2010. Then the waiting began, because the honest summary of RFC 5011 is that it’s an act of faith: you publish the new key and trust that resolvers out there are following the protocol and updating themselves. You can’t SSH into a hundred thousand recursive resolvers you don’t operate to check.
The problem was never the math
The rollover was first scheduled for October 11, 2017. A few weeks before the date, ICANN postponed it — for a full year. Not because of anything wrong with the cryptography. Because of a number that had only just become visible.
Around the same time, resolvers had started supporting RFC 8145, a mechanism with the dry name “Signaling Trust Anchor Knowledge in DNS Security Extensions.” It let a validating resolver report back, in the course of normal queries, which trust anchors it was holding. For the first time, the root operators could get a rough census of what keys were actually configured in the field. And the census said something alarming: of roughly 12,000 validating resolvers reporting in September 2017, about 4% were configured with only KSK-2010 — the old key, and not the new one. If the rollover went ahead, those resolvers would keep validating against a key that was about to stop signing the root, and every DNSSEC-signed lookup they made would fail. For the users behind them, large chunks of the internet would simply stop resolving.
Four percent doesn’t sound catastrophic until you remember these are resolvers, not users — a single misconfigured ISP resolver can sit in front of millions of people. And the worst part was that nobody could fully explain the number. Were those resolvers genuinely not ready, and RFC 5011 was failing on them? Or was the telemetry itself misleading — resolvers that would have been fine, reporting in a way that looked scary? The data was new, self-selected, and ambiguous, and you do not roll the key the entire internet’s DNS trust hangs from on top of a number you don’t understand. So they stopped, spent a year studying it, and set a new flag day: October 11, 2018.
It worked, quietly
The second time, it went through. On October 11, 2018, resolvers began validating with KSK-2017, and the feared wave of outages didn’t materialize — a small number of users behind misconfigured resolvers hit failures, but the internet at large didn’t notice, which for infrastructure work is the highest possible praise. A few months later, on January 11, 2019, ICANN set the revoke bit on KSK-2010 — the RFC 5011 signal that tells resolvers “this key is dead, drop it” — and on March 22, 2019, the old key was removed from the root zone entirely. The thing that had never changed in eight years was gone, and the mechanism for changing it had been proven to work.
The lesson people took away from 2018 wasn’t about RSA key sizes or hold-down timers. It was that the hard part of rotating a global key isn’t the rotation — it’s visibility. The cryptography was solved in the 2000s. What nearly derailed the rollover was the plain operational question of knowing what state a hundred thousand machines you don’t control are in, and the fact that the only tool for answering it was itself brand new and hard to read. We had built a beautiful automated protocol for changing the key and then discovered we were flying it half-blind.
If that sounds like an old story, it isn’t. The second-ever root KSK rollover is underway right now: the new key, KSK-2024, was generated in 2024, has sat in the root zone since early 2025, and is scheduled to take over signing on October 11, 2026, with the retirement of KSK-2017 to follow — a deliberately slower, more heavily instrumented process than the first, precisely because 2018 taught everyone that the surprises live in the resolvers you can’t see, not in the key you can. DNSSEC’s adoption is still patchy and its validation patchier, which is its own long-running embarrassment. But the root key, at least, is no longer a thing we’re afraid to touch. It took until 2018 to find that out.