The impact of this incident is very unevenly spread.
Over 10 years ago, Mozilla shipped OneCRL [1]. We remediated this incident in a couple of hours for Firefox users who were protected automatically without the need for updates. Chrome, Apple and Microsoft have similar tooling. Since then, we've built more advanced tooling to handle revocation at scale for leaf certs (CRLite [2]).
However, for the rest of the ecosystem, there has been little to no investment in revocation and for those clients, this is a massive problem. Hopefully, we'll see folks invest now. Efforts like upki [3] look promising.
Firefox Mobile 158.0 on Android 17 (Pixel 7) connects to all the demo sites with zero warnings. Any idea why the "automatic protection without updates" doesn't seem to be working with that combination?
I'm hoping some organizations will step forward to fund work on this.
(Note that curl on Ubuntu 26.10 now has CRLite support baked via upki -- unfortunately that doesn't help with this because AIUI CRLite is only for end entities, not for intermediate certificates.)
You mean you and Chrome stopped supporting normal CRLs and OCSP, replacing (for normal users) these distributed-trust technologies with central lists controlled by browsers
You mean the response was uneven. The impact was a complete and total potential compromise of literally everyone. Every domain. Every client using one of the public root programs. Every user of those clients.
I mean, it's an emergency, but the acute severity is debatable; many times a month the major browsers quietly deal with vulnerabilities that yield attackers direct control of victim computers; here, attackers get the ability to spoof websites if they're in a position to hijack connections.
The middlebox situation, with non-updatable roots, is very annoying. But middleboxes also have unfixable RCEs, so...
It's totally fair to evaluate different projects by how well positioned they are to update their WebPKI trust anchors. I'm just saying, keep perspective.
Keep in mind that a _lot_ of software out there uses TLS as its only trust anchor for updates, commands or similar.
This can and will be weaponized for take-overs of systems. They will be of the consumer kind, but they will also be of the industry kind. Let's pray that they will not be of the state or military kind.
So it other words, it allowed attackers to do the one thing that Web PKI is supposed to prevent.
Being able to quickly update trust roots is at best a hack to paper over the issue when there is no good reason why every podunk CA should have authority to generate certificates for every domain in the first place. It's an inexcusable design flaw. Or a feature that gives power to root stores at the expense of everyones security, from the perspective of the root store operators.
Which is to say, sub-CAs in itself are a giant loophole that needs plugging. Especially since AFAIK nobody is bothering with name constraints (RFC5280 section 4.2.1.10).
Screening process has been more involved in that since around the late 2000's. I wouldn't call 20 years "very recently," especially in the context of the history of the web.
The Go and Rustls upgrades specifically block the intermediate certs because neither of these use the system revocation lists, I believe. Once you upgrade to a trust store without the root cert for the CA this will also solve the issue. Not sure about other ecosystems.
Linux - the platform which offers no ability to revoke an intermediate certificate, thus necessitating every certificate library to implement its own blocking?
Windows is already blocking the compromised CAs as of today with no need for users to upgrade anything.
Every certificate library has to implement its own blocking no matter which platform you run on. Windows provides an API to do this at the system level, which means nothing if a given application vendors their own TLS library.
It's uncommon on Windows for applications to use a third party certificate verification library because Windows provides an API that does it. On Linux, you have no choice because all the platform provides is a bundle of PEM certificates.
> An attacker with the private key of any of the compromised intermediate CAs could impersonate any website or service.
And how many more times do we need to read this before we replace Web PKI with a system that isn't as completely insane?
We already have a functioning system for delegating domain ownership in a hierarchical fashion where compromises are limited to only the subtree under the compromised entity. Let's start using it for domain validation sooner rather than later.
It's been obvious for a while now that root-of-trust schemes create juicy targets, and that juicy targets attract all sorts of bad behavior that wouldn't be economical if we used a web-of-trust.
Building a web-of-trust scheme that works might not be easy, but also it's not like we've been trying exceptionally hard lately either.
DANE replaces this system with one that has the same basic cryptographic properties, but in which you can't eject bad actors --- or even monitor them with a mandatory transparency log. It's strictly worse than this.
It has the property of aligning cryptographic authentication with the domain name hierarchy. Yes, you trust the hierarchy. You trust LE too, so what? We need to move away from any authority being able to authenticate anyone, and DNS naturally aligns with the goal.
You've been vocal for a long time against DNSSEC deployment too for no discernable reason.
Why are you against every single thing that tries to make DNS useful for securing anything?
People have debated DANE endlessly here, so I'm not sure how much more I have to add at this point. However, browsers, at least, do swiftly handle revocation for CAs, as indicated in this comment: https://news.ycombinator.com/item?id=50167244. They also handle EE revocation to varying extents, with Firefox having complete coverage and other browsers having partial coverage.
trying to understand where this service is mainly used. judging by a quick google search ...
is this related to Government of India related websites and their services/websites like tax filing etc? is the impact expected to be concentrated around indian citizens availing these services?
It doesn't have to, you can set X509 name constraints which say "the CA may only issue/may not issue certificates for these Subjects," or you can set CAA in DNS, for example.
(Of course, the former isn't really intended to constrain trusted Root CAs, moreso internal PKIs. But it is an option people should know about!)
You can pin, but almost nobody does (and half the world that does pin is probably pinned to Let's Encrypt).
Those who manage the root domains should have been able to pin who they trust, and then if you WANTED to you could add someone beyond what .com or .egg or .muffin lists.
https://kagi.com/search?q=dns+CAA CAA is one way, but you can also pin the actual certificate or certificate chain (which has its downsides, mind you).
Web PKI is cursed. Why should anyone have to ship a new version of Go, Electron, webpki-roots, etc. (possibly over a now-compromised TLS connection!) just to revoke a certificate? Why are so many packages independently shipping high-sensitivity trust-bootstrap credentials? And why can any CA still vouch for any domain instead of each being limited to a specific ccTLD or other non-total shard of subject names?
Also, it's quietly, in the email space, that we're seeing real innovation while browser vendors just say "no" to everything. We should be distributing trust, not centralizing it in browser vendor security teams and Let's Encrypt.
Web PKI doesn't mandate that every client ship with its own root certs. The clients you listed can be configured to use a centrally managed trust store, if that's what you want. The fact that they don't by default is a deliberate tradeoff that the clients made, not some standard's fault.
This isn't, like, one misbehaving client. It's the way the ecosystem has evolved. GP didn't specify whether they were criticizing the standard or the ecosystem more broadly, but even if they were criticizing the standard, it's pretty fair to evaluate a standard at least partly based on how it gets implemented in practice. There's a reason cryptography tools for the last decade or more have been focused on being misuse-resistant.
There is no one ecosystem. The world of web clients is highly diverse. Expecting every client implementation out there to converge onto one particular trust store is, first of all, magical thinking, but second of all, it reintroduces the single point of failure that web PKI is meant to avoid. The OS trust store is just one of many possible trust stores, there's nothing ordained about it. The problem is not with some standard, it's with a mistaken pre-conceived notion that there must only be one.
Many web clients have chosen to default to trust stores within their own ecosystem for usability and reproducibility reasons. When I as a user run "pip install requests", I expect to be able to issue HTTPS requests. I do not expect an implicit system dependency on ca-certificates - that might not even exist on my system!
If you want all of the web clients on your system to use a centrally managed trust store, then it's your responsibility to (a) know and enforce what clients exist on your system and (b) configure them all to use that trust store.
I'm not making an argument for a single centrally-managed trust store. The thing that seems most obviously bad to me here, as a non-crypto-expert, is that any CA can issue a certificate for any domain. This is probably made worse because there are hundreds of CAs, and AFAICT functionally every trust store trusts all of the same ones.
I don't know how we fix this, but it does seem like a big problem.
Yes. I am criticizing the de facto web PKI, not the somewhat better design intent. The design intent is still bad though: we should never have given CAs universal issuance privileges.
The web has continued to go the wrong way in practice. (For example, browser makers rejected RFC9102 DNSSEC stapling, which could have enabled an incremental migration away from CA-centric hell.) Everything fix the existing web PKI community makes just seems to centralize trust more.
For example, Let's Encrypt has been useful for moving people off unencrypted services, but side it's now ubiquitous on the internet and had to work within the broken universal-grant CA model, it now represents a terrifying central point of failure. Had browser makers supported DANE instead, and had we embraced DNSSEC instead of casting aspersions on it, we wouldn't have needed Let's Encrypt because ownership of a domain name would have already, by that very fact, let you authenticate to anyone.
> Had browser makers supported DANE instead, and had we embraced DNSSEC instead of casting aspersions on it, we wouldn't have needed Let's Encrypt because ownership of a domain name would have already, by that very fact, let you authenticate to anyone.
Had we been using DANE instead of WebPKI, the response to this sort of compromise would be "aw, shucks, someone proved to be untrustworthy and they're in such a critical role in the ecosystem that there's no way to route around this DNS provider being untrustworthy."
> The design intent is still bad though: we should never have given CAs universal issuance privileges.
This is sort of right and sort of wrong. It's true that you could have a DNSSEC-style regime where only one entity could issue for a given domain, but as long as you're going to allow third party issuance at all, then you have to contend with the fact that most CAs want to issue in the big gTLDs (.com, .org, etc.) and so narrowing issuance doesn't help that much in many cases.
> Had browser makers supported DANE instead, and had we embraced DNSSEC instead of casting aspersions on it, we wouldn't have needed Let's Encrypt because ownership of a domain name would have already, by that very fact, let you authenticate to anyone.
This is ahistorical. By the time that DANE was designed, the WebPKI model was well entrenched and there was a huge population of browsers that couldn't really be upgraded and also only supported the WebPKI. If you wanted something that was acceptable to those browsers, it had to be WebPKI and it had to be cross-signed, hence Let's Encrypt.
I suppose you could argue that at some point after that one could have tried to deploy DANE and phase out LE and the rest of the WebPKI. I don't think that's true for the reasons that have been discussed on HN about DNSSEC, but that's really not the same as you wouldn't have needed LE in the first place.
The impact of this incident is very unevenly spread.
Over 10 years ago, Mozilla shipped OneCRL [1]. We remediated this incident in a couple of hours for Firefox users who were protected automatically without the need for updates. Chrome, Apple and Microsoft have similar tooling. Since then, we've built more advanced tooling to handle revocation at scale for leaf certs (CRLite [2]).
However, for the rest of the ecosystem, there has been little to no investment in revocation and for those clients, this is a massive problem. Hopefully, we'll see folks invest now. Efforts like upki [3] look promising.
[1] https://blog.mozilla.org/security/2015/03/03/revoking-interm...
[2] https://hacks.mozilla.org/2025/08/crlite-fast-private-and-co...
[3] https://github.com/rustls/upki
Firefox Mobile 158.0 on Android 17 (Pixel 7) connects to all the demo sites with zero warnings. Any idea why the "automatic protection without updates" doesn't seem to be working with that combination?
I'm on the same Firefox version on android and they are blocked now
Pixel 10 with 158.0.1 still connects to the emSign ones..
edit: Disabling all the eMudhra CAs in Android settings does nothing, cool
More on upki:
https://discourse.ubuntu.com/t/an-update-on-upki/77063
My interest in building OneCRL support into upki has definitely increased this week. Issue:
https://github.com/rustls/upki/issues/229
I'm hoping some organizations will step forward to fund work on this.
(Note that curl on Ubuntu 26.10 now has CRLite support baked via upki -- unfortunately that doesn't help with this because AIUI CRLite is only for end entities, not for intermediate certificates.)
You mean you and Chrome stopped supporting normal CRLs and OCSP, replacing (for normal users) these distributed-trust technologies with central lists controlled by browsers
You mean the response was uneven. The impact was a complete and total potential compromise of literally everyone. Every domain. Every client using one of the public root programs. Every user of those clients.
I mean, it's an emergency, but the acute severity is debatable; many times a month the major browsers quietly deal with vulnerabilities that yield attackers direct control of victim computers; here, attackers get the ability to spoof websites if they're in a position to hijack connections.
The middlebox situation, with non-updatable roots, is very annoying. But middleboxes also have unfixable RCEs, so...
It's totally fair to evaluate different projects by how well positioned they are to update their WebPKI trust anchors. I'm just saying, keep perspective.
Keep in mind that a _lot_ of software out there uses TLS as its only trust anchor for updates, commands or similar.
This can and will be weaponized for take-overs of systems. They will be of the consumer kind, but they will also be of the industry kind. Let's pray that they will not be of the state or military kind.
So it other words, it allowed attackers to do the one thing that Web PKI is supposed to prevent.
Being able to quickly update trust roots is at best a hack to paper over the issue when there is no good reason why every podunk CA should have authority to generate certificates for every domain in the first place. It's an inexcusable design flaw. Or a feature that gives power to root stores at the expense of everyones security, from the perspective of the root store operators.
Here's a question for Mozilla et. al.:
It's cute you have a magic DIY CRL. You accepted these assholes into your root program.
They lost the KEYS.
What is the screening process for CA's that can't be bothered to use a HSM?
Even Honest Achmed wasn't this careless.
>What is the screening process for CA's
it's done in the open, and you're even allowed to voice your cute opinion during the multi-week public commentary periods.
here is an overview https://wiki.mozilla.org/CA/Root_Inclusion_Overview
They didn't lose the keys, a sub-CA did.
Which is to say, sub-CAs in itself are a giant loophole that needs plugging. Especially since AFAIK nobody is bothering with name constraints (RFC5280 section 4.2.1.10).
IIRC up until very recently the screening process was "they report that they're good boys."
Screening process has been more involved in that since around the late 2000's. I wouldn't call 20 years "very recently," especially in the context of the history of the web.
From what the internet is telling me, the keys were found on https://repository.emsign.com/certs/<redacted>.crt
Which, if true, is definitely a lot of fun.
Hope you people are ready for lots and lots of suddenly hackable IoT.
Source? That would be an incredible blunder, but that also appears to be the kind of URL one would expect to find certs at.
That's actually a good point - tons of IoT is "secure enough" to sign things, but not secure enough to bother pinning anything ...
Also who hasn't left their private keys hanging around web-accessible, eh? lol
Running Go? Upgrade to 1.27.3 or 1.26.10.
Rust with rustls? Upgrade to rustls-webpki 0.103.16 and/or webpki-roots/webpki-root-certs 1.0.10.
Python? Upgrade certifi to 2026.10.19.
Homebrew ca-certificates? 2026-09-25_1.
Electron? 44.9.1, 43.8.1, or 42.11.16.
The Go and Rustls upgrades specifically block the intermediate certs because neither of these use the system revocation lists, I believe. Once you upgrade to a trust store without the root cert for the CA this will also solve the issue. Not sure about other ecosystems.
Windows? Upgrade to Linux.
Linux - the platform which offers no ability to revoke an intermediate certificate, thus necessitating every certificate library to implement its own blocking?
Windows is already blocking the compromised CAs as of today with no need for users to upgrade anything.
Every certificate library has to implement its own blocking no matter which platform you run on. Windows provides an API to do this at the system level, which means nothing if a given application vendors their own TLS library.
It's uncommon on Windows for applications to use a third party certificate verification library because Windows provides an API that does it. On Linux, you have no choice because all the platform provides is a bundle of PEM certificates.
This probably has an unimaginable blast radius of existing software out there.
are there any consequences to eMudhra?
> What are the consequences?
> An attacker with the private key of any of the compromised intermediate CAs could impersonate any website or service.
And how many more times do we need to read this before we replace Web PKI with a system that isn't as completely insane?
We already have a functioning system for delegating domain ownership in a hierarchical fashion where compromises are limited to only the subtree under the compromised entity. Let's start using it for domain validation sooner rather than later.
>And how many more times do we need to read this before we replace Web PKI with a system that isn't as completely insane?
You say this like it's easy and simple and we are simply not choosing to do so.
It's been obvious for a while now that root-of-trust schemes create juicy targets, and that juicy targets attract all sorts of bad behavior that wouldn't be economical if we used a web-of-trust.
Building a web-of-trust scheme that works might not be easy, but also it's not like we've been trying exceptionally hard lately either.
> We already have a functioning system for delegating domain ownership in a hierarchical fashion
Well, yeah. DANE should have won. NameConstraints would be nice tooo, but who's going to make CAs use it?
And FFS, clients could at least check CRLs once in a while.
DANE replaces this system with one that has the same basic cryptographic properties, but in which you can't eject bad actors --- or even monitor them with a mandatory transparency log. It's strictly worse than this.
It has the property of aligning cryptographic authentication with the domain name hierarchy. Yes, you trust the hierarchy. You trust LE too, so what? We need to move away from any authority being able to authenticate anyone, and DNS naturally aligns with the goal.
You've been vocal for a long time against DNSSEC deployment too for no discernable reason.
Why are you against every single thing that tries to make DNS useful for securing anything?
People have debated DANE endlessly here, so I'm not sure how much more I have to add at this point. However, browsers, at least, do swiftly handle revocation for CAs, as indicated in this comment: https://news.ycombinator.com/item?id=50167244. They also handle EE revocation to varying extents, with Firefox having complete coverage and other browsers having partial coverage.
trying to understand where this service is mainly used. judging by a quick google search ...
is this related to Government of India related websites and their services/websites like tax filing etc? is the impact expected to be concentrated around indian citizens availing these services?
The way the web works, any CA (Certificate Authority) can create TLS certificates for almost any website.
This means that the people with these keys could MITM almost any https connection people make, not only the ones that this CA issued.
It doesn't have to, you can set X509 name constraints which say "the CA may only issue/may not issue certificates for these Subjects," or you can set CAA in DNS, for example.
(Of course, the former isn't really intended to constrain trusted Root CAs, moreso internal PKIs. But it is an option people should know about!)
"any CA (Certificate Authority) can create TLS certificates for almost any website."
This has always been incredibly stupid.
You can pin, but almost nobody does (and half the world that does pin is probably pinned to Let's Encrypt).
Those who manage the root domains should have been able to pin who they trust, and then if you WANTED to you could add someone beyond what .com or .egg or .muffin lists.
Why? How do you pin a website to a specific cert provider? And what happens when you want to switch?
https://kagi.com/search?q=dns+CAA CAA is one way, but you can also pin the actual certificate or certificate chain (which has its downsides, mind you).
Because it makes you only as secure as the the LEAST secure CA. Such as the really bad ones that don't use Hardware Security Modules.
It’s used by Indian companies and also they provide digital signature services. It’s a widely used platform.
Web PKI is cursed. Why should anyone have to ship a new version of Go, Electron, webpki-roots, etc. (possibly over a now-compromised TLS connection!) just to revoke a certificate? Why are so many packages independently shipping high-sensitivity trust-bootstrap credentials? And why can any CA still vouch for any domain instead of each being limited to a specific ccTLD or other non-total shard of subject names?
Also, it's quietly, in the email space, that we're seeing real innovation while browser vendors just say "no" to everything. We should be distributing trust, not centralizing it in browser vendor security teams and Let's Encrypt.
Web PKI doesn't mandate that every client ship with its own root certs. The clients you listed can be configured to use a centrally managed trust store, if that's what you want. The fact that they don't by default is a deliberate tradeoff that the clients made, not some standard's fault.
This isn't, like, one misbehaving client. It's the way the ecosystem has evolved. GP didn't specify whether they were criticizing the standard or the ecosystem more broadly, but even if they were criticizing the standard, it's pretty fair to evaluate a standard at least partly based on how it gets implemented in practice. There's a reason cryptography tools for the last decade or more have been focused on being misuse-resistant.
There is no one ecosystem. The world of web clients is highly diverse. Expecting every client implementation out there to converge onto one particular trust store is, first of all, magical thinking, but second of all, it reintroduces the single point of failure that web PKI is meant to avoid. The OS trust store is just one of many possible trust stores, there's nothing ordained about it. The problem is not with some standard, it's with a mistaken pre-conceived notion that there must only be one.
Many web clients have chosen to default to trust stores within their own ecosystem for usability and reproducibility reasons. When I as a user run "pip install requests", I expect to be able to issue HTTPS requests. I do not expect an implicit system dependency on ca-certificates - that might not even exist on my system!
If you want all of the web clients on your system to use a centrally managed trust store, then it's your responsibility to (a) know and enforce what clients exist on your system and (b) configure them all to use that trust store.
I'm not making an argument for a single centrally-managed trust store. The thing that seems most obviously bad to me here, as a non-crypto-expert, is that any CA can issue a certificate for any domain. This is probably made worse because there are hundreds of CAs, and AFAICT functionally every trust store trusts all of the same ones.
I don't know how we fix this, but it does seem like a big problem.
Yes. I am criticizing the de facto web PKI, not the somewhat better design intent. The design intent is still bad though: we should never have given CAs universal issuance privileges.
The web has continued to go the wrong way in practice. (For example, browser makers rejected RFC9102 DNSSEC stapling, which could have enabled an incremental migration away from CA-centric hell.) Everything fix the existing web PKI community makes just seems to centralize trust more.
For example, Let's Encrypt has been useful for moving people off unencrypted services, but side it's now ubiquitous on the internet and had to work within the broken universal-grant CA model, it now represents a terrifying central point of failure. Had browser makers supported DANE instead, and had we embraced DNSSEC instead of casting aspersions on it, we wouldn't have needed Let's Encrypt because ownership of a domain name would have already, by that very fact, let you authenticate to anyone.
> Had browser makers supported DANE instead, and had we embraced DNSSEC instead of casting aspersions on it, we wouldn't have needed Let's Encrypt because ownership of a domain name would have already, by that very fact, let you authenticate to anyone.
Had we been using DANE instead of WebPKI, the response to this sort of compromise would be "aw, shucks, someone proved to be untrustworthy and they're in such a critical role in the ecosystem that there's no way to route around this DNS provider being untrustworthy."
> The design intent is still bad though: we should never have given CAs universal issuance privileges.
This is sort of right and sort of wrong. It's true that you could have a DNSSEC-style regime where only one entity could issue for a given domain, but as long as you're going to allow third party issuance at all, then you have to contend with the fact that most CAs want to issue in the big gTLDs (.com, .org, etc.) and so narrowing issuance doesn't help that much in many cases.
> Had browser makers supported DANE instead, and had we embraced DNSSEC instead of casting aspersions on it, we wouldn't have needed Let's Encrypt because ownership of a domain name would have already, by that very fact, let you authenticate to anyone.
This is ahistorical. By the time that DANE was designed, the WebPKI model was well entrenched and there was a huge population of browsers that couldn't really be upgraded and also only supported the WebPKI. If you wanted something that was acceptable to those browsers, it had to be WebPKI and it had to be cross-signed, hence Let's Encrypt.
I suppose you could argue that at some point after that one could have tried to deploy DANE and phase out LE and the rest of the WebPKI. I don't think that's true for the reasons that have been discussed on HN about DNSSEC, but that's really not the same as you wouldn't have needed LE in the first place.
Why don't these massive package managers use the host OS CA store, with an optional preconfigured "additional" store if clients need them?
Is the update cadence of CAs that different between the sources? Is the list of CAs included so different from OS vendors and the package managers?
Creating a dependency on OS specific APIs solves this one niche problem but causes a hundred more.
> Also, it's quietly, in the email space, that we're seeing real innovation […]
Can you elaborate on this innovation in the email space around PKI?