The jabber.ru post referenced here presents clear evidence (in the section titled "Network") that the malicious actor was able to reroute traffic going to the legitimate jabber.ru server. An attacker in this position does not need an RCE to get a cert, they can just get one issued the normal way, because they do effectively control the IP address that the domain is pointing to.
One suggestion for anyone concerned about this weakness. You can use the CAA record to pin the domain to a specific certificate authority, issuance method, and account. This is imperfect, as CAA record validation is not mandatory yet. But by March 2027 all the CAs a supposed to have support.
Sprinkle some DNSSEC on the CAA record too, if you'd like.
That's right, it's easier to setup such MiTM using an intermediate server, because only getting the private key of the certificate won't get you the user's traffic due to PFS.
You either need to disable PFS on the server, or export TLS master keys for each session in some way, or MiTM.
>the various ACME clients like acme.sh are run with elevated privileges
Its really not that difficult to not grant excessive privileges - at the very least for recurring ("cron") runs, once filesystem structure, cache invalidation triggers and web server configuration are in place. Its a shame this is still taught in the "just run as admin" style.
> TLS wiretapping with root-CA-signed certificates is a thing that both happens and verifiably has happened. (...) This being a fact rather than a conspiracy theory tends to upset people.
Maybe what people get upset about is catchy misleading [0] summaries like this, which suggest [0] a CA - nation state collusion, despite the actual story going in a completely different [0] direction? The thing that would be actually big news [0]?
[0] in the eye of the beholder of course, as always
I could see this actually being a real parallel reconstruction for a state actor that did issue certificates from a compromised CA. If any evidence points back to them, they can just say the server was hacked with the acme RCE to generate different certs. There probably won't be a way to legally verify that such a thing never happened.
I thought certificate transparency was the thing that was supposed to prevent exactly what this article is describing. What if anything is incorrect about my model of the world in this respect?
Basically, CT did indeed worked as designed, but there was no monitoring by the domain authors (which to be fair there are a dearth of solutions of the time).
On a related note, Let's Encrypt also issued the presumably-interception certificates. This can be possibly something that requires interception at the VPS level (otherwise we already detected the BGP leaks). Presumably, Hetzner was forced to do a raw interception and then redirecting all relevant ports to a middlebox for inspection and CA issuance (and since that the ACME spec is well-defined, they can simply check if the handshake contains the TLS ALPN challenge and then redirect them to special code that will reply with the correct things).
Certificate transparency worked exactly as designed in this case. Monitoring public certificate transparency logs for anomalies is a different story entirely.
By breaking the software facilitating https via ACME itself, no anomalous certificate transparency logs would have needed to have been created at all.
The front door is locked quite tightly with a watchful security camera, but the window has been left unlocked. Also no one is watching the camera feed.
The wrong part is that Let's Encrypt was willing to issue a valid cert to anyone who can temporarily redirect traffic. The authorization should have been done better, for example, sending a certificate to operator's email.
Nothing, although it's more mitigate than prevent per se. They simply did not have alerting set up against the CT logs. It is one of the lessons they highlighted in their own postmortem.
Yeah I suppose the prevent part came from the Browser/CA forum giving the CA that did it the death penalty like they did for Kazakhstan's CA in 2015 but if the men with guns point them at executives of browser providers and say "trust this CA or else" then CT is more of a cosmetic system than anything else.
What I more meant is that it's a reactive arrangement rather than a proactive one, so it cannot be preventative outright. Domain owners are expected to actively monitor the CT logs for abuse, and take action if they see any. This necessarily means that abuse can still happen, at least for a little while.
The most striking thing about these types of conspiracy theories is people seem to completely forget that whoever you imagine you can threaten generally doesn't have the ability to do the thing you want them to do: they'd have to delegate it.
Because unlike in that case, the CA in this story is not suspected to have done anything wrong, despite what the post's wording might suggest. See my other comment: https://news.ycombinator.com/item?id=48340259
CT indeed worked out pretty well. At least until bots started hammering crt.sh making it unreliable, and those that want to be alerted to newly issued certificated appeared in the logs need to pay for some purpose-built service instead of just adding a relevant query to their feed reader.
The jabber.ru post referenced here presents clear evidence (in the section titled "Network") that the malicious actor was able to reroute traffic going to the legitimate jabber.ru server. An attacker in this position does not need an RCE to get a cert, they can just get one issued the normal way, because they do effectively control the IP address that the domain is pointing to.
One suggestion for anyone concerned about this weakness. You can use the CAA record to pin the domain to a specific certificate authority, issuance method, and account. This is imperfect, as CAA record validation is not mandatory yet. But by March 2027 all the CAs a supposed to have support.
Sprinkle some DNSSEC on the CAA record too, if you'd like.
> This is imperfect, as CAA record validation is not mandatory yet. But by March 2027 all the CAs a supposed to have support.
Is that true? My read of Section 1.2.1 in [1] suggests CAA checking has been mandatory since 2017‐09‐08.
[1] https://cabforum.org/working-groups/server/baseline-requirem...
That's right, it's easier to setup such MiTM using an intermediate server, because only getting the private key of the certificate won't get you the user's traffic due to PFS.
You either need to disable PFS on the server, or export TLS master keys for each session in some way, or MiTM.
This is a reminder why you should use E2EE messengers only.
>the various ACME clients like acme.sh are run with elevated privileges
Its really not that difficult to not grant excessive privileges - at the very least for recurring ("cron") runs, once filesystem structure, cache invalidation triggers and web server configuration are in place. Its a shame this is still taught in the "just run as admin" style.
> TLS wiretapping with root-CA-signed certificates is a thing that both happens and verifiably has happened. (...) This being a fact rather than a conspiracy theory tends to upset people.
Maybe what people get upset about is catchy misleading [0] summaries like this, which suggest [0] a CA - nation state collusion, despite the actual story going in a completely different [0] direction? The thing that would be actually big news [0]?
[0] in the eye of the beholder of course, as always
I could see this actually being a real parallel reconstruction for a state actor that did issue certificates from a compromised CA. If any evidence points back to them, they can just say the server was hacked with the acme RCE to generate different certs. There probably won't be a way to legally verify that such a thing never happened.
I thought certificate transparency was the thing that was supposed to prevent exactly what this article is describing. What if anything is incorrect about my model of the world in this respect?
Basically, CT did indeed worked as designed, but there was no monitoring by the domain authors (which to be fair there are a dearth of solutions of the time).
On a related note, Let's Encrypt also issued the presumably-interception certificates. This can be possibly something that requires interception at the VPS level (otherwise we already detected the BGP leaks). Presumably, Hetzner was forced to do a raw interception and then redirecting all relevant ports to a middlebox for inspection and CA issuance (and since that the ACME spec is well-defined, they can simply check if the handshake contains the TLS ALPN challenge and then redirect them to special code that will reply with the correct things).
Certificate transparency worked exactly as designed in this case. Monitoring public certificate transparency logs for anomalies is a different story entirely.
By breaking the software facilitating https via ACME itself, no anomalous certificate transparency logs would have needed to have been created at all.
The front door is locked quite tightly with a watchful security camera, but the window has been left unlocked. Also no one is watching the camera feed.
The wrong part is that Let's Encrypt was willing to issue a valid cert to anyone who can temporarily redirect traffic. The authorization should have been done better, for example, sending a certificate to operator's email.
Nothing, although it's more mitigate than prevent per se. They simply did not have alerting set up against the CT logs. It is one of the lessons they highlighted in their own postmortem.
Yeah I suppose the prevent part came from the Browser/CA forum giving the CA that did it the death penalty like they did for Kazakhstan's CA in 2015 but if the men with guns point them at executives of browser providers and say "trust this CA or else" then CT is more of a cosmetic system than anything else.
What I more meant is that it's a reactive arrangement rather than a proactive one, so it cannot be preventative outright. Domain owners are expected to actively monitor the CT logs for abuse, and take action if they see any. This necessarily means that abuse can still happen, at least for a little while.
Do the executives implement program features?
The most striking thing about these types of conspiracy theories is people seem to completely forget that whoever you imagine you can threaten generally doesn't have the ability to do the thing you want them to do: they'd have to delegate it.
So given that this is widely accepted to have actually happened why has the CA involved not received the death penalty like the Kazakh one did?
Because unlike in that case, the CA in this story is not suspected to have done anything wrong, despite what the post's wording might suggest. See my other comment: https://news.ycombinator.com/item?id=48340259
CT indeed worked out pretty well. At least until bots started hammering crt.sh making it unreliable, and those that want to be alerted to newly issued certificated appeared in the logs need to pay for some purpose-built service instead of just adding a relevant query to their feed reader.
What LI vendors can break https?
The sloppy ones who want a huge headache and leave a publicly auditable trail a mile long that get analysis blogs written about their mistakes.