Every couple of months someone re-discovers SSH certificates, and blogs about them.
I'm guilty of it too. My blog post from 15 years ago is nowhere near as good as OP's post, but if I though me of 15 years ago lived up to my standards of today, I'd be really disappointed: https://blog.habets.se/2011/07/OpenSSH-certificates.html
I think the scary reality is most people conflate "keys" and "certificates". I have worked with security engineers that I need to remind that we do not use SSH certs, but rather key auth, and they have to think it through to make it click.
Another useful feature of SSH certificates is that you can sign a user’s public key to grant them access to a remote machine for a limited time and as a specific remote user.
I've known SSH certs for a while but never went through the effort of migrating away from keys. I'm very frustrated about manually managing my SSH keys across my different servers and devices though.
I assume you gathered a lot of thoughts over these 15 years.
A big problem I have with ssh carts is that they are not universally supported. For me, there is always some device or daemon (for example tinyssh in the initramfs of my gaming pc so that I can unlock it remotely) that only works with “plain old ssh keys”. And if I have to distribute and sync my keys onto a few hosts anyway, it takes away the benefits.
If your use case is such that you are frustrated about managing keys, host or user keys, then yes it does sound like SSH certs would help you. E.g. when you have many users, servers, or high enough cartesian product of the two.
In environment where they don't cause frustration they're not worth it.
Not really more to it than that, from my point of view.
Yes. Caveat: It might not really be worth it if all your infrastructure is managed by these newfangled infrastructure-as-code-things that are quick to roll out (OpenShift/OKD, Talos, etc.) and you have only one repo to change SSH keys (single cluster or single repo for all clusters).
There are some serious security benefits for larger organizations but it does not sound as if you are part of one.
Exactly. We'd had discussions about building https://Userify.com (plug!) around SSH certificates, but elected to go with keys instead, because Userify delivers most of the good things around certificates without the jank and insecurity.
It's not that certificates themselves are insecure themselves, it's that the workflows (as the parent points out) are awful. We might still add some automation around that (and I think I saw some competitor tooling out there if you're committed to that path) but I personally feel like it's an answer to the wrong question.
I've had very good experiences with SSH Communication Security company's (the guys who invented SSH) PrivX product to manage secure remote access, including SSH certificates and also cert based Windows authentication. It supports other kinds of remote targets too, via webui or with native clients. Great product.
The author lists all the advantes of CA certificates, yet doesn't list the disadvantages. OTOH, all the many steps required to set it up make the disadvantages rather obvious.
Also, I've never had a security issue due to TOFU, have you?
Choosing to use TOFU is a distinct choice from the choice of using the keys generated by SSH, instead of using certificates.
If you do not want to use TOFU, for extra security, you just have to pair the computers by copying between them the corresponding public keys through a secure channel, e.g. by using a USB memory.
Using certificates does not add any simplification or any extra security.
For real security, you still must pair the communicating computers by copying between them the corresponding certificates, through a secure channel, e.g. a USB memory.
When you use for HTTPS the certificates that have come with your Internet browser, you trust that the installer package for the browser has come to that computer through a secure channel from the authority that has created the certificates. This is usually an assumption much more far fetched than the assumption that you can trust TOFU between computers under your control.
Certificates may be useful in big organizations, if other functionality is needed beyond just establishing secure communication channels, e.g. if you want to use certificate revocation.
In the list of "advantages" enumerated in the parent article, more than half of them are false, because if certificates are implemented correctly, completely equivalent actions must be executed when SSH keys without TOFU are used and when certificates are used.
Perhaps the author meant by writing some of the "advantages" that the actions that supposedly are no longer needed with certificates are done by an administrator, not by the user. However that is also applicable with SSH. An administrator could install the certificates, so that no action is required from the user, but an administrator can also install the SSH public keys, so that no TOFU is ever needed from the user.
Using certificates requires exactly the same steps like using keys generated by SSH (i.e. generating certificates and copying them between computers through secure channels, to pair the servers and the authorized users), but it may need additional steps, caused by the fact that certificates provide additional functionality.
Touche.. actually a good point, but actually those are two different situations.
With one, I'm accessing a website and trusting that the certificate is signed by someone I trust; so the trust in my browser certificates (which include certificates from hundreds of certificate authorities all over the world, any one of which could be compromised, robbed, or controlled by an adversarial person or even government) is extended to the site that I'm visiting. To say this is weak sauce rather understates how bad this actually is. (To paraphrase Churchill, this is the worst possible design, except for all the rest.)
With the other, I'm logging into a server for the first time (and I could simply deploy the same trusted host key to all my ssh servers via an autoscaling configuration or whatever). I think it's debatable if TOFU is worse or better than your (granted clever) metaphor.
(to those who'd recommend userify, yes - great for the client login issue and definitely increases security, but to parent's point, TOFU is still needed unless you want to distribute host pubkeys)
Certificates provide extra features, like revocation.
However, if you do not need the extra features provided by certificates, using SSH-generated keys is strictly equivalent with using certificates and it requires less work.
TOFU is neither necessary nor recommended, it is just a convenience feature, to be used when security may be lax.
The secure way to use SSH is to never use TOFU but to pair the user and the server by copying the public keys between the 2 computers through a secure channel, e.g. either by using a USB memory or by sending the public keys through already existing authenticated encrypted links that pass through other computers. (Such a link may be a HTTPS download link.)
When using certificates, a completely identical procedure must be used. After certificates are generated, like also after SSH keys are generated, the certificates must be copied to the client computer and the server computer through secure channels.
Just to make it clear: this does not mean that it is fine to blindly accept the message on first use.
The "secure way" implies copying the server's public key as well, which people generally don't do, right? Which is equivalent to verifying the fingerprint shown with the TOFU message, correct?
Anyone tried out Userify? It creates/removes ssh pubkeys locally so (like a CA) no authn server needs to be online. But unlike certs, active sessions and processes are terminated when the user access is revoked.
We're in the process of updating the experience to this century! ;)
We've always taken the stance that crusty is better than vulnerable, but it turns out that not having a modern experience after 15 years is starting to feel like maybe we need to step up the features and shininess :)
In our dev/stg environment we reinstall half our machines every morning (largely to test our machine setup automation), and SSH host certificates make that so much nicer than having to persist host keys or remove/replace them in known_hosts. Highly recommended.
With the recent wave of npm hacks stealing private keys, I wanted to limit key's lifetimes.
I've set up a couple of yubikeys as SSH CAs on hosts I manage. I use them to create short lived certs (say 24h) at the start of the day. This way i only have to enter the yubikey pin once a day.
I could not find an easy way to limit maximum certificate lifetime in openssh, except for using the AuthorizedPrincipalCommand, which feels very fragile.
Does anyone else have any experience with a similar setup? How do you limit cert max lifetime?
If you generate short lived certificates via an automated process/service then you don’t really need to manage a revocation list as they will have expired in short order.
Hmm. For user certs you can have the service sign them for, say an hour, so long as you can ssh to your server in that time then there’s no need for any other interaction.
Sure you need your signing service to be reasonably available, but that’s easily accomplished.
On top of that you would need something to secure DNS. Like DNSSEC or at the very least use DNS with TLS or DNS over HTTP. None of these are typically enabled by default.
Anything that uses system-resolved is probably doing DNSSEC validation by default. It's becoming much more common.
Additionally, as I mentioned, openssh itself has support for validating the DNSSEC signature even if your local resolver doesn't. I actually don't think it can use the standard resolver for SSHFP records at all, but I'm not sure.
Every couple of months someone re-discovers SSH certificates, and blogs about them.
I'm guilty of it too. My blog post from 15 years ago is nowhere near as good as OP's post, but if I though me of 15 years ago lived up to my standards of today, I'd be really disappointed: https://blog.habets.se/2011/07/OpenSSH-certificates.html
I think the scary reality is most people conflate "keys" and "certificates". I have worked with security engineers that I need to remind that we do not use SSH certs, but rather key auth, and they have to think it through to make it click.
Another useful feature of SSH certificates is that you can sign a user’s public key to grant them access to a remote machine for a limited time and as a specific remote user.
I've known SSH certs for a while but never went through the effort of migrating away from keys. I'm very frustrated about manually managing my SSH keys across my different servers and devices though.
I assume you gathered a lot of thoughts over these 15 years.
Should I invest in making the switch?
A big problem I have with ssh carts is that they are not universally supported. For me, there is always some device or daemon (for example tinyssh in the initramfs of my gaming pc so that I can unlock it remotely) that only works with “plain old ssh keys”. And if I have to distribute and sync my keys onto a few hosts anyway, it takes away the benefits.
If your use case is such that you are frustrated about managing keys, host or user keys, then yes it does sound like SSH certs would help you. E.g. when you have many users, servers, or high enough cartesian product of the two.
In environment where they don't cause frustration they're not worth it.
Not really more to it than that, from my point of view.
Yes. Caveat: It might not really be worth it if all your infrastructure is managed by these newfangled infrastructure-as-code-things that are quick to roll out (OpenShift/OKD, Talos, etc.) and you have only one repo to change SSH keys (single cluster or single repo for all clusters).
There are some serious security benefits for larger organizations but it does not sound as if you are part of one.
You will have to manage your SSH CA certificates instead of your keys.
The workflows SSH CA's are extremely janky and insecure.
With some creative use of `AuthorizedKeysCommand` you can make SSH key rotation painless and secure.
With SSH certificates you have to go back to the "keys to the kingdom" antipattern and just hope for the best.
Exactly. We'd had discussions about building https://Userify.com (plug!) around SSH certificates, but elected to go with keys instead, because Userify delivers most of the good things around certificates without the jank and insecurity.
It's not that certificates themselves are insecure themselves, it's that the workflows (as the parent points out) are awful. We might still add some automation around that (and I think I saw some competitor tooling out there if you're committed to that path) but I personally feel like it's an answer to the wrong question.
I've had very good experiences with SSH Communication Security company's (the guys who invented SSH) PrivX product to manage secure remote access, including SSH certificates and also cert based Windows authentication. It supports other kinds of remote targets too, via webui or with native clients. Great product.
The author lists all the advantes of CA certificates, yet doesn't list the disadvantages. OTOH, all the many steps required to set it up make the disadvantages rather obvious.
Also, I've never had a security issue due to TOFU, have you?
TOFU is convenient, but not necessary.
Choosing to use TOFU is a distinct choice from the choice of using the keys generated by SSH, instead of using certificates.
If you do not want to use TOFU, for extra security, you just have to pair the computers by copying between them the corresponding public keys through a secure channel, e.g. by using a USB memory.
Using certificates does not add any simplification or any extra security.
For real security, you still must pair the communicating computers by copying between them the corresponding certificates, through a secure channel, e.g. a USB memory.
When you use for HTTPS the certificates that have come with your Internet browser, you trust that the installer package for the browser has come to that computer through a secure channel from the authority that has created the certificates. This is usually an assumption much more far fetched than the assumption that you can trust TOFU between computers under your control.
Certificates may be useful in big organizations, if other functionality is needed beyond just establishing secure communication channels, e.g. if you want to use certificate revocation.
In the list of "advantages" enumerated in the parent article, more than half of them are false, because if certificates are implemented correctly, completely equivalent actions must be executed when SSH keys without TOFU are used and when certificates are used.
Perhaps the author meant by writing some of the "advantages" that the actions that supposedly are no longer needed with certificates are done by an administrator, not by the user. However that is also applicable with SSH. An administrator could install the certificates, so that no action is required from the user, but an administrator can also install the SSH public keys, so that no TOFU is ever needed from the user.
Using certificates requires exactly the same steps like using keys generated by SSH (i.e. generating certificates and copying them between computers through secure channels, to pair the servers and the authorized users), but it may need additional steps, caused by the fact that certificates provide additional functionality.
Are you pairing computers by copying certificates to visit this site?
Touche.. actually a good point, but actually those are two different situations. With one, I'm accessing a website and trusting that the certificate is signed by someone I trust; so the trust in my browser certificates (which include certificates from hundreds of certificate authorities all over the world, any one of which could be compromised, robbed, or controlled by an adversarial person or even government) is extended to the site that I'm visiting. To say this is weak sauce rather understates how bad this actually is. (To paraphrase Churchill, this is the worst possible design, except for all the rest.)
With the other, I'm logging into a server for the first time (and I could simply deploy the same trusted host key to all my ssh servers via an autoscaling configuration or whatever). I think it's debatable if TOFU is worse or better than your (granted clever) metaphor.
(to those who'd recommend userify, yes - great for the client login issue and definitely increases security, but to parent's point, TOFU is still needed unless you want to distribute host pubkeys)
> Also, I've never had a security issue due to TOFU, have you?
This is a bit like suggesting you've never been in a car crash, so seat belts must not be worth considering.
Do you feel that beyond the obvious and documented work in setting them up, there are disadvantages to using SSH certificates?
Certificates provide extra features, like revocation.
However, if you do not need the extra features provided by certificates, using SSH-generated keys is strictly equivalent with using certificates and it requires less work.
TOFU is neither necessary nor recommended, it is just a convenience feature, to be used when security may be lax.
The secure way to use SSH is to never use TOFU but to pair the user and the server by copying the public keys between the 2 computers through a secure channel, e.g. either by using a USB memory or by sending the public keys through already existing authenticated encrypted links that pass through other computers. (Such a link may be a HTTPS download link.)
When using certificates, a completely identical procedure must be used. After certificates are generated, like also after SSH keys are generated, the certificates must be copied to the client computer and the server computer through secure channels.
> TOFU is neither necessary nor recommended
Just to make it clear: this does not mean that it is fine to blindly accept the message on first use.
The "secure way" implies copying the server's public key as well, which people generally don't do, right? Which is equivalent to verifying the fingerprint shown with the TOFU message, correct?
Your ISP or telecom has to be compromised for TOFU to be relevant to anything. In practice that never happens.
Anyone tried out Userify? It creates/removes ssh pubkeys locally so (like a CA) no authn server needs to be online. But unlike certs, active sessions and processes are terminated when the user access is revoked.
We're in the process of updating the experience to this century! ;)
We've always taken the stance that crusty is better than vulnerable, but it turns out that not having a modern experience after 15 years is starting to feel like maybe we need to step up the features and shininess :)
In our dev/stg environment we reinstall half our machines every morning (largely to test our machine setup automation), and SSH host certificates make that so much nicer than having to persist host keys or remove/replace them in known_hosts. Highly recommended.
With the recent wave of npm hacks stealing private keys, I wanted to limit key's lifetimes.
I've set up a couple of yubikeys as SSH CAs on hosts I manage. I use them to create short lived certs (say 24h) at the start of the day. This way i only have to enter the yubikey pin once a day.
I could not find an easy way to limit maximum certificate lifetime in openssh, except for using the AuthorizedPrincipalCommand, which feels very fragile.
Does anyone else have any experience with a similar setup? How do you limit cert max lifetime?
All those articles about SSH certificates fall short of explaining how the revocation list can/should be published.
Is that yet another problem that I need to solve with syncthing?
https://man.openbsd.org/ssh-keygen.1#KEY_REVOCATION_LISTS
If you generate short lived certificates via an automated process/service then you don’t really need to manage a revocation list as they will have expired in short order.
But then you can't log in if your box goes offline for any reason.
Hmm. For user certs you can have the service sign them for, say an hour, so long as you can ssh to your server in that time then there’s no need for any other interaction.
Sure you need your signing service to be reasonably available, but that’s easily accomplished.
Maybe I misunderstand?
You can also address TOFU to some extent using SSHFP DNS records.
Openssh supports checking the DNSSEC signature in the client, in theory, but it's a configure option and I'm not sure if distros build with it.
On top of that you would need something to secure DNS. Like DNSSEC or at the very least use DNS with TLS or DNS over HTTP. None of these are typically enabled by default.
Anything that uses system-resolved is probably doing DNSSEC validation by default. It's becoming much more common.
Additionally, as I mentioned, openssh itself has support for validating the DNSSEC signature even if your local resolver doesn't. I actually don't think it can use the standard resolver for SSHFP records at all, but I'm not sure.
Sadly services such as Github don't support these so it's mostly good for internal infrastructure.
They do, for Enterprise customers only: https://docs.github.com/en/enterprise-cloud@latest/organizat...
They've rolled their host key one time, so there's little reason for them to use it on the host side.