What an aSSHole!
Wait that means I have to expose a port. But I have zero exposed ports.
This post was created by the pangolin-gang.
practically need to rotate logs every hour if you foolishly leave ssh open on a open network. usually a bastion with ssh listening to port 34000 or such is a good first step.
that said the conditions for this exploit are fascinating. no pam, shadow password set to *, using a alternative to glibc. The sacrifices to be more efficient feel on par with replacing a tank’s armor with paper mache and hope.
The best armor is to not get hit!
tldr: disable password auth in ssh.
Also name misses the point I think, it’s a musl-hole.
Yeah, any application checking passwords would be vulnerable.
Further, OpenSSH using musl (pronounced muscle) must not use PAM (pluggable authentication modules),
which Alpine doesn’t ship by default. Maybe one or the other remote administration suite depends on it, but those are often suspectible to exploits anyway. But you don’t expose either to WAN, right?
The only time I ever exposed my SSH port to the WAN, I got about 750 connection attempts from China and 250 attemps from Russia per hour.
Yet some people will still say with a straight face “tHeRe iS nO hYbRiD wAr!1!1”
Not in Ba Sing Se at least…
New friends from all over the world!
yeah, while reading this i was going “What maniac exposes SSH to WAN anyway…?” the whole time. I fail to see any reason or point that can be justified exposing SSH to WAN.
I used to have a synology NAS that would sleep of death itself. Synology support asked me to port forward SSH to the open internet and set no password so they could pull logs over the next few weeks.
Yes, it was legit synology support not a scammer.
No, I haven’t bought synology ever again.
What do you use now? Qnap?
I build my own stuff now. I tried qnap after synology but it just sucked in different ways.
Not the person you responded to, but I’ll give my unsolicited advice: I switched to Unifi NAS. QNAP has had a lot of security nightmares, and a bad habit of EOLing devices.
The best option might actually be using an old workstation and going with something like TrueNAS, but I enjoy the convenience of a purpose built box.
One big difference between Unifi NAS and something like QNAP or Synology, is that the Unifi NAS only does NAS work, it doesn’t run containers or “apps”. You need separate compute if you want to run containers.
One thing I didn’t know before buying a UniFi Nas was it does not support raid0 or jbod unfortunately
The sounds like a big one. Doesn’t affect me, but you’d think a lot of people would want one of those.
Unifi actually sounds better. I’m the same and want my NAS to be optimized by someone smarter than me. I’ve been using qnap and don’t really care for half the crap they offer. They’re not as enshittified as Synology, but they’re getting there.
Any VPS provider just gives you an SSH IP and port. So that’s the default state, until you do something else, which I imagine many don’t, since key based authentication was always considered secure.
So I don’t know the reason, but that’s the standard practice with most providers, like Digital Ocean or Linode or whatever. You can call them maniacs but you shouldn’t be surprised at something so ubiquitous.
99.999% of that is over IPv4. I leave IPv6 open to WAN for my own use and it’s not a problem.
Thanks for the intel
Yeah it’s crazy. Are they just constantly pinging every possible ip address on every possible port every second? I opened a port for 1 day and the traffic made me uncomfortable enough to rearchitecture my entire self host setup
Most ssh ports should be accessed through a VPN anyway, not exposed directly to the internet
Are you sure the VPN server has the smaller attack surface than the SSH server?
Wireguard and apps based on it (like Tailscale) don’t respond at all unless the data is signed with the key of a configured peer (or with Tailscale, a system on your tailnet that passes the ACL). It’s literally impossible for an outsider/attacker to detect with a port scan since it just ignores unauthenticated packets.
The Wireguard codebase is also extremely small. The core Wireguard implementation (the version in the Linux kernel) is around 5,000 lines long, so an experienced developer can read through all the code if they dedicate a day or two to it, and understand how it works.
For comparison, OpenVPN is around 70,000 lines of code and OpenSSH is around 150,000 lines of code. There’s a huge number of features in SSH!
That is a convincing argument. I wasn’t thinking of wireguard, but it makes sense that it’s quite small and focused, being in the kernel.
I’m dealing more with OpenVPN and that can be very annoying, that had me lean in the opposite direction earlier (and lets not even speak of the Cisco crap).
Why aren’t there more minimal implementations like tinyssh?
There’s Dropbear SSH, which is the most common choice for a minimal SSH server and client, but even that is around 50,000 lines of code.
Because all those extra features are really useful. You can run all sorts of things over ssh, from sophisticated file transfer to remote development and tunneling
Ssh is remote code execution. Under any implementation, any authentication methods should be the absolute last line of defense after you’ve properly locked down access
Well, yeah but more features in remote code excution = more vulnerabilities. For example sftp: you could also just run ftp over vpn for the same effect. Or just cat the file over ssh. Instead, we have one big monolith. (same as with everything going over tcp)
OpenVPN, no. Wireguard, yes.
Can they please take a break from having vulnerabilities
nah, find and fix as many as possible.
It seems like the pattern emerging is that AI is great at finding these obscure vulnerabilities that are only exploitable on very particular systems with very particular, non-typical configurations, and only on nights where unicorns fart candycorn.
Sure, fix them, but the hype is too much. Also, disable password auth on SSH already. Jesus. Keys arnt that hard.
More like AI is much faster at finding obscure vulnerabilities, because it doesn’t get tired of digging into the mind-numbing details of human-generated spaghetti code. Give a human the same kind of task, and they would die of boredom 12 hours into it.
We’re entering an age where AI is quickly cleaning up the stupid security holes that have been created by decades of human mistakes, and will continue to be this way until increased AI security reviews will become more standard. The tribals wielding spears will continue to get shot by superior firepower until they adapt to the same weapons and protection themselves.
“Cleaning up” seems a little generous. It’s finding them, but someone still has to patch them.
This example demonstrates my point though. It requires an extremely unusual system configuration that may or may not even exist in the real world it is so unusual. AI doesnt get tired, like you said, but its not about spaghetti code (both musl and openssh are gold-standard opposite of spaghetti code), its about trying every possible software combination in ways a human wouldnt bother.
Yeah and this isn’t a new paradigm, in the sense that tools to find exploits have gotten better repeatedly and allowed those with enough knowledge to use the tools to find them, regardless of if they could manually craft the exploit themselves. There’s a multitude of bug bounties that have been claimed by those who knew how to let a reverse engineering tool like ghidra hammer away at something with variation for long enough. And the models are capable of novel exploits on their own, but generally they are just doing the same, hammering ghidra with an MCP at accelerated scale. So “only ai would have found this bug” is technically untrue and moot.
I agree about the AI, but desktop Linux has never been under haxxor pressure the same way windows or apple has. I also think the newer parts of the userbase are less technical than even ten years ago. Previously undocumented bugs found and squashed is great for everyone.
Have any of these new vulnerabilities been desktop linux?
How would i know? The write up only touched a handful of things.
This particular vulnerability is precisely about logins with key. Using password login renders this vulnerability useless.
Other way around. This requires a passwordless account configured with sha256 or sha512 password hash, musl system with openssh installed, no PAM, and password authentication allowed:
When logging into a passwordless account, the attacker can gain access by inputting more than 256 characters in the password field.
Using password login renders this vulnerability useless.
This isn’t the case, though I can see how the article could lead you to believe that.
It’s directly based on password rather than key input. The entire vulnerability is that typing a password longer than 256 characters automatically defaults to
crypt()outputting the*character instead of a normal password hash, which is coincidentally the same default character used as the “hash” (not actually a value you could get from a normal hash, intended to be impossible to get from any normal password for that reason) for the built-indaemonaccount.When the Ars Technica article says “Second, the target system account must be configured to be passwordless”, what they mean is the default
daemonaccount, which has a hash set to*by default, and thus wouldn’t normally be accessible via any hashed password… unless you happen to find out that the system chooses to not make a hash, and just send*if you type more than 256 characters.Look at https://sshhole.com/ if you want the details.
Technically they’re correct though, in a way. If every account has a password, the system isn’t vulnerable.
You’re also not vulnerable if you use two factor auth, because all 2FA modules require PAM, whereas this vulnerability only exists when PAM is disabled or not installed.
Oh, thanks for details! Missed that.
Given this is a vuln when entering the password, just turn off password auth and use ssh keys only. It’s what you should be doing anyway.
Easier said than done.
If you have to support the sort of user that barely managed to download a copy of putty a few years ago and kept using this, then walking them through key generation and use is a nightmare.
One could fairly argue that maybe such people shouldn’t have access to a system, but it is a reality.
Of course the likelihood that a system using musl is something that class of user would be accessing is lower.
Maybe now that ssh is generally “just there” for Windows users, you can more easily work with them to generate a key, or give them a zip file to make them have a key in a standard way. But the interaction of ssh and windows file security can be… Interesting, though mostly an issue on the ash server side.
The vuln is specifically for key-based accounts:
Second, the target system account must be configured to be passwordless. Passwordless accounts are commonly used for accounts that use certificate-based authentication.
“passwordless” here means that the account doesn’t have a password. What the person you’re replying to means is that password authentication should be completely disabled in the SSH config, as that’s the only way to exploit this issue.
But passwords need to be enabled for SSH. You must submit a faulty password of a specific kind, and the SSH server misinterprets it for passwordless accounts, which means the server must be configured to allow you to attempt a password.
When logging into a passwordless account, the attacker can gain access by inputting more than 256 characters in the password field.
If you disable passwords in SSH, there is no password field. There is no way for you to input anything. The connection will just be refused as the only auth method is an ssh key. If there is no matching key, SSH will refuse the connection.
If you don’t do that, the daemon will fall back to a password, which might then be vulnerable. However, no sane administrator would leave password auth enabled. Usually, you use the SSH password once - to copy an SSH key on the system. Then you disable it.
For added security, there is a pam module for 6-digit OTP
Fuck. OpenWRT might fit that bill.
Edit: it does, kind of:
https://forum.gl-inet.com/t/sshole-exploit-does-not-impact-gl-oem-firmware/72657
The person who discovered *sshole said OpenWRT is affected, but by default, that OS uses the Dropbear package, not OpenSSH, for SSH connections (though maintainers provide instructions for installing the latter).
Alpine? The one that doesn’t use glibc? Wasn’t this why they chose a different C runtime? For security. Ooops.
Dude. One issue in the time of how many in glibc…
Yes, one bug invalidates the whole project. As everyone knows, no bugs in glibc were ever discovered.
I feel like alpine is more well-known for being lightweight than secure
“Security” and getting rid of that pesky copyleft licensing in order to make it so corporate users have to give back as little as possible.
Well that sucks. IIRC a lot of my docker images are alpine based, but they’d be safe due to never having their SSH ports exposed I assume.
Why would you even have an SSH server running in a Docker container (unless the purpose of the container is being a jumpbox or similar)?
I can’t emphasise it enough how much more secure you get when you use rootless, read-only containers that are limited to truly just the software you want to run and its dependencies.
Alpine is great for dev containers, but the final container should be:
- immutable on the app layer (any executable path needs to be read only)
- have only the absolutely necessary GNU tooling, service management, etc. included (for most software you don’t even need these tools, and why would you give an attacker who gain access to your container, an easy way to e.g. scan your network?)
- only the data mounted, rw only if absolutely necessary (e.g. if you host a Jellyfin server, just an example, you should have the mounted media be RO, and only the config and caches be writable, mounted noexec of course)
Not just rootless and read-only, but distroless/chiseled too. There’s no reason to have unneeded parts of an OS in your Docker containers. It should just be one app along with any libraries it needs (or just statically compile the app)
unless the purpose of the container is being a jumpbox or similar
What’s wrong with a rootless jumpbox?
Nothing, my point was that SSH in a container is unnecessary unless it’s for the specific purpose of the container.
While I have not done it, I could see it being appealing as it’s quicker than SSHing into the parent box first for a commonly used tool. I had a direct SSH into a systemd container for a while before deciding the extra config wasn’t worth it.
Could that not be configured with a proxyjump through the host machine still, if one needed to have this?
RIP. I am glad I just walled off my own garden from the internet. Though I dont think I ever had a need for my own stuff to have exposed ssh ports. That sounds scary before I knew about this.
