I think Catalina 10.15.4 broke SSH | Hacker News Reader

I think Catalina 10.15.4 broke SSH

feed.tyler.io

733 pointschmaynard340 comments

To summarize OP:

Their 10.15.4 macOS built-in ssh terminal command is unable to reach hostnames when a port number higher than 8192 is used.

EDIT:

Comments differ; one indicates issues SSH'ing to lower than 8192 ports, another indicates no issues SSH'ing to higher than 8192 ports.

chaos:1111

Catalina is broken in many ways.

This complain and Remote Access in (so I can SSH to my $4k MacBook) disables itself anytime the computer is restarted.

But more importantly, I’ve still not found a Thunderbolt Display that doesn’t routinely crash screen manager services upon idle user activity. 3 x $300 thunderbolt3 dock solutions later and not a one hasn’t crashed this computer. All main brands, two of which sell accessories in the Apple store.

Problem also existed with a top of the line 13” MacBook Pro.

I’ve just gotten used to the shoddy-ness that is Catalina. Figure if I go to the bathroom, upon return I have a fresh, new clean desktop environment. Feature not a bug. Yay!

This is why I went back to Mojave. Apple has had a history of breaking dev environments on release for people who don't code under their ecosystem of dev tools (well on second thought, they make life difficult at times even for people that do), and I don't see that trend changing in the future.

Eventually every new release has stabilised, but it seems that doesn't hold true for Catalina.

A history of breaking dev environments? I'd argue they have a history of breaking everything on release. I do agree that they have a history of stablizing after a few months, but it's been longer than that.

Yea, Apple doesn't care about backwards comparability. You can run some ancient 32-bit Windows games on Win 10 (so long as they don't use DRM that uses low-level things like direct CD/DVD I/O). If you have a collection of physical disc Mac games, there is a good chance 0% of them run on 10.15 (or many of the releases leading up to it).

There is a good chance that less than 10% of your mac games from a few years ago work on current macos since they killed 32bit support. And now there will be essentially no new mac games since they will not support vulkan and stopped updating openGL.

we are getting close to 10 years since I owned a Mac with an optical drive (and definitely a decade+ since I used an optical drive on my mac very often)

haha oh boy. I was actually just about to install Catalina today, figuring I'd put it off long enough and everything has to be smooth by now (and system update bugs me about it often enough)... But lo and behold, I log into HN and see this thread....

Just FYI, you can disable that notification:

https://www.macworld.com/article/3447396/how-to-stop-getting...

It does not prevent the red notification dot on the System Preferences app, but it does mean at least you don't get the notifications pop up on your screen.

I’ve had zero problems with Catalina. You can also find complaints about every release of MacOS going back a decade+.

The unix illusion breaks more and more with every release

Every day they stray further from BSD's light

The main reason why I won't buy the new MacBook Air is that it's Catalina only. Good thing my Mac Mini shipped with Mojave despite my ordering it months after the Catalina release. It's really the Windows Vista of macOS.

I've tried to find a thunderbolt3 dock that worked perfectly but none have - not even limited to Catalina. My monitors will randomly switch refresh rates or resolutions or not even display picture. Plug them into a pc and they work every single time.

I have Catalina. It doesn't play nicely with a Dell D6000 powerbrick / dock.

The display is fine but it won't charge at the same time.

I have not installed the Dell 'driver'; it loads a kext so probably won't work anyway. I'm not upset about that. Docking should not require a kernel module.

That's about it. Catalina has been fine every other way.

The thing is with this charging bullshit is, it worked fine prior to 10.15.4. Prior to that version, my Mac charged and outputted to multiple screens at the same time.

Try a Henge dock. Been using one for 6 years. No monitor or thunderbolt issues.

I actually have one but haven't really used it yet since I decided to just use windows and be done with all the mac problems for now. Still use my MBP but only from the couch.

Wow, there are so many mac problems in your workflow that WINDOWS is easier to use? Ouch.

The Dell U3419W (with Thunderbolt 3) works exactly as promised for me.

https://www.dell.com/en-us/work/shop/accessories/apd/210-arc...

I've had a lot of compatibility issues between my U3818DW and macOS and Dell doesn't care. Even more, their support staff on their public forums don't even acknowledge their buggy USB-C implementation and insists on blaming Apple. [1]

[1]: https://old.reddit.com/r/UsbCHardware/comments/ettgrg/dell_r...

That's a shame. This monitor is spectacular over DisplayPort with my good old 2015 MBP.

Indeed, the image, price and size are spectacular. But this atitude has put me off Dell and they won't get my business anytime soon.

That's true for most Macs, too: people are very prone to believing that their experience is universal rather than a hardware failure or local configuration issue.

FWIW, the monitor I talked about is the standard issue monitor for most employees (~500) at my office. My company has MBPs ranging from 2016 to 2020 of varying sizes and I've never heard anyone say anything negative about the connectivity to their monitor.

It has USB-C, not Thunderbolt.

I completely disabled everything to do with sleep. “Solved” the problem for me.

The first thing I install on a fresh Mac is amphetamine.

Just curious, how is this different from Caffeine?

http://lightheadsw.com/caffeine/

It has more features. Check https://apps.apple.com/us/app/amphetamine/id937984704?mt=12 for details.

The one you linked to looks like a simple wrapper of caffeinate(8).

Caffeine might be better in this scenario, but amphetamine has some auto triggers and custom settings for keeping the system awake but still turning off the screen, etc.

One difference is that Amphetamine isn't updated as well, also seems more hackish. I switched to Caffeine a while ago.

Or just running caffeinate -t 10000?

Maybe I will get flamed for saying it. What a shockingly tasteless name for an app.

Edit: guessing that downvoters haven't encountered a lot of people suffering from addiction

The most popular third party software installation tool for MacOS is called Homebrew. Alcoholics exist. That doesn't make the name offensive.

Amphetamines have legitimate therapeutic uses, it's not only crippling addiction.

Is there a world-wide approved list of names and topics that we're allowed to use to avoid infringing on sensibilities of 100% of world population?

I take prescribed amphetamine [0] twice every day. The taste of the name is not in the name, it's from whatever else you had in your mouth at the same time.

[0]: https://en.m.wikipedia.org/wiki/Lisdexamfetamine

Yeah, I've had my problems in the past and whatever that app is, I wouldn't be keen on having that on my computer, it would only serve to remind me.

My issue with Catalina is that every time i open up the laptop and log in (so sleep, not reboot) it has forgotten the Apple-ID password and needs to be entered. I've tried all suggested solutions (I'm not alone) including resetting the NVMRAM etc. But so far no luck.

I'm holding off installing Catalina on my main machine. And now they seem to focus on 10.16 instead.

I have two CalDigit TS3+ docks (home and office). At home, I have a 4k monitor in the display port and a Thunderbolt 27" plugged into the TB3 port using an adapter. My previous dock couldnt handle the 27" at all, so I had to plug that in directly to the Mac. Usually when I needed to wake the machine, I had to unplug both the doc and the 27", log in, and then plug them back in again. Now with the CalDigit, it just works. It's also like $300, so I guess TB3 is hard and they know it =)

I am still on Mojave tho, so may suck on Catalina.

Figure this is the place to jump in here. I tried a couple cheaper docks and sent them back to Amazon immediately. I bought a CalDigit, and it's been rock solid for many months now. I connect an external display (Asus 27" 4K) to it, remove it, use the built-in display, and use it in clamshell, off and on, all through the day. Not one problem at all. There's no sugar-coating it; they're at the top of the range for TB3 docks, but mine's been worth every penny. I've been on Catalina since launch day.

I know "me too" is discouraged here but yes I have the same experience (Caldigit TS3+ dock, works great in Mojave). Expensive, yes, but at least now it's down to $250 both at apple as well as amazon.

I had this problem as well on a 13" MacBook Pro.

My "fix" was to go HDMI to USB-C (instead of Thunderbold to USB-C).

I understand this might not be viable for everyone, but it resolved the issue for me.

", I’ve still not found a Thunderbolt Display that doesn’t routinely crash screen manager services upon idle user activity. "

I've been using a Dell U2515H for almost six years on my late 2013-model MBP and thunderbolt port, never had an issue. I'm also going through a Henge thunderbolt dock. It's not a macOS problem.

I think Dell U2515H doesn't have Thunderbolt, only Display Port. So I wouldn't call that a Thunderbolt display, if it doesn't have ability to chain further Thunderbolt devices.

naive take, i have same problem but if i boot into windows on the mbp it works fine. how do you explain that?

Trivially easy to explain:

The monitor you have didn't properly implement the Thunderbolt spec, and since Windows has looser adherence to the spec than macOS, things work fine.

This happens with web browsers every decade or so. "Browser X" follows the Javascript spec to a tee, which breaks millions of poorly written websites, so "Browser X" has to degrade its performance or lose market share, and thus we have lots of sites that are out of spec.

Make sense?

Your explanation could be plausible but do you have any evidence to back it up? Most curious is the fact that people are complaining about a specific OS version with regard to the problems. Did the spec change between OS releases?

Assume this is true. Why is it a good thing? If the looser spec handling on Windows fixes the bug without introducing other problems, then from the user perspective, Windows is doing the correct thing and OSX is failing.

LG 4K works perfectly for me.

I've tried every T3 dock available. They all have bugs that render them unusable for me. The one that was the closest to being good -- OWC 12 port I think -- wouldn't tolerate MBP sleep. After wake from overnite sleep (maybe the Mac would go to hibernate -- I didn't investigate further) the dock would need to be reset. I've never had the MPB crash though, but I haven't gone back to trying docks now with Catalina.

There certainly is something particular to your environment causing this crash. Such a bug would be in all the news.

I haven’t encountered that, but have other more minor gripes. When in clamshell and an external monitor is plugged in and you restart all you have actually done is shutdown (you you have to open up the laptop and turn it on again). The way things break for ‘security reasons’ which you have to hunt for though the settings page. Eg VMWare Fusion won’t work unless you happen to know that it needs enabling in security settings, but some breakages are even more obscure and don’t generate an error message.

Oh, so that's why my computer restarts from time to time when I get away from my desk. And I'm not even on Catalina yet, just use a 13" MacBook Pro.

My 2013 Mac Pro does this too - it's actually (at least in my case) a kernel panic.

> This complain and Remote Access in (so I can SSH to my $4k MacBook) disables itself anytime the computer is restarted.

I've found IPv6 stops working after sleep, the appropriate area in the network pane is blank (I use RA not DHCPv6). Since the Mac updates its DNS records and puts IPv6 addresses in I've found accessing via hostname stops working, but then of course I can use the IPv4 address which works fine.

Yep. I turned off ipv6 support on my router an computers, and still use RA. No more issues on local network except one ... the DNS settings on my MacBook constantly revert to a default value, killing my host name access to my docket containers. But at least it’s a quick fix.

Yeah both of my TB3 docks result in crashes after unplugging or plugging in while the display is off. Very annoying behavior.

I've been using the Belkin Thunderbolt 3 dock for years now, and have had 0 issues with crashes.

So, some quick debugging here...

In his screenshot the bad login hangs at "Connecting to clickontyler.com port" (noting that no port number appears and no period at the end).

While I can’t be sure exactly which "ssh" patch Apple may have, this seems to be the relevant file and logging code (starting at line 448):

https://github.com/openssh/openssh-portable/blob/master/sshc...

In that code, the only thing that can set the "strport" value that is used in the log is a call to getnameinfo().

If that string is corrupted in any way, e.g. not terminated or perhaps has invisible characters that trigger bad terminal behavior (such as invisibility), the act of logging it might produce the apparent hang seen here.

Again, a guess but it is possible that getnameinfo() is not necessarily processing the record correctly (for whatever reason). One such example is in the "getnameinfo" man page at the end, under CAVEATS, where they show an example of not simply trusting the result of the first call.

Good sleuthing, but the missing port number is simpler than that. I just blacked it out of the screenshot. I know very well that running sshd on a non-standard port has no benefits security-wise, but it does lessen the length of my log files from dumb script kiddies. I redacted the port in the screenshot for that reason.

You should mention that in the caption, or use a non-black colour as a mask.

>I know very well that running sshd on a non-standard port has no benefits security-wise

I don't know if Mac OS is different but on other unices ports above 1024 are not privileged, meaning that anybody can bind them. Now it increases the attack surface only a tiny bit (you have to have your sshd offline, and the attacker have local access, and them bind a fake sshd to your port in order to MitM. And even then they won't be able to spoof the server key unless it's not chmoded correctly).

Still, better safe than sorry IMO, I also use a non-standard sshd port but I keep it in the low range. In my experience it's more than sufficient to get rid of 99% of dumb attacks that generally don't bother looking beyond port 22.

I think using a non-standard port is a good layer of security, among other layers.

My personal suggestion though is to use 1022 because it's below 1024. This means only root is allowed to bind to it. Preventing possible connection jacking attacks if an attacker is able to crash your own server and run theirs to harvest your passwords.

You might add a few "-v"'s to your "ssh" command-line for more verbose debugging information.

A port is mentioned in this line, you may want to redact it. Where I put X's below, is a port number.

> So, I tried ssh ip-address -pXXXXXXXXX

Thanks, but that's not the port number :-) That was just for illustrative purposes.

Ok great.

Have you tried running ssh in lldb/gdb and dumping a stacktrace when it hangs? Might have to copy the ssh binary to a temp dir to avoid SIP denying ptrace.

Doesn't even need to go this hardcore; simply reading the verbose output would show where things are getting stuck.

The verbose output didn't seem to point out the exact system call or libc call that got stuck. A lldb/gdb bt stacktrace could pinpoint what's hanging (for example, some people mentioned parsing /etc/services). I don't think this has been resolved yet?

Disable password auth and go with keys only, and your logs will go quiet.

Maybe there is something funny in /etc/services on this machine that throws the call into an infinite loop? Perhaps near the bottom beyond port 8192?

"It just works" -- Is Apple too large now? Is this a QA problem, product team problem? Management? Catalina is still stumbling and Im surprised to be honest after the past 4 years.

My feeling is that Apple beancounters have decided macOS is mostly a gateway to Xcode for iOS development, anything else is just to help sell laptops. The stuff in "anything else" doesn't need to actually work well, just exist so it can be something on the features list.

I feel like they are doing random deprecations with replacements that don't work as well as the original. As in, leaving the deprecated thing unmaintained but present in the install would be a better outcome. I wonder why they are wasting so much time doing this when they appear to have a working system. I'm not even talking about big items like 32-bit support or opengl but completely random libraries that work fine.

This is a completely standard failure mode of large organizations. You have a product that works perfectly fine the way it is, but you also have an entire team of people whose job it is to do something with that product. The existing product has already been optimized for years and most changes are moves away from optimal rather than towards it, but they can't get paid to do nothing, so they change things that were better the way they were.

This is related to the thing where what customers want most is bug fixes for existing bugs but what marketing wants most is new features to sell to new customers and marketing tends to win, which causes the number of bugs to go up rather than down over time.

It's also a problem of company culture and career ladders. Fixing bugs and making a more stable product isn't going to line you up for a promotion - but some fancy new feature no one asked for will.

I think this is why Google's products have gotten worse over the last 6 years.

As just another random instance, I updated my MacOS about a year ago and now I can only change the last 3 parts of my MAC address, the remainder appear to be fixed.

I know my hardware has the ability to change my entire MAC address - I don't get why they are doing this.

Branding?

The leading octets in MAC addresses are often called "vendor prefixes", and are assigned to various hardware vendors. Apple probably wants to ensure that all their devices show up in ARP scans and MAC lookups as Apple devices.

To make it harder to spoof specific devices, perhaps. Commercial end-user OS vendors generally don't think your computer being able to do something implies you should have control over that capability.

I guess they are getting ready to run MacOS on Arm rather than AMD64

It does help that there is no overall competitor to MacBooks in terms of ease of use or (now that the butterfly keyboard is dead) build quality.

There are decent build PC laptops but you have to run Windows or Linux on them. Windows is a dumpster fire these days with ads in the start menu, the use of "dark patterns" to herd people into MS cloud, and out of control unnecessary telemetry. Linux is fine only if you have a lot of time on your hands to troubleshoot edge case issues and hunt for drivers. Linux also still (through no fault of its own) can't run a lot of apps that many people need.

Linux is the only option IMO, but I have a very high yakshaving tolerance. That said, if you run a recent Ubuntu, most stuff "just works" as long as you don't need Photoshop or the Office suite.

Or 4k monitors, or screen sharing when running more than one monitor. That's the reason I haven't switched from macos back to linux (I was all in on linux until about 5 years ago when I started to care about display quality and working remotely).

You know, in 5 years, many things changed.

What's wrong with 4k monitors? I'm typing this on Fedora machine with one (default install with no tinkering, Gnome on Wayland).

I'm using Linux with 3x 4k monitors at work. I set the scaling to 150% and it just works. I'm using Awesome WM.

Two 4k monitors here on Ubuntu Mate, works great. Shared my screen last week.

External monitor support on macOS is terrible. When it does work, you can't turn on HiDPI resulting in a tiny UI. And the latest 16" macbooks simply kernel panic: https://discussions.apple.com/thread/250876794

I would agree were it not for the hairy yaks. As a startup founder I just don't have time for my computer to not "just work." This is the primary thing that keeps me absolutely glued to Apple.

I do kind of like MacOS, but am concerned about their lack of strong interest in it.

I would pay for a "vertically integrated" open hardware Linux laptop. I've seen some promising projects but none are mature enough.

The second issue is apps, but that can be mitigated by having a Windows VM.

But this entire thread of comments and even the topic of the post is proof that it really just doesn't work.

I would argue that any major Linux distro at this point "just work" just as well as MacOS

Or do any kind of serious 3D or audio related work.

Is this still an issue with Ubuntu? I haven't had any problems with drivers nor software for... 6 years and 7 laptops?

I've had problems with printer drivers consistently since I switched to (K)ubuntu on the desktop in 2010ish. Since 2 years however, they are basically gone. That's thanks to IPP becoming more commonplace.

Then I'm having issues with PTP from my phone. Windows is fine but Plasma is broken. The phone also offers an MTP mode which thankfully works.

When I bought a Lenovo netbook in 2015, I was unable to set the screen brightness. It took a few years but eventually the issue got fixed with a new version of Kubuntu.

On my brand new ThinkPad T495 I'm having an issue with the graphics drivers, which crash and require me to issue an ACPI reboot when I close the lid and reopen it again. Pretty sure it's this issue as the error messages, symptoms and working workarounds all match. https://gitlab.freedesktop.org/drm/amd/issues/883

KDE connect made plugging my phone into my computer basically obsolete, at least for me.

It might be that I use pretty standard hardware and don't have any fancy requirements, but really, I have evangelized several people and installed mostly Xubuntu in their laptops and I haven't had problems.

My APU stuck in OpenGL 3.3 without video hardware decoding, would like to get OpenGL 4.4 and hardware video decoding back that it had with the AMD proprietary driver.

Support for 4K monitors and multiple screens is still pretty miserable and causes stuttering, freezing, and crashing

There are plenty of good laptops and Windows is absolutely fine.

Windows is not "absolutely fine". Ignoring the garbage heap of bad/inconsistent ui/adverts/nagware. Its just not even capable of running a lot of dev software. The guide for running ruby on rails on windows is basically just to install a linux VM.

If you pick your laptop for linux support then you will have literally no driver issues. I'm running fedora on a Dell XPS and it runs flawlessly (Well the fingerprint scanner needs a 3rd party program).

Ubuntu is generally even easier since they bundle in proprietary drivers.

As if, I bought a Linux Laptop from Asus with Ubuntu, and my APU is still to get the OpenGL 4.4 that it had with the proprietary AMD driver, instead I should be happy that the open source version at least offers me OpenGL 3.3.

Well no business end-user or any typical Mac user is going to be bothered about something technical like 'SSH' breaking their system. Only actual devs here would care.

For those business users, it just still works. For developers it's a problem.

Apple's made huge inroads with developers over the last few years, partly coasting off of a social dislike for Microsoft. There's enough Apple fandom out there that they can probably annoy developers a good deal more without affecting the inroads. After all, exactly what can a dev do about it anyway?

Switch to OpenBSD ;-)

Apple developers are perfectly fine.

UNIX developers, well, support OEMs that sell BSD and GNU/Linux laptops.

Macs have a fairly large share of devs, especially in the startup centers like SF and NYC. Most startups end up with macs as the default computer because of the developer experience as well as the ability to manage them for a consistent user experience using MDM solutions like Jamf or Fleetsmith (both Apple-only)

You seem to be nurturing some stereotype of Mac users. Just check (photos of) any Silicon Valley or MIT Cafeteria to maybe calibrate your worldview.

Catalina I haven’t had much problems with, however noticed some odd stuff. Like the Apple Menu and System Preferences it reports one update available but if I go look - nothing. Then was playing with the new TV app and went to watch one of the Apple TV+ shows and all I get is a black screen with audio when watching a show.

Then even before Catalina, my AirPods mic seems to act odd, can hardly hear it and it messses with audio output too when listening to music, sounds like I’m listening to hold music on a telephone unless I disable the mic using a third party app. I think having a old Bluetooth chip might be the reason though since I have a older MacBook while it works great on my iPhone.

Almost everyone in my office has issues with Bluetooth headphones mysteriously disconnecting - the sound output drops even though Bluetooth is still connected.

Very annoying and can't find a resolution.

A focus on security produces problems like these.

I can't blame them too much. It's probably worth it.

Probably just trying to make Catalina thinner.

Here's an actual bug report:

* https://openradar.appspot.com/radar?id=4931259776106496

From that and the discussions.apple.com. post, hyperlinked elsewhere in this discussion, it appears that the >8192 condition varies according to what the hostname actually is.

The bug report is datelined 2020-04-26, interestingly. There might be a bug in the bug reporting system. (-:

> The bug report is datelined 2020-04-26, interestingly. There might be a bug in the bug reporting system.

No, you can type whatever date you want. The "add a new radar" screen is just a bunch of text input boxes: https://i.imgur.com/nNf457J.png

The ability to type whatever date one wants is often considered to be a bug. The ability to post-date reports a month into the future sometimes is, too. (-:

You can not only type whatever date, but also whatever non-date. The site just assumes good intentions and is working as designed.

OpenRadar is community-maintained, and rather poorly at that these days.

I can't reproduce this. macOS 10.15.4, ssh'ing to a very high (5digit) port with a hostname no problems.

Same here, 5 digits and a hostname = no problem. There must be some other factor(s) in play.

Ditto. There's a post a few levels above where they are digging into the source of ssh, I'm following that!

I can't either, in fact all 10+ of the hosts that I routinely access have ports higher than OPs issue.

Offtopic but why are people using high port numbers? Additional security due to a nonstandard port? If so, does that go together with anything additional like port knocking? Or is it multiple hosts on the same IP, but different ports?

Some people think that it adds to security. Some people want to reduce noise in logs.

A relatively common use case is multiple devices behind a NAT, where each port goes to a different device.

It's to keep my logs cleaner. It doesn't add any security value since the port is still open. I don't allow password auth. I was just always annoyed with how many times port 22 was getting hit everyday by attackers.

I had the same problem on a MacBook after upgrading to 10.15.4. However, I wasn't using a port number higher than 8192, the socket was 75 with a hostname. The problem was solved when I replaced the hostname with its IP or plugged in an Ethernet Cable. I tried to restart mDNSResponder and flush the dns cache and switch to a different DNS server. Nothing works so far.

I experienced a similar issue with a git repository hosted on a high port, `brew install openssh` fixed it even though the homebrew `ssh` was not first on my $PATH. Didn't bother to investigate further.

Possibly that overwrote the config file for the system ssh?

I was thinking homebrew's git perhaps had a different $PATH (or was using shared objects?) that used the different openssh. Just guessing, didn't seem worth my effort at the time.

Isn't /usr/local/bin first on $PATH?

Homebrew tends not to install to bust/local/bin when that would replace a native version.

I suspect this is due to a feature being enabled for canonicalization and that the key part is the presence of the colon rather than the port number. On a 10.15.4 system, I see a line in the debug output which is not present in the screenshot:

> debug1: resolve_canonicalize: hostname example.org:7999 is an unrecognised address

If instead I use `-p` or a config-file option, everything works as expected.

hostname:port is not a valid destination according to ssh syntax. A destination may either be [user@]hostname or a URI of the form ssh://[user@]hostname[:port].

The Apple forum article cites by the OP indicates it's a problem with `-p <PORT>` as well.

https://discussions.apple.com/thread/251226509

That's a fair point. I use either a ssh config file with all the correct options in it, or the -p option if I'm doing it without a config file, perhaps that's why I've never had a problem.

"I don’t want to end up on Hacker News again bitching about Catalina." Pretty sure that guarantees getting to the front page :D

This is truly the darkest timeline for that poor blogger. :-P

Those are the magic words. F*ck SEO, add that and you'll get a couple million page views.

Is macOS /user/bin/ssh just upstream OpenSSH, or does Apple maintain a fork? If no fork, this would be an upstream OpenSSH problem, no?

Apple includes a customized version of OpenSSH. From what I recall from the last time I looked at it, the changes were mostly integrating the key retrieval mechanisms with the rest of macOS. For example, Apple's ssh-add can store key passphrase in Keychain with the -K option, and then later access those passphrase with the -A flag.

If using the upstream version there is one line to add to a startup script or to your zshrc (et. al.) file ...

ssh-add -A > /dev/null

... and one default value to place in your ssh config file...

AddToKeychain Yes

... to get around this issue. It works fine after that.

(On mobile. Sorry for formatting)

Those stored key passphrases are visible with the Keychain Access application, Kind: "application password", name: "SSH: /full/path/to/key", in the login & iCloud keychains.

https://opensource.apple.com/source/OpenSSH/OpenSSH-235/

Looks like that's for macOS 10.15.0 as version.h for that OpenSSH release is for 7.9p1 (macOS 10.15.4 has OpenSSH 8.1p1).

It is, I pulled it from https://opensource.apple.com/release/macos-1015.html.

I'm pretty sure they use libressl now.

Maybe a weird ControlMaster/ControlPath config? I have had issues with the ControlPath result being too long with certain hostname/port combinations in the past -- which resulted in ssh to ip working but ssh to hostname not working. As a result, I haven since started using %C instead of %l%h%p%r in my ControlPath config.

If you have Homebrew or something similar, I recommend installing openssh through there -- you get a newer version to boot.

It's usually not a full replacement. SSH for macOS has some integration built in that current OpenSSH does not have, like Keychain integration.

I thought Homebrew patched OpensSSH using Apple's keychain patch, but looking at the formula right now I see

  # Please don't resubmit the keychain patch option. It will never be accepted.
  # https://github.com/Homebrew/homebrew-dupes/pull/482#issuecomment-118994372

Sadly the homebrew-dupes repo seems to have been deleted so this comment can't be read anymore.

Archive of the discussion: https://archive.is/hSB6d

> We are uncomfortable continually supporting a 1900+ line patch which upstream hasn't signed off on that has the potential to both compromise OpenSSH security and Keychain security. From 10.11 it will also be impossible to edit plists in /System/* without disabling rootless, which isn't a configuration we'll be intentionally supporting.

> Sadly the homebrew-dupes repo seems to have been deleted so this comment can't be read anymore.

They're kinda bad at that in general :/

Really? I have my ~/.ssh/config file set to “AddToKeychain” on all entries and it doesn’t seem to be a problem.

I prefer ssh-agent anyway, but yeah, I think they did remove the keychain integration patch.

Oh god no. Homebrew managing openssh has been the cause of more command-line instability and forced reinstalls than anything else I’ve encountered in the last few years of OS X (sorry, macOS). I’ve started installing stuff from source again just to prevent a cascade of Homebrew upgrades breaking everything.

Why don't people use MacPorts instead? I've never had any problems with it.

Homebrew wants to screw around in /usr, Macports installs itself in /opt and doesn't interfere with things in the MacOS world.

Set your PATH to have /opt/local/{bin,sbin} and everything Just Works.

What is the practical difference between /usr/local and /opt or /opt/local? I don't think macOS puts anything in /usr/local.

I sometimes use NetBSD's pkgsrc on macOS because it installs super cleanly in any prefix you like and never, ever breaks the system. It doesn't have everything, and you will occasionally encounter a package that won't build, but it doesn't even dream of taking over /usr/local or disrupting your system. You could install it into your home directory if you wanted to (which I have done, on systems where I don't have root or enough ownership to just throw things anywhere)

I always build SSH from source myself using my own scripts and meta-makefiles. Both the most recent OpenSSH release, and the latest one supported by HPN-SSH (for use on high-latency links).

OpenSSH 8.2p1 notably has support for using FIDO U2F 2FA keys to secure SSH keys, it works perfectly, as long as your server also runs 8.2p1 (only the client needs to be compiled with libFIDO2).

As for the Catalina train wreck, it's clear both hardware and software quality is on a severe downward trend at Apple, you can either rant and moan about it, or take control back by switching to Linux or BSD, which is what I am doing, very slowly and deliberately.

Never experienced this in a decade or so of using Homebrew's OpenSSH, but you can absolutely use something other than Homebrew to get a more up-to-date and standard OpenSSH install if you prefer.

> I’ve started installing stuff from source again just to prevent a cascade of Homebrew upgrades breaking everything.

Since you crossed that line, do yourself a favour and check out nixpkg.

I always do this.

I'm not sure what's the current state, but there are features on SSH I wasn't able to use due to the version provided being old.

I know that `Include` on `config` is/was one.

    Include "some/path"

This is something I use frequently that wasn't available on previous built in versions.

> I'm not sure what's the current state, but there are features on SSH I wasn't able to use due to the version provided being old.

> I know that `Include` on `config` is/was one.

That's both terribly out of date info and hardly ever true as far as I can tell.

The Include directive was a new feature of OpenSSH 7.3, released on 2016-08-01.[1] Apple shipped OpenSSH 7.3 in macOS 10.12.2[2][3], released on 2016-12-13. That's a very reasonable four months gap.

I only use the system ssh because stock OpenSSH didn't integrate well with system keychain many years ago (not sure about the current state). But I've been using the Include directive for a long time.

[1] https://www.openssh.com/txt/release-7.3

[2] https://opensource.apple.com/release/macos-10122.html

[3] https://opensource.apple.com/source/OpenSSH/OpenSSH-209.30.4...

In case it's slow for you too: <removed since they wanted it taken down>

It’s been taken down on purpose: https://news.ycombinator.com/item?id=22738841

I'm surprised nobody noticed this bug at Apple before the release. Is there nobody there that connects by hostname to a ssh server with a port > 8192?

I'm not. Not all 'testers' actually try to test edge cases. The /good/ testers do try edge cases, but for every /good/ tester you have, you'll have hired 100+ testers who do little more than check that the standard happy-path works correctly and sign off as "passes tests".

The good testers all tend to fall into what Bruce Schneier calls the 'Security Mindset' way of thinking: https://www.schneier.com/blog/archives/2008/03/the_security_...

>Not all 'testers' actually try to test edge cases.

Yeah, but surely macOS devs are eating their own dog food.

"I've learned that Apple engineers have internal tools which allow them to delete macl xattr as well as to bypass other Catalina privacy and sandbox protections without rebooting and disabling SIP.

"Inside Apple they don't suffer the same problems as external users and developers."

— https://twitter.com/lapcatsoftware/status/121929275891082854...

And a simple shell script to test it would be easy.

Well, I hope for them they put it in their automated regression test suite now.

> Is there nobody there that connects by hostname to a ssh server with a port > 8192?

I use alternative port but < 1023 since binding to those ports requires root. And I've never seen it being used. I'm not saying it's not, just that I did not see it in 10 years.

So it probably really is not that common.

Probably not. I doubt I’ve used a port that high.

Why not? I regularly use 5-digit ports - 3-digit prefix plus 2-digit 'official' port - for various systems which reside behind a NATting router.

Ports <1024 require root access to bind, so on a multi-user system it would be insecure to run ssh on such a high port.

(Granted, multi-user hosts are very rare nowadays).

I'm pretty sure it's generally not advisable to use ports this high since they are used for other purposes.

All purposes are equally as valid on unallocated port ranges. Inconfigurable port ranges with no fallback when those ports are in use is bad design.

Isn't 8080 used almost everywhere ?

8080 is less than 8192. This bug only happens when the port is _higher_ than 8192.

sorry my bad what was i thinking

Not for SSH. Common alternate SSH ports are 222 or 2222 which are well below.

443 to pass the stupid corporate firewall. I used that once 15 years ago anyway.

For webservers sometimes, never seen that used for ssh.

< 8192, and that’s mostly the alternate HTTP port.

8080 < 8192

I haven't in ten or twenty years, no.

One more data point: just tried to repro with hostname and port 8193 and failed, so the issue is probably more intricate than described.

(Guest in my test: OpenSSH 7.6p1 on ubuntu bionic, stock config other than sshd port.)

Tried with dropbear and sshd on Arch, port 9022 from 10.15.4 Beta (19E258a) - no issues with a variety of hostnames that end up at the same host.

Just updated to 10.15.5 Beta (19F53f) and still no issues with dropbear or sshd.

I've been a Linux user for the last ~15 years, and now I need to do some iOS development so just a few days ago I've ordered a Mac Mini. I guess I'm in for a bumpy ride. Oh well.

I used linux on all my laptops/work machines for about 7 years, then switched to macbook pros 5 years ago. Definitely some things you have to adapt to (I still miss focus-follows-mouse). But for the most part, you'll find it's a smooth ride. When bumps like this happen, they tend to push out a fix quickly - especially when it gets traction on HN like this.

I used a mac for almost 8 years. When things worked, and you shared the same preferences as the designers (or were willing to adapt), things were pretty good.

If you had different preferences, mostly too bad. Maybe if you reboot with system protection turned off, you can edit the config file, and hope it doesn't get reverted.

If things didn't work, like when I was getting static for audio 25% of the time I hit Play in iTunes from a shoutcast server for a whole major release, there wouldn't be any useful help on the internet. Maybe somebody had a similar problem 3 releases ago, but that fix doesn't work anymore. Other problems, or irritants are often the same way.

With Windows, most of the problems you run into are fixable, and easy to find. With an open source OS, at least you can dig in and try to fix your own problems.

For me, focus-follows-mouse is most useful for terminal windows. It is a feature you can enable in iTerm2.

This doesn't help across applications of course, and there's a reasonable argument that the inconsistency is worse than the absence -- but for me, iTerm2's FFM feature helps.

Didn't know iTerm2 did that. Thanks, I'll check it out - might help a lot.

macOS is POSIX UNIX. it definitely has focus follows mouse.

You just mean that hover over a lower window allows scrolling right? MacOS has that.

Not just scrolling.

https://en.wikipedia.org/wiki/Focus_(computing)#Focus_follow...

It's honestly not that bad, but I'd advise to go into it with an open mind. A lot of switchers get angry at macOS when their habits from other systems don't work with it (e.g. wanting to maximize all the windows)

You don't have to have that open of a mind. I was angry at macOS because I didn't have good window management, but it was really easy to install a third party utility to do so.

I use Amethyst which is much easier to setup than my old Linux WMs. There are also tools like Yabai which are more customizable.

Sure, I guess you can also do that. I'd still advise to not go out and "fix" everything the minute you boot up OS X. It's worth learning how and why it works before you hack it.

If you're a linux user for that long you've probably got experience updating CLI utilities. That's all that's required here, simply installing a new openSSH.

I recently made the transition (from ~10yrs Linux) to Mac and it was really smooth. At the end of the day it's just a Unix system with a really nice looking Window Manager and lots of supported apps. If you don't like the included version of SSH, just use a different one, same as linux.

I'm on mac OS 10.15.4. SSH has worked on all betas and the official release from Apple.

Using a port above 8192 and connecting by hostname?

Yes, but maybe I've mucked around with my ssh too much.

As an aside; is there a reason to host SSH on a non-standard port? I recently came across a system that had it listening to a really high port number. I dismissed it as security through (bad) obscurity but is there a valid security reason to do this?

EDIT: Thanks to everyone who answered my question! It makes sense to me now why one might do this.

It massively reduces the number of script kiddie attacks. It's not hard to find SSH on a non-standard port but most SKs don't know or don't bother.

Other possible reason is NAT. If you've got several machines or VMs but only one public IP you can port forward different public ports to port 22 on different machines. Not the only solution by a long way but a relatively straightforward one.

There's actually a good reason to not use SSH on a non-privileged port: It allows an unprivileged user to bind their own binary to the port when SSH restarts or otherwise stops listening.

That unprivileged user will not have the SSH host key, which will create a warning for any user who connects, just as though someone had conducted a man-in-the-middle attack.

Of course, there are plenty of privileged ports to choose from.

https://www.google.com/search?q=random+number+between+1+and+...

IMHO, this isn't "security through obscurity" but it's a way of weeding out automated attacks and reducing logs filling up with completely avoidable entries. I'd say it has valid "sysadmin" reasons but not "security" ones.

Given 2 boxes with the exact same SSH setup (key auth, fail2ban, or whatever else you use) I'd prefer to admin the one with a non-standard port solely for the fact that it's not undergoing constant attack which uses resources (albeit tiny).

Valid reason:

Testing something that uses ssh, but the test host already has a sshd running on port 22, and one does not want to disrupt that setup for the test. Or running tests as a local, non admin, user and one does not want to bother the admin with modifying the system sshd setup for those tests.

Other reasons:

Those doing it /instead/ of running on port 22 are usually doing it for one of at least two reasons:

1) a false sense of security. If you do an internet search, you'll find plenty of blog posts boasting that using an alternate port is a security feature (it is not, it is security via obscurity); or

2) to reduce the log growth from all the script kiddie scans that target port 22 (note, no security is added here, but one's log files don't grow quite as rapidly either).

Slighty reduces your risk from all the automated spam. Most things that are scanning the entire internet trying to brute force weak passwords and stuff aren't trying 65,000 ports on each host. Any sort of worm/botnet will probably be in the same situation.

The only hosts we have with any SSH exposed to the world at all are a couple of bastion hosts. Day-to-day we access everything else through a VPN, so its only exposed at all as an emergency backup in case the VPN breaks. Really no inconvenience to having it moved to a high port.

It reduces the amount of endless bruteforce attempts somewhat, so log files are slightly more readable. Although recently this seems to be significantly less effective compared to the previous 20 years...

If it's exposed to the internet, definitely. Port 22 gets hammered.

Indeed. In fact it's hammered so bad that it made my home machine crawl at times. By changing ports it went from several thousands attempts per hour to a few per day.

I use Chrome SSH to ssh into my WSL2 debian instance on Windows 10. Native terminals don't support mouse events. But port 22 can already be in use by the host system, so I have WSL2 configured to listen on port 222

There are pros and cons. It does mean that you should get less ssh bot spam but it also means that you run ssh on a non privileged port - one that a malicious application running as non-root might attempt to exploit.

The author didn't say it had anything to do with security. I have one or two SSH daemons out there listening on non-standard ports because of stupid limitations of middleboxes I'm forced to use, or because they're port-forwarded through something that already listens on 22 itself.

Security-wise, it seems pointless; my daemons on random non-standard ports still get hammered, and fail2ban takes care of keeping the log spam down just as easily as it does the ones on 22.

Security-through-obscurity isn't a bad thing. It's just bad to overestimate what it can do, or for it to be your only security.

I have my publicly-accessible SSH port on not-22, just to avoid the log messages from scanners. I'm well aware it does not, on its own, actually "secure" anything, but it brings more convenience to me for it to be a bit obscure, and it certainly isn't hurting anything.

Significantly less log spam.

I faced the same problem two weeks ago with the previous version of Catalina (I don't remember the correct number but was a previous on 10.15.4) and git (I use SSH to authenticate with the server). So I did a report to Bitbucket with a solution that worked for me after investigate more about the problem: https://twitter.com/di3goleite/status/1239596891471581189?s=...

Thank you about that clarification. Also your website seems to be down actually.

"I’m not even going to go into it. I don’t want to end up on Hacker News again bitching about Catalina."

Apple's stance is that it didn't happen unless someone reports it using Radar (internal) or bugreport.apple.com (external). Unfortunately, they don't believe in Linus's Law, which states that "given enough eyeballs, all bugs are shallow."

Things that end up on the front page of Hacker News get fixed.

This is an anecdote but the latest update forced me to rebuild my Mac from a hard/factory reset. My Dell D6000 on my 2019 MacBook Pro no longer charges the laptop. I've tried.

My Mac's resources were getting gobbled up by an internal process I coudln't terminate and my keychain was borked and I couldn't log in after a reset (to try and get around the resource hogging). Recovery didn't get me any where so I used Recovery over the Internet to do a clean install.

I'm running 10.15.4, no issues as of yet. And this all occurred after the security update. I'm running on the version prior for now but will make sure I've got a good backup and give it another go.

I had a d6000 and had to update the firmware (with a borrowed windows machine) before it would charge my mbp.

Hadn't considered that, I'll give it a go. Thanks!

If anyone comes across this comment, this 100% worked.

I've moved to Linux for 99% of my computing but still use macOS for some audio production work. Catalina is unusable for me personally (most of the software I need just silently crashes) so I disabled the upgrade prompt:

  sudo softwareupdate --ignore "macOS Catalina"
  defaults write com.apple.systempreferences AttentionPrefBundleIDs 0
  killall Dock

Apple should really slow down on major releases of macOS or stop altogether in my opinion. macOS Mojave is a great OS and it's basically feature complete. Just stick with that, introduce bug fixes and security patches as needed and I think people will be happy.

At this point I’m thinking maybe the permissions on my local private key got screwed up. So, I blow away ~/.ssh and recreate all of my keys from a backup

Is that a common thing to do, or any reason why the OP would do that? Doesn't ssh reject your key, saying it does that if there's such a problem? And even if not wouldn't it be advisable to at least look at the permissions; I mean suppose they're not -rw------- or so, wouldn't you want to know that, and also why they are not ok?

> Am I and this one other forum poster just doing something totally bizarre yet the same?

One might uncharitably suggest that using macOS and expecting standard decades-old Unix behaviour is itself bizarre … but that's also true of using Linux with systemd (viz., nohup no longer nohups, or systemd-resolved, or innumerable other broken bits).

It's almost as though no-one cares about quality anymore.

I agree, that’s an annoying bug/feature.

However, there is an amazingly easy workaround, assuming the IP and port don’t change often: create a ~/bin shell script that connects via IP and port, make it executable, and add ~/bin to PATH.

This workaround doesn’t excuse Apple of doing something so egregiously stupid, but it’s so easy that you may as well do it and move on.

Well, the description of this bug is not generally reproducible, so whatever is causing it, it's not as simple as using a high port with a server name.

I tested this specifically on a number of servers that I run with port numbers > 10000, using /usr/bin/ssh on macOS 10.15.4, with and without IP addresses. Nothing broke for me.

Catalina also broke `apropos`. https://apple.stackexchange.com/questions/374025/errors-from...

Catalina seems to have bust Wifi monitor mode on tcpdump on my MacMini 2018, yet it works fine on my Mac Air.

Still not sure if that is my machine, or a general fault - but the lack of monitor and promiscuous mode is playing havoc with IPv6 multicast packets from VMware Fusion VMs.

"I don’t want to end up on Hacker News again bitching about Catalina. I just hope I’ve stuffed this post with enough keywords so that anyone else searching on Google might come across the answer." Ok. And did you actually report it as a bug to Apple?

Whether or not he did, there's absolutely nothing wrong with posting about a problem in case others come across the same. In fact it's damned helpful.

It is time to move to Linux or Windows desktop. Really if you are not hostage of Apple ecosystem then decent desktop is much more reliable in my experience (got old MacBook which is ok too but eg can't connect to old vpn on it)

What’s the backstory to why apples bash has to be different than gnu bash?

I love having the Linux kernel with a nice UI but there are some useful commands that are missing.

There are ways to get them set up but in any case it’s kind of a pain

Wow, just the other day this started happening to me as well with one of my serversfrom my MacBook. It used to work fine, but now only that laptop can’t connect to it. iIt’s on a high port too.

I'm using ssh to a named server online using a port >8192 from a 10.15.4 machine right now.

Have had no issues with it at all.

>I’m not even going to go into it. I don’t want to end up on Hacker News again bitching about Catalina.

Whoops

> Next, I ssh into a different server and then hop to the problematic one. It connects without any trouble. At this point I’m thinking maybe the permissions on my local private key got screwed up. So, I blow away ~/.ssh and recreate all of my keys from a backup. Still can’t login.

Someone should have paid more attention to that verbose SSH output first.

Are you talking about the new apple feature?... non working ssh? It's all the rage.

In my experience the breakage is with IPv6. Try forcing IPv4.

'ssh -4 <hostname>'

Thank god I did not upgrade. Software upgrades are the best way to waste time.

They're the best way to stay secured and receive new features.

The issue with them is lack of testing before deploying them.

I think proprietary software and staying secure are contradictory ideas.

You can just get the security patches for Mojave. You are trying to make it sound like of you are on previous version there are no security patches. Factually untrue.

Am I the only one who thought this article was about Apache Tomcat?

If I'm still on Sierra, which version should I upgrade to?

I have a MacBook Pro (Retina, 13-inch, Mid 2014) and Mojave runs very well on it. I have no plans to upgrade to Catalina.

Looks like the post was just deleted.

laughs in Linux

Please don't do this here.

“I’m not even going to go into it. I don’t want to end up on Hacker News again bitching about Catalina.”

Welp.

With this foresight he sadly didn't enable CloudFront and the website went down.

Yeah, it's a $5/month DigitalOcean box with only my blog on it and nothing else. All assets come off a CDN and Varnish is sitting in front of WP, but looks like that still wasn't enough this time. It worked fine for my previous two HN'ings earlier this year.

I am still able to access it. So I don't think it's doing so bad.

Which is more common? Someone says "I don't want to end up on ___ news again" and they really mean they don't? Or they say that and they really want to? It feels like the old "Please, Br'er Fox, don't upvote this post. I don't want anyone to see it."

(I'm about 80% kidding but amused. Lockdown.)

Sadly, I mean it. Don't want to go through this again. Taking down the post for now.

I wouldn’t trip at all. There are great folks on HN, and you can tell from comments who are the sour ones just because. I personally enjoy reading your posts. I worked at Apple for a long time, loved it. I still use their products and want to understand what bugs exist. I use SSH daily for many things.

The post was excellent, I’ve been locked away from Catalina updates now that my work mac is my primary Mac so I like keeping abreast of all the little gotchas I might be hitting. And it’s a great debugging chain for something that is truly weird. Sorry you’ve had bad past experiences here, the quote gave me quite a chuckle

Go through what? Why would you take down the post as it’s probably useful to others?

> it’s probably useful to others?

That's my intent. I'll put it back online for others to find once the fuss dies down.

Hacker News can be a cesspool at times.

One important thing to always remember is that unless someone posted their article to Hacker News themselves they might have had absolutely no expectation that a huge audience was about to descend and dissect everything they wrote. They might have just been talking off the cuff, mentally noodling around or even just using the process of writing stuff down as a means to sort their thoughts. Far too often HN commenters work from the assumption that an author is intending to make A Big Point and very uncharitably deconstruct every sentence the author wrote.

It's only a matter of time before we see a reply along the lines of "OBVIOUSLY 10.15.4 did NOT break SSH, the author just didn't do X Y and Z to fix a very OBVIOUS mistake in their SSH config".

> Why would you take down the post as it’s probably useful to others?

More broadly, Tyler doesn't owe the world anything in this regard. If he wants to post it, cool. If he wants to remove it, cool.

Wow. Yes. Sorry to see you take it down. I liked the post and was just amused by the one line.

I was wondering if it was because you were afraid of retaliation by Apple.

understood. in line to respect your wishes I encourage everyone to flag this post.

dang, join in if you must.

>I don’t want to end up on Hacker News again bitching about Catalina.

Ends up on Hacker News again bitching about Catalina.

I don't wanna pay taxes.

Paying taxes anyway

>I don’t want to end up on Hacker News again bitching about Catalina.

To be fair, full context tempers that sentence:

> I’m not even going to go into it. I don’t want to end up on Hacker News again bitching about Catalina. I just hope I’ve stuffed this post with enough keywords so that anyone else searching on Google might come across the answer.

> I changed the server to listen on standard port 22 and tried connecting via the hostname once again. Holy crap, it worked.

Shocking!

Granted high ports shouldn't be broken, but running SSH on a non-standard port is security (read obscurity) theater at best.

There's really not much benefit, unless you need multiple sshds on the same IP, but at that point I'd question the sanity of the approach.

Security through obscurity shouldn’t be used as an edict that something is not effective. You are talking about the fact that it doesn’t increase the security of the protocol itself or the passphrases/keys used. This is true. However, there are tons of bots out there that scan 22 and try to exploit common logins. There are presumably quite a few less that are port scanning every machine for every possible high port and attempting to handshake ssh and then try logins. Do you disagree with that? If not, this is not security through obscurity, it has a very real impact on the volume of bots that have knowledge that this service is running and are actively exploiting it. It’s just a different type of security, it’s discoverability of the service.

Here’s another example. Say you have a web server running that is only for internal employee use. But you want to expose it externally so that they can reach it without a VPN. Even if you follow proper security protocols, why would you not turn off search engine indexing on this page, and limit the pages that link to it? It will not increase the inherent security of the protocol or the user accounts, but it will drastically lower the # of bots using up CPU and iptables entries trying to fail2ban or blacklist them.

Security is a spectrum and you want to have defense in depth. Moving ssh to a nonstandard port is a security best practice and you shouldn’t be advising people not to use it. But should they also have good key setting, fail2ban, ip whitelisting/blacklisting, etc? Of course they should.

> Moving ssh to a nonstandard port is a security best practice

For whom? Could you please cite this?

In time, any server with port 22 exposed to the Internet will have a system log with hundreds of failed authentication attempts per minute from IP addresses all over the world. By simply moving it to a high port number, the attempts are rare and troubleshooting is easier without all the noise. And it is script kiddies, typically the login is root, ubuntu, or similar and a password of password, god, other silly things that people actually use.

Or you can just firewall the box?

For me SSH on a standard port get connection attempts and every few minutes. SSH on a non-standard port get connections attempts every few weeks.

Reply on news.ycombinator.com