My servers are on AWS addresses. If someone searches for servers (as opposed to routers, phones etc.) AWS might be a preferred address range. No experience whether scan rates depend on the address used.
There are open port scanners that just check what ports are open on which IPs, and there are separate ssh login brute-forcers. Once your machine gets picked up by the former, the latter will pile up.
I have two servers on adjacent IPs, both with ssh listening on a high port. One gets hammered with login attempts and the other does not.
Often with default parameters such as zmap setting ip id to 54321, having tcp initial window at 65535, having no SACK bit set and masscan with no SACK bit either, tcp initial window at 1024, tcp maximum segment size 1460 (which is strange to put below initial window size!), (older versions having fixed src port 61000 or 60000 from documentation examples and no MSS set), all of which are extremly uncommon in legitimate traffic and thus easily identified.
Even those so called "legitimate" scanners (emphasis on the "") seem to use these tools with little or no extra configuration.
With this setup the last time my high-port ssh (key-only) has got an attempt on it was 2023-07-26 (previous intruders get permanently firewalled).
Well, unless the http server or our dns resolver has a remote code execution vulnerability.
So directly I don't see the risk you describe. Of course considering maximum defense in depth you might have point.