Comment by seanhunter | Hacker News Reader

Comment by seanhunter | Hacker News Reader

Once upon a time around 2001 or so I used to have a static line at home and host some stuff on my home linux box. A windows NT update had meant a lot of them had enabled this optimistic encryption thing where windows boxes would try to connect to a certain port and negotiate an s/wan before doing TCP traffic. I was used to seeing this traffic a lot on my firewall so no big deal. However there was one machine in particular that was really obnoxious. It would try to connect every few seconds and would just not quit.

I tried to contact the admin of the box (yeah that’s what people used to do) and got nowhere. Eventually I sent a message saying “hey I see your machine trying to connect every few seconds on port <whatever it is>. I’m just sending a heads up that we’re starting a new service on that port and I want to make sure it doesn’t cause you any problems.”

Of course I didn’t hear back. Then I set up a server on that port that basically read from /dev/urandom, set TCP_NODELAY and a few other flags and pushed out random gibberish as fast as possible. I figured the clients of this service might not want their strings of randomness to be null-terminated so I thoughtfully removed any nulls that might otherwise naturally occur. The misconfigured NT box connected, drank 5 seconds or so worth of randomness, then disappeared. Then 5 minutes later, reappeared, connected, took its buffer overflow medicine and disappeared again. And this pattern then continued for a few weeks until the box disappeared from the internet completely.

I like to imagine that some admin was just sitting there scratching his head wondering why his NT box kept rebooting.

The lesson for any programmers reading this is to always set an upper limit for how much data you accept from someone else. Every request should have both a timeout and a limit on the amounts of data it will consume.

That doesn't necessarily need to be in the request itself.

You can also limit the wider process or system your request is part of.

While that is true, I recommend on the request anyway, because it makes it abundantly clear to the programmer that requests can fail, and failure needs to be handled somehow – even if it's by killing and restarting the process.

I second this: depending on the context, there might be a more graceful way of handling a response that's too long then crashing the process.

Though the issue with ‘too many byte’ limits is that this tends to cause outages later then time has passed and now whatever the common size was is now ‘tiny’, like if you’re dealing with images, etc.

Time limits tend to also defacto limit size, if bandwidth is somewhat constrained.

Deliberately denying service in one user flow because technology has evolved is much better than accidentally denying service to everyone because some part of the system misbehaved.

Timeouts and size limits are trivial to update as legitimate need is discovered.

Oh man, I wish I could share some outage postmortems with you.

Practically speaking, putting an arbitrary size limit somewhere is like putting yet-another-ssl-cert-that-needs-to-be-renewed in some critical system. It will eventually cause an outage you aren’t expecting.

Will there be a plausible someone to blame? Of course. Realistically, it was also inevitable someone would forget and run right into it.

Time limits tend to not have this issue, for various reasons.

But not putting the limits, leaves the door open to a different class of outages in the form of buffer overflows, that additionally can also pose a security risk as could be exploitable by an attacker. maybe this issue would be better solved at the protocol level, but in the meantime size limit it is.

Nah, just OOM. Yes, there does need to be a limit somewhere - it just doesn’t need to be arbitrary, but based on some processing limit, and ideally will adapt as say memory footprint gets larger.

> putting yet-another-ssl-cert-that-needs-to-be-renewed in some critical system

I found a fix for this some years back:

    openssl req -x509 -days 36500

> Practically speaking, putting an arbitrary size limit somewhere is like putting yet-another-ssl-cert-that-needs-to-be-renewed in some critical system. It will eventually cause an outage you aren’t expecting.

No, not at all. A TLS cert that expires takes the whole thing down for everyone. A size limit takes one operation down for one user.

That's a lead into one of my testing strategies. It's easy to set the timeouts too short randomly, the buffer size too small. Use that to make errors happen and see what the system does. Does it hiccup and keep going or does it fall on it's face?

Then you kill your service which might also be serving legitimate users.

It depends on how you set things up.

Eg if you fork for every request, that process only serves that one user. Or if you can restart fast enough.

I'm mostly inspired by Erlang here.

Fork at every request isn't going to make a fast server.

As a former boss used to say: "Unlimited is a bad idea."

Around the same time, or maybe even earlier, some random company sent me a junk fax every Friday. Multiple polite voicemails to their office number were ignored, so I made a 100-page PDF where every page was a large black rectangle, and used one of the new-fangled email-to-fax gateways to send it to them. Within the hour, I got an irate call. The faxes stopped.

Circa 1997 a coworker lamented that he had signed up for some email list, and attempts to unsubscribe weren’t working (more of a manual thing, IIRC). I made the suggestion to set up a cronjob to run hourly, to send an email request to be unsubscribed. It would source a text file containing the request to be unsubscribed. And with each iteration, it would duplicate the text from the file, effectively a geometric progression. The list owner responded about a week or so later, rather urgently requesting that my coworker cut it out, saying that he would remove him from the list. Apparently the list owner had been away on vacation the entire time.

I enjoyed reading this, thank you for sharing. When you say you tried to contact the admin of the box and that this was common back then, how would you typically find the contact info for an arbitrary client's admin?

tech-c / abuse addresses were commonly available on whois.

You can also find out who owns a general group of IP addresses, and at the time they would often assist you in further pinpointing who is responsible for a particular address.

Back then things like postmaster@theirdomain and webmaster@theirdomain were read by actual people. Also the whois command often worked.

I work for one of the largest Swiss ISPs, and these mailboxes are still to this day read by actual people (me included), so it's sometimes worthwhile even today.

I setup a new mail server with Stalwart and have been getting automated mails to my postmaster address (security treat results mostly).

Pretty neat.

I tried to contact Hetzner and others about customers scanning my ports. Nobody cares about that. I took issue when I kept getting firewall alerts for port scans on open Plex ports.

I went down a crazy rabbit hole and found a bunch of domains that were random parts of street addresses. Obviously created automatically and they were purposely trying to make it harder to find related domains.

A responsible domain owner still will read them. My own postmaster is a catch-all for all my domains, such that typos in the username still get caught. Has proven to be invaluable with the family domain, where harried medical staff make mistakes in setting up accounts for my parents.

I always liked the RP DNS record (https://www.rfc-editor.org/rfc/rfc1183) but no one seems to know about it or use it any more. The only reason my servers don't have one now is because route53 doesn't support it.

tangent

I had a lazy fix for a down detection on my RPi server at home, it was pinging a domain I owned and if it couldn't hit that assumed it wasn't connected to a network/rebooted itself. I let the domain lapse and this RPi kept going down around 5 minutes... thought it was a power fault, then I remembered about that CRON job.

That's why everyone else is lazy and just ping google.com

You’d be surprised to know, that in a majority of the cases of NT installations in that era, providing services, there were very, very few admins around to even notice what was going on. Running services like this on an NT box was done ‘in order to not have to have an admin’, in so many thousands of cases, it cannot be underestimated.

Disclaimer: I put a lot of servers on the Internet in the 90’s/early 2000’s. It was industry-wide standard practice: ‘use NT so you don’t need an admin’.

What was it about NT that made an admin unnecessary? Just marketing?

Didn't get why that WinNT box was connecting to your box. Due to some misconfigured Windows update procedure?

I never found this out, but there was some feature where NT would try to negotiate an encrypted connection to communicate and that’s the port it was connecting on. It’s a long time ago. It’s possible the box had been pwned, and that was command/control for a botnet or something. Lots of internet-facing windows boxes were at the time because MS security was absolutely horrendous at this time.

That’s awesome! Thank you for sharing.

Reply on news.ycombinator.com