Comment by Animats | Hacker News Reader

Comment by Animats | Hacker News Reader

Parent Full thread

"Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems."

Yes. My favorite was a Coyote Point load balancer bug. If the last character of the HTTP options is "m", the connection will not get past the load balancer.[1] I found this because a web crawler was having trouble with one site. Fortunately, I knew someone with their own Coyote Point load balancer, and was able to establish that the connection went into the load balancer and never came out.

The load balancer has a big file of rules which contain regular expressions. Somewhere, I think there's a "\m" where they meant "\n". Reporting this to the vendor, along with a Python program to demonstrate the problem, was of course futile; they suggested "upgrading the software". I demonstrated that the bug existed on their own load balancer on their own site. I finally added a completely useless field to the HTTP header so that the last character was not "m".

[1] https://www.webmasterworld.com/webmaster_hardware/3312997.ht...

Tell them they've violated their contract and you are migrating from them ASAP.

That doesn't work when it's software on your client's computer. Remember when all the web devs revolted and ditched IE6 back in 2001, because MS was just taking the Mickey? Yeah, me neither.

Right. I didn't have a Coyote Point unit. Many other sites did, and they all appeared to be down to my web crawler until I figured out the problem.

(Current web crawler problem: sites that won't let you read their robots.txt file if they don't like your user-agent string.)

>sites that won't let you read their robots.txt file if they don't like your user-agent string

That's hilarious. So do you use borrow a browser's user-agent or do you ignore the robots.txt?

I'd assume they want you to crawl their website. When they say you are ignoring their robots.txt file, tell them you actively prevented you from seeing it and you could only assume that mean they WANTED to be crawled.

That would get them to fix the issue pretty quickly :-)

No `robots.txt` indeed means weapons free, but crawler gleans useful info from it, e.g. which areas of the site are dynamically generated.

More likely the site is trying to serve custom versions of `robots.txt` to different bots, with good intentions, and the code is buggy.

The strict interpretation is that if "robots.txt" returns 403 Forbidden, it's interpreted as "deny all". That's what the Python library does. We list those sites as "Blocked".

> upgrading the software

they told you this because they probably fixed the bug in a newer release. what response were you expecting, exactly? to dive in with a hex editor?

for what it's worth we run into the same issues with our network devices, it's a pain in the ass, and that's why we're shifting to SDN.

Proper response would be something like

"You've obviously put a lot of time in to this. Thanks. We actually had found that bug 4 months ago in R1892, and it's patch in R1899. Please upgrade, or let us know that you're, in fact, using R1899 and still experiencing the issue (it's issue #847 on our bug tracker at foobugz.xyz)"

Or maybe...

"You've obviously put a lot of time in to this. Thanks. We actually had found that bug 4 months ago in R1892, and we're planning on a maintenance release next quarter. If you'd like to test out our upgrade version, please contact jaz@balanceco.com and let them know you want to be in the testing group - he'll get you the appropriate documentation and code. Thanks for helping fix and test these issues!"

right, so in other words, a software upgrade.

In other words, acknowledge that someone did some work, and actually ask if they're using the latest version, and vitally - acknowledge that they read the correspondence and that upgrading will address the issue.

"Update your software" is often a BS answer that can introduce many other issues or breakages in production systems. Without a clean "downgrade" path (which many companies don't provide) you run a risk of introducing more problems.

i think you're misunderstanding me. i'm agreeing with you - network device vendors are universally terrible across the board, and that's why nobody should count on them unless they absolutely have to.

in other words, what i'm saying is to expect that kind of response is naive, and that you should seek alternatives.

Reply on news.ycombinator.com