Hacker News Reader

Comment by gwbas1c | Hacker News Reader

> Access that does not require authentication should never be a crime.

Careful, this could legitimize things like accidental denial of service. Depending on circumstances, even basic scraping could cause problems.

(I need to be vague to avoid violating an NDA.) A major internet site had a URL that went something like somedomain/group?id=xxxxx. It turns out that a simple scraper, that called id=1, id=2, id=3, ect, ect, caused a major problem! This was because rendering these pages required significant resources; so most active pages were kept in RAM. Of course, the scraper tried to read everything.

Of course, no one thought the scraper was malicious in any way!

View on HN

>A major internet site had a URL that went something like somedomain/group?id=xxxxx. It turns out that a simple scraper, that called id=1, id=2, id=3, ect, ect, caused a major problem!

This is a failure on the part of the developers at that "major internet site". Using a guid instead of consecutive IDs, a rate limiter, hell even just a cache...or all of the above. There are lots of solutions here.

You have to take robot scraping and indexing into consideration, and assume people will ignore robots.txt. (Certain bots, i.e. msnbot/bingbot are quite aggressive!)

No, that is a failure of the developer of the scraper. I am definitely pro scraping, but you have to be a good neighbor.

How the hell is the scraper dev supposed to anticipate how poorly-written these particular views are with no backend knowledge? If not an automated scraper, a thundering herd from content gone viral would trigger the same result.

Scraping is not an intended purpose for most websites. Unless the website specifically states that this is an intended function, it is not reasonable to assume so. In fact it may be in violation of the terms and conditions of the given website.

I don't intend people named Steve to access my open site so I can sue all Steve's for their felonious behavior?

If the law assumed that only intended functions are permissible, innovation would be a crime. By definition, innovation is finding new and unforeseen uses for resources.

You both make good points. If you make the law too strict you punish reasonable uses of the website, like scraping a few publicly available pages to help users. If you make it too lenient you permit DOS attacks.

It’s not easy to craft a law that will punish bad behaviour without blocking innovation.

I've done some scraping work -- one of my rules of thumb is to always assume the worst of their site and try to be as gentle as possible.

Oh come on, you're trying to scrape the data out of a black box. You have no idea what their infrastructure is like, and for your purposes, you don't really care.

Of course, some sense is more than welcome, but if my scraper makes one request every 2 sec knocks down your server, it's your fault, not mine.

>> This is a failure on the part of the developers at that "major internet site". Using a guid instead of consecutive IDs, a rate limiter, hell even just a cache...or all of the above. There are lots of solutions here.

You are right, but few organizations are sophisticated.. or wealthy enough to employ all of that. I mean, a couple years ago there was a thing that Google's Docs could be enumerated.

And that's Google, they can obiously afford to get competent people working on that, yet they made a mistake (and who doesn't?).

Fair enough. It still shouldn't become a criminal issue.

>You have to take robot scraping and indexing into consideration, and assume people will ignore robots.txt. (Certain bots, i.e. msnbot/bingbot are quite aggressive!)

Who owns LinkedIn again?

Weev went to jail for exploiting a similar flaw in AT&T's website[0]. They had a page that, when provided an ICC-ID, would return the matching customer's email address. He supplied a range of valid ICC-IDs and scraped the returned addresses. He was eventually convicted[1].

[0]: https://arstechnica.com/gadgets/2010/06/ipad-3g-user-e-mail-...

[1]: https://www.wired.com/2013/03/att-hacker-gets-3-years/

And while Weev totally sucks as a person, IMO, it was wrong for him to be convicted in this case. He was punished from AT&T's negligence.

Although, not purposefully exfiltrating loads of data after you've found a vulnerability is like, ethical reporting 101.

Otherwise you get situations like Uber paying out an enormous "bug bounty" totally-not-in-exchange for having their stolen data destroyed. If that person had simply pointed out that they had credentials published in a public repository, how much would they have been paid? Probably somewhere within an order of magnitude of the program's stated maximum payout.

Punk test. Advocacy groups are way less likely to want to turn your case into a test case if you are a racist asshole.

Are you suggesting that someone should do time for running a script that happen to stumble on one of your bugs?

If the activity caused actual damages and was outside the scope of normal usage? Yes.

You're still culpable if your actions break your neighbor's window, even if it was accidentally while you were opening it.

Unless I'm missing something, you're proposing criminal penalties for tort liabilities.

Yes, if my crappy software costs you money by knocking your site offline by accident, I should make you whole.

I think it has to be something substantially more impactful, clearly intentionally malicious, or in some other way much worse than aggressive timeouts before we start thinking criminal penalties.

either I read it wrong the first time as well, or he edited it, but reading it now it clearly says "while opening it" which is a criminal act, in context.

No, when I responded it read "be held liable." I must've commented while it was being edited.

I think you could be sued for damages, but that's not the same as a criminal case.

Say a business publishes a phone number and they typically get X calls per day.

After doing something that pisses a lot of people off, they start getting 1000X calls per day on the same number, almost all complaints.

This cases actual damages (no "normal" customers can get through) and is also clearly outside the scope of "normal" usage.

Do you think the same rules apply?

Yeah, that's not how the comment read when I responded. It said "be held liable," not "do time."

I must've responded while he was editing it, and I didn't catch the change.

Honestly, we all know the wild west mentality of the internet (yes, it is post national as in 'above the law') and therefore, everyone should assume attacks like that and build defenses against them. Building a service which could be brought to a 'major problem' with simple requests leading to high server loads is just negligent. What would that site do if someone actually wanted to attack it?!?

I am not saying that the trouble with the law enforcement in the internet is a neither a good nor a bad thing. Actually, it depends and in the 'real' world I am pretty happy that the law enforcement works quite good where I live. I think the thin line is somewhere where I start to fear my own governments more than the bad guys (while not having any evil intentions or plans at all).

> the wild west mentality of the internet (yes, it is post national as in 'above the law')

No. It's only been ahead of most laws for a while, as all frontiers are while they remain frontiers. But all frontiers eventually close, and laws catch up with them as they do so. That is what we're seeing now, and have been for a decade or more.

Honestly that just means the website sucked and it went down because it sucked. Making it not suck is the solution, persecuting the people who stumbled into your suckiness is not.

> Careful, this could legitimize things like accidental denial of service.

Are you saying that the writers of a bot that causes accidental issues with a site due to poor development standards on that site should spend years in prison with a federal felony conviction?

A while ago I was introduced to a client whose site "was the target of hackers that were deleting all of the content from the CMS". Here's what I discovered:

- the password verification form to access the admin area did the verification check in JavaScript, not on the backend. So if you have JS disabled and click "Submit" on the Admin login form, you're into the admin area.

- the "delete" button in the admin area was implemented as an <a href=...> that simply did a GET request (violating the idempotent nature of GET requests).

Looking at the logs, it was pretty clear who the "hacker" was: Google. They'd come, follow all of the links, make their way into the admin site, and follow all of the delete content links.

I consider the work that the original developers did to be grossly negligent, and I certainly don't fault Google for anything.

Accidental denials of service are indeed a common occurrence. By the way, it's "etc" from latin et cetera - I assume you didn't want to refer to electro-convulsive therapy :-)

As any fule kno, this is how Molesworth writes, ect ect ect.

In general, the law is capable of dealing with this kind of issue - it can look at the intent of the owner of the service.

cf. for example the law on trade secrets. If you take "reasonable steps" to safeguard the secret, and impose NDAs on the people you do grant access, then courts will punish competitors who steal them, even if your security happens to suck.

> Careful, this could legitimize things like accidental denial of service. Depending on circumstances, even basic scraping could cause problems.

I have to "deal" with that problem every day. Misconfigured scrapers are dealt with by apache as are idiots who try to DoS the site (an intelligent attack still needs manual intervention, though).

Reply on news.ycombinator.com