Hacker News Reader

Comment by mnmkng | Hacker News Reader

I think of web scraping as nothing less than automation of human work. There really is no hacking or unauthorized access involved. It's either me, using a browser, to read something publicly accessible on the internet, or it's my computer, which I've programmed to read those things for me, so I can focus on creative work or spend time with friends and family.

This applies not only to personal life, but businesses as well. Instead of hiring 100 interns to collect some data manually, you hire one programmer to automate that data collection process.

I think that the ability to automate tasks on the internet is absolutely crucial to further development of our society and limiting it in any way will be detrimental to the world as a whole. The amount of information these days is so vast, that no human labor force could possibly analyze that information and use it to drive our progress.

Disclaimer: We run a web scraping platform (https://apify.com)

View on HN

> I think of web scraping as nothing less than automation of human work.

I agree that scraping should be allowed. We wouldn't have Google otherwise.

However, there's something to be said for the fact that some activities are okay at a small scale but become problematic at a large scale—particularly the scale which becomes possible when a task is automated.

For instance, I recently bought an expensive camera and have been having fun walking around the city and taking interesting photos. Many of the photos have people in them. I don't think there's any harm in this.

I store my photos in Aperture, which automatically performs facial recognition on everything in my library. Perhaps some day, if I take enough pictures, Aperture will notice that the same stranger is present in two completely different images. That might be kind of cool—I can see myself fancifully trying to imagine this person's life story. I don't think there would be any harm in that, either.

However, if I aggregated millions of photos from different sources, and used them to track people's movements across the city, that would clearly be a huge problem! Sure, it would be merely automating the work a human could do, but the scale just changes everything!

Can the case be made that recording in public is a right (as if always should be) but trying to track where everyone is at every point of time is stalking at a mass scale which should be illegal as stalking on a one to one scale is illegal?

For reference this is what one site (findlaw) has given as what constitutes the crime of stalking:

The crime of stalking can be simply described as the unwanted pursuit of another person. Examples of this type of behavior includes following a person, appearing at a person's home or place of business, making harassing phone calls, leaving written messages or objects, or vandalizing a person's property.

This sentiment exactly.

Sure the information is free. But to find my property records, phone number, and "aggregate them" like Spokeo, fastpeoplefinder and similar sites, is akin to digital stalking IMO

Maybe I am wrong and my take is too simplistic/naive (as I admit, I haven't thought too much about this topic), but imo aggregating publicly available data shouldn't count as stalking. Acting on that data in a way that could be constituted as "stalking" definitely is real stalking though and should count as a crime.

For example, if someone's ex (who was explicitly warned that he was unwelcome) continues showing up at the doorsteps because they remembered your address from a long time ago, then it is stalking. Them knowing the address isn't.

In light of this, I don't see how it matters whether they remembered the address from past experiences or just found it through a website that aggregates publicly available info. As long as the data was obtained legally and without breaking any other harassment clauses, why would just the knowledge of something be a crime?

I see it just like firearms. Having a firearm (in a lot of US jurisdictions) is not a crime, as long as it was obtained legally. Doing harmful actions with it (such as threatening people or shooting someone who didn't pose a threat to your life) is a crime. Having a firearm feels like just having data, in this scenario. As long as you don't use that data for criminal actions, why would just the potential of you being able to do something criminal with that data is a crime?

> The crime of stalking can be simply described as the unwanted pursuit of another person. Examples of this type of behavior includes following a person, appearing at a person's home or place of business, making harassing phone calls, leaving written messages or objects, or vandalizing a person's property.

Except for the last point, that sounds like ads :)

ads vandalize property by causing unwanted images to appear on and unwanted sounds to emit from personal property

Vandalism usually means permanent damage. You can call the police on your neighbor for playing loud music at 3 am, but I don't think it would be correct to accuse them of vandalizing your property.

Is a highway billboard vandalism when the owner of that billboard is allowing the ad?

Or are you referencing computers with ads in them (websites, etc.)? By that definition, anything disproving a conspiracy theorist is vandalism to them. That’s not how that works.

The case I thought of is how public records are great, but the way Spokeo and friends scrape them, post them, and sell them isn't.

Right! Ensuring a process has the right amount of friction can actually be super important, to ensure it's available but isn't used wantonly.

This is a great way to phrase it. Friction really is essential for so many things where we're gradually loosing it.

I totally agree with the first part where you say it's fine to scrape. It is just an automation.

But you don't mention what you do with this data after. 1. You store it somewhere (not in your brain) 2. You extract value out of it (directly or indirectly)

That's why I understand why it is a problem for those who publish this data.

One thing I always try to say to everyone who argues about web scraping: web scraping is not a problem, problem is what you do with the information you scraped.

Disclaimer: We crawl the web for news (https://newscatcherapi.com/)

> That's why I understand why it is a problem for those who publish this data.

I still don't understand why it's a problem for those who publish the data.

If the "data" is facts (e.g. lists of values associated with objects, like the colors available on automobiles), this data is not protected by copyright, there is no remedy if it's republished, and if a business relies on limiting access to this data, that business needs a new business plan. Charge for access to cover the costs of obtaining and organizing the data, but understand that clients are legally allowed to make copies and use them however they see fit, even if that impacts the supplier's ability to charge for access.

If the "data" is prose, it's under copyright and republishing without a license has remedies under law. Maintaining copies of articles for the purpose of processing them to obtain other data (e.g. how many nouns? what adjectives are near those nouns? etc...) isn't protected.

Databases are actually protected under some copyright laws in some countries.

For copying the database wholesale, yes. But the individual records, formatted on a web page, and scraped out by a bot is not copying the database files or its schema wholesale. They're still individual facts.

So if somebody compiles the data, charges an N amount for access, it's fine that some entity buys a one time access and then releases it for free?

This is usually explicitly prohibited.

If it's not prohibited for some reason, e.g. because the term of the prohibition, such as copyright, has expired, why not.

Are news just facts? Or do they also incorporate the bias of the author?

Sure, you could argue that news are facts, and that they should be free from copyright. But that would only work if your news is just bullet points of the factual stuff. But if any element of bias is present, it become a distinct property of the author, hence it should be copyrightable.

Of course, with today's tech, it's easier to remove the bias element from the factual data.

I'm agree, 100%.

I think of placing a zillion cameras in public places as nothing less than automation of human work. Instead of hiring a million agents to record what the population is doing, you hire one programmer to automate that data collection process.

Totally different? Yes, but it shows that scale can affect whether something is OK to do or not, at least for some (I accept that people watching the streets can be useful, but also have my doubts about doing that at scale, whether by cameras or by hiring a million agents)

With both cameras and copy-pasting web content, there’s the issue what you do with the data. If, for example, I start scraping all the articles on a newspaper’s web site, publish them on a web site, adding my own ads, most people would think that shouldn’t be legal.

If you agree, we’re now haggling over the price (https://quoteinvestigator.com/2012/03/07/haggling/). That’s where things get difficult, but I think the entire spectrum from white (scraping for this goal is fine) via grey to black exists.

If, for example, I start scraping all the articles on a newspaper’s web site, publish them on a web site, adding my own ads, most people would think that shouldn’t be legal.

Publishing? Ads?

Republishing, duh.

OK, let's reformulate that: would people be opposed to just scrapping or to the illegal republishing and adding ads?

Another similar trick: "If someone will block ads and then murder the publicist and burn his house, most people would find that it should be illegal".

Advertising isn't some zero-sum game that suddenly makes you lose because you happened to see it. Republishing is - the original publisher loses out on the value of his publication because that means either reduced ad revenue or reduced subscription. I'm guilty myself of reading republished articles, but it is a loss for the publisher.

I hope you realize that what you wrote isn't related to my point at all. It's fine, but why put that as an answer to my comment is beyond me.

Weird take, but I see this sort of thing show up a lot, so let's give it a shot:

> If, for example, I start scraping all the articles on a newspaper’s web site, publish them on a web site, adding my own ads

Using a Q-Tip to murder someone is illegal. We don't have special laws against it—it's just that murder is illegal, we have criminal punishment for murder, and the ordinary machinery of the courts and the law is sufficient to handle any instances of murder by Q-Tip. Because it being done with a Q-Tip is the least important part.

If you're scraping and republishing someone else's content, it's not the scraping part that's the problem.

While I have substantial agreement with your point of view, when web content is substantially ad-funded, automated scraping effectively bypasses the “payment”.

I say this as someone who runs an adblocker, installs them for family, and doesn’t derive income from hosting ads, so I’m not pro-ad; I just realize that while it’s not a crime for me to dump the whole bowl of waiting room candies into my backpack, that’s going to be frowned upon.

This is orthogonal to the purpose of scraping.

If I run a search engine which potentially links real human eyes back to you, then should I pay the "ad" toll as well?

I don't believe so.

I do believe there is an agreeable middle-ground, but Google walked away from that conversation years ago.

Advertisements are harmful to every layer of society, mostly because they prey on you to instill desires that you probably wouldn't have had on your own (it being the case that this is their entire added value proposition). They should not be tolerated, and the fact that they ever were is a travesty.

That being the case, any argument founded on "but advertisements" does not hold water.

It's not founded on "but advertisements" it's founded on "but paying for content". The fact that websites are developed by people (wages), hosted on infrastructure (hardware renting) and require countless other jobs is somehow magically forgotten in these discussions.

It's not up to me or anyone consuming content to help with figuring out the right way to pay for it. What is up to me is the choice to not be potentially exposed to malware any time I go to read the news.

While you may have a point, that point just doesn't matter within the context that we're talking about. Making companies money is not my responsibility.

It absolutely is your responsibility if you are consuming the content.

Consuming the content means agreeing to the premise that the content is paid for using ads.

If you disagree with said premise you may happily browse another website.

Malware is already illegal to install and you are free to sue the website for damages.

I don't like ads either but trying to justify that their should be a choice of browsing a website without the ads it hosts is ludicrous.

> Consuming the content means agreeing to the premise that the content is paid for using ads.

It is in fact not this way, because the content arrives with or without the ads. In the EU EULAs (the dystopian construct that you'd expect to enforce that bit of lunacy) that purport to apply to content you have already accessed are not legally valid. Leaving the legal interpretation aside, me doing one thing doesn't mean consent for something else. Believing otherwise is both unethical and amoral, stances I don't hugely feel like interacting with.

> It is in fact not this way, because the content arrives with or without the ads

But that's simply a technical implementation detail.

If the articles you read had first party ads or videos with embedded ads in them, that choice wouldn't exist like it already doesn't when you watch TV.

> Leaving the legal interpretation aside, me doing one thing doesn't mean consent for something else. Believing otherwise is both unethical and amoral, stances I don't hugely feel like interacting with.

Yes and you stealing content (consuming without paying for it) is somehow moral?

It's easy to find all kinds of free content, should it be music, movies or series but lets not kid ourselves in thinking that it is some great human right to have access to things we didn't pay for.

When you dump the bowl of candies into your backpack, there are no more candies left for anyone else. On the other hand, when you scrape, all the original data and its underlying use case remain just as they are. You simply having scraped took nothing away from anyone.

I wrote a scraper for myself only to keep an eye on several used truck websites. I was able to do this for craigslist, so I could run it every couple days and check the nearest 75 cities, the list of truck ads, filtering on/out certain keywords, and the pages for the interesting trucks, making a report for myself to help find the vehicle I want.

I could not do this for any of other several large sites advertising used trucks, like commercialtrucker.com, truckpaper.com, and machinio.com. I kept bumping into artificial javascript and captcha limitations that were not worth my time to try to work around.

The thing is that I will probably find what I want on a craigslist site that I can scrape, I'm getting a lot of great info from them, and I'm not going to bother with any of the ad-based sites, it takes to olong to run all the manual searches. They have effectively done a disservice to their (presumably paying) customers who want to sell their vehicles.

If AutoTempest covers your search criteria, I’ve found them to be excellent to do repetitive searches. Somewhat fortunately for you, they suck for craigslist searches, but have great coverage for all the other major sites.

> I think of web scraping as nothing less than automation of human work.

I did this with Python regarding market prices for ETFs, Stocks, and Mutual Funds.

I wrote Python scripts, one of which was to web-scrape current market values, investment distribution, &c for ETFs, stocks, and Mutual Funds for whic I was invested in. I then would have to manually port that into a spreadsheet for "my own special graphs" and such.

I'm sure there are places online which would do this for me if I logged in, entered in all of my information and more, and provide that all for me ... but this story should sound familiar. And due to personal reasons, I've been lagging behind for far too long.

My Point: This information is publicly available and and it serves my purposes. There's no reason why I should not be able to do this. Yes, it's my fault for not using the information [and yes, the information dies with me], but the point is that I automated the gathering of that information for my own self -- and "everyone" else on the planet has that same information.

I never felt like I was illegal doing this. I'm happy to know if there is a "save" way to do this type of aggregate situation otherwise.

> There really is no hacking or unauthorized access involved.

I'm not sure about that. If I had a phone book full of phone numbers (those heavy ones from 80s), would calling every number in that book to find the one I'm looking for be legal/ethical?

P.S: I agree that "Web Scraping Is Vital to Democracy".

> If I had a phone book full of phone numbers (those heavy ones from 80s), would calling every number in that book to find the one I'm looking for be legal/ethical?

Sure, why not?

I have no idea, I've never done it.

I can tell you haven't cold called ever in your life.

Reply on news.ycombinator.com