Comment by beat | Hacker News Reader

Comment by beat | Hacker News Reader

Some years ago, I worked for a big box retailer on their public-facing catalog api. It was a pretty exotic thing when it came out, as that data was all trade-secretive - which was dumb, because then they just had web scrapers brutalizing their front end to get the data that was public anyway.

So on the back end, the api collected data from many different sources within this very large company, and put it in nice api form. This was technically cached data, and could get stale, but really, most of it was sufficiently up-to-date most of the time.

Turned out the biggest customer for this data was the company itself. Lots of individual projects, needing some obscure piece of data or another, could get it from the api, rather than having to hunt for whatever department actually owned the data, and working out access arrangements on a piecemeal basis. I'm sure it saved countless thousands of hours of work, and greatly improved both quality and time to market for projects.

Looks like maybe IKEA could use something like that...

That's funny, for a few years I worked for an organization that did the front end crawling / brutalizing of most of the major retailers for their data. Afaik most still do not intentionally provide any data publicly and many spend extravagant amounts on anti-crawling tech to prevent it, resulting in an expensive arms race over ostensibly public data.

Yeah, it's just dumb, pseudo-paranoid thinking. Crawlers are going to get it anyway, and it's publicly available, so why not make it easier for everyone?

And like I said, our biggest customer was our own company. It saved an incredible amount of hassle.

I always wondered what those companies think they'll gain by hiding the products they sell.

If you put something on a public site it can be harvested, and there's really no stopping that. It's like video DRM: you can spend billions on the tech, but ultimately, if nothing else, anyone can still aim any camera at the screen and make a copy.

There's only two reasons I can think of that companies would bother with this:

1) they are ignorant or delusional about it and think they can beat the crawler (which could be a third party making money by convincing them they can do it)

2) It forces the crawler to spend more money chasing their blocking

I'm not sure what the benefit of 2 would be, though, unless you could make them spend so much money that their entire operation is unprofitable.

Aside, as a developer, it might actually be entertaining for a while to work on (2) - constantly changing strategy to defeat scrapers - but I can also see getting burned out pretty quickly.

>Aside, as a developer, it might actually be entertaining for a while to work on (2) - constantly changing strategy to defeat scrapers - but I can also see getting burned out pretty quickly.

Well if you replace 'scrapers' with 'advertisements', there are plenty of worthwhile OSS projects that might welcome help :)

Making that sort of business unprofitable might not be a super realistic goal, though.

This is very reminiscent of the Bezos API memo at Amazon a while back...

For the interested: https://homepages.dcc.ufmg.br/~mtov/pmcc/modularization.pdf

Absolutely - most bigger companies have silos and can make excellent use of things like this.

Reply on news.ycombinator.com