Google Play Store in Numbers. Open Source Crawler for Mobile Apps Data | Hacker News Reader

Google Play Store in Numbers. Open Source Crawler for Mobile Apps Data

github.com

39 pointsmarcellolins15 comments

The robots.txt forbids any user agent from crawling the pages this crawler crawls. It's unwise to do this in the open.

He's not exposing his personal server IP addresses. He's doing it in private, and opensourcing the code for others to do the same.

robots.txt are laughably unenforced policies, and are disrespected every day.

In practice robots.txt is just a suggestion; even Google doesn't fully respect it.

Google specifically recommended using robots.txt in Copiepresse v. Google. I haven't noticed them disobeying robots.txt on my own sites.

I've seen several blog posts on exceptional cases before, but the only case I've personally witnessed is how the crawl-delay directive is ignored. Granted you can set it if you're a webmaster tools user, but even then the setting disables itself after a few months.

robots.txt is a nice way to make spiders ignore to index content, but doesn't imply not to access those

That is not how robots.txt has been interpreted by any search engine. Robots.txt tells them not to access certain pages and says nothing about whether they should index those pages. (It's possible to index a page based solely on what links to it.) If you want your content crawled but not indexed, you should use the robots meta tag with a value of NOINDEX.

Shameless self-promotion: MixRank's API has all of this and much more available, including Appstore. Email in profile if you're interested.

Its a shame to think people seeing this as a "self promotion", when its not. What i am promoting is the data that this crawler obtained, and all the insights people can get for free, just by using it. As opposed to an API, that will limit either the access for the data or the way you can access it, acessing the raw data on a database directly has no limits.

Anyway, thanks for the reference, i will make sure to add it to the project's wiki as a related link.

i don't think he's accusing you of self promotion, he's calling out his own recommendation for mixpanel as being self-promotion. Smilliken is "co-founder of mixrank" according to his profile.

Woah, now i feel bad. Sorry about that Smiliken : (

I don't have time to review the code right now, but feel free to reach out with any questions (email in profile). Thanks for open-sourcing this!

Link to our api docs: https://mixrank.com/api/documentation#/playstore/apps/{pname...

Okay... So, having been down this road and beyond already, what do you think of this guy's approach which is nice and visible for inspection?

Might help not only him but some of us learn further, us tinkerers, those who might have a faint curiosity in c# etc if you shared. Then you'd be able to plug away shame-free.

Applause Analytics (www.applause.com/mobile-analytics) provides some of this data plus data on the iTunes app store.

TY for the reference. I know that apps annie also provide some really cool analytics. Regarding the itunes data, i also wrote a crawler for that, which is on github aswell

Reply on news.ycombinator.com