Hacker News Reader

Comment by ddxv | Hacker News Reader

Hi, I'm interested in scraping steam too. Do you have the scraper code available open source or one you recommend?

Have you looked over the data that OP is providing here and determined that it doesn't meet your needs?

Generally it's polite to avoid scraping if you can help it, so I'd start by considering whether OP is already providing what you are looking for.

On the other hand you need to be a paid member to download the raw scraped data, so it isn't unreasonable to want to learn how to scrape it instead.

Good idea, let's save $5 and sink dozens of hours into building our own instead

Isn't that the hacker spirit, wanting to put things together yourself? Let's revisit the comment that started this thread:

> Hi, I'm interested in scraping steam too.

I was responding directly to your objection with having to pay for it as the reason to do this, not hacking for the sake of hacking.

Hacking for the sake of hacking VS hacking for the sake of saving money, why does it matter?

Jesus Christ, it's 5 bucks. 5 bucks is not a reason to roll your own version.

Roll your own version because you want to roll your own version, not to save 5 bucks.

Its not $5. It is free or $5/month. The reason OP wants to scrape doesnt matter. His question was reasonable to ask here and could lead to finding out about some open source projects. You have not added anything to the conversation besides being wrong about the price.

you seem to be really hung up on the 5 bucks while at the same time being angry that people are hung up on the 5 bucks, it's just 5 bucks man, if people want to pay it or not doesn't matter, let people hack it away if they want

Let's expand our own skillset by investing time instead of money (paying someone else). Sounds like a reasonable proposition to me.

I wrote a simple scraper for a 'steam game semantic search' app I built a while ago.

It definitely won't fetch all the data that this person does though. It only fetches the current list of games on Steam, their store page information and some reviews for the game.

The code quality probably isn't amazing, but it might give you an idea of how to get started with your own scraper.

https://github.com/Netruk44/steam-embedding-search/blob/main...

Thanks! That's perfect, just want somewhere to get started.

https://steamdb.info/faq/#how-are-we-getting-this-informatio...

I found this explanation from steamdb that points to the various projects and libraries they use to gather all the data they have. It's not a how-to, but it has very useful info.

Reply on news.ycombinator.com