HNHacker News
TopNewBestAskShowJobs

IWantAllTheData

16 karma · joined July 15, 2024

submissionscomments
IWantAllTheData··on Show HN: I scraped 3.2B TikTok profiles and 9B posts to build this search engine
Thanks for letting me know. I will choose a better wording.

Eng. Rate (%) is Engagement Rate. Which is a common metric marketers look into when hiring an influencer.

IWantAllTheData··on Show HN: I scraped 3.2B TikTok profiles and 9B posts to build this search engine
All the scrapers are in Python but im slowly transferring it to Golang, i find it much faster.

The website is NextJS.

IWantAllTheData··on Show HN: I scraped 3.2B TikTok profiles and 9B posts to build this search engine
So many useful stuffs in the comments. Thank you, will give this a read as well.
IWantAllTheData··on Show HN: I scraped 3.2B TikTok profiles and 9B posts to build this search engine
Thank you. Yes proxy bill is quite high!

Yes you are right, the ID enumeration method doesn't work with TikTok.

IWantAllTheData··on Show HN: I scraped 3.2B TikTok profiles and 9B posts to build this search engine
I appreciate the positive comment. Thanks!

Fun fact: I did the UI Design, frontend/backend web development, database, servers, scraping and everything else by myself. It took me few months but i guess its worth it.

IWantAllTheData··on Show HN: I scraped 3.2B TikTok profiles and 9B posts to build this search engine
I use proxy.
IWantAllTheData··on Show HN: I scraped 3.2B TikTok profiles and 9B posts to build this search engine
Will check it out thanks. I'm always ready to take down the site if TikTok reach out me to.
IWantAllTheData··on Show HN: I scraped 3.2B TikTok profiles and 9B posts to build this search engine
Yes, I did receive some backlash, which is understandable. However, this is a very useful tool for content creators to find out what's trending on TikTok. It is also used by brands to analyze market sentiment about them.

I can't share the specifics, but i can explain the methodology in private. My contacts are on the website.

IWantAllTheData··on Show HN: I scraped 3.2B TikTok profiles and 9B posts to build this search engine
All the data were scraped without creating a TikTok account, AKA public data. Google.com has scraped more TikTok profiles and videos than me.

In fact, one the way i did it is by scraping google search results itself.