It'd be very hard to extend the existing solution to meet that use case.
5 karma · joined March 24, 2023
It'd be very hard to extend the existing solution to meet that use case.
- Requests to download even a small amount of data get rate ACLed (it says slow down/too many requests) - It seems like this is a known issue and that common crawl is no longer well maintained. https://groups.google.com/g/common-crawl/c/BvMGYUY-dro
Are there any alternatives for accessing a large amount of web crawl data?
Thanks!
heh heh heh
(I added most of these into a feed reader and it works pretty well, although it didn't have time to grab all the podcasts)
What's neat is it ingests the points from HackerNews, so you can splice the feeds together in a way that preserves the ranking of both feeds.
website: www.spronket.com tutorial video: https://www.youtube.com/watch?v=ug4ZqLro0-o
lmk if you have any questions or feature requests
www.spronket.com