There are other, more scalable ways to do this. Reach out to us if you want to talk more :)
87 karma · joined February 24, 2016
It can be rough to manage Kubernetes when you just want to focus on your app. I was having that same problem when building side projects and focused on building a platform to deploy dockerfiles on a managed Kubernetes instance.
I'll probably make it a bit smaller than that in the next version.
(We don't throttle requests either)
I'll put a (very high) cap on the storage space to prevent abuse :)
I'd love to get your feedback on this!
For now, I only save the links to the data on other websites. This way, the data is always up to date.
In order to acquire those datasets, I wrote a couple of scrapers using chromeless.