Hacker News Reader

Comment by 3pt14159 | Hacker News Reader

I just wish there was a single VM that wasn't SSD backed. I write web crawlers sometimes and I usually only have to parse the HTML / JS once into JSON or SQL, but I like to keep the data backed up just in case I need to re-parse it later, or to see the changes between crawls. It's completely impossible for me to even fit it on their $640 / month box. But I love Digital Ocean for everything else.

View on HN

Couldn't you potentially push that aspect elsewhere, into a data storage solution like S3 or something (not sure of the cost efficiency, not a huge cloud guy myself). Then you'd just be utilizing the DO server for processing. It would have the added benefit of allowing you to scale your crawling horizontally if you wanted, as all your servers could have access to the same set of data.

The traffic between datacenters would be significantly slower and more costly, though. Besides, S3 is not a replacement for a real drive.

It would be slower than on disk for sure, so if that's a concern, it wouldn't be a good choice. If the poster is just storing off results, pushing it to S3, or even potentially a document database could be a logical choice. They said they're just using it for occasionally re-parsing at a later date, so immediacy doesn't seem like a huge concern. Just a thought, at any rate.

Yeah I could, and I currently do - but I don't like to. I want the data on the same network for lots of reasons and I also don't like maintaining two sets of deployment systems. I'd prefer to just use Cloud66 + Digital Ocean and focus on writing code, not setting up and maintaining servers.

Oh yeah, I totally understand that, was just offering an alternative as a non-ideal workaround. I have the same set up for a service I offer, primarily because (afaik) it's hard to beat S3 pricing.

Vultr, who have offerings very similar to DO, have a class of VMs called "Storage Series"[1]. In my experience their uptime is similar to DO and their performance is better. Might be of interest to you:

[1] https://www.vultr.com/pricing/

Take a look at serverhub.com ssd cached vps, $5 per month for 500GB of diskspace.

Have you looked into the Kimsufi boxes? Sometimes they are available for 5 EUR/month. They come with a Atom cpu and 500 GB of storage.

That's why I like TransIP. Cheap SSD-backed VM + large network drive (up to 400TB).

Reply on news.ycombinator.com