Hacker News Reader

Comment by nopurpose | Hacker News Reader

It is a cool project. S3 can be cost efficient, but only if you don't touch data :)

Their price calculation doesn't mention cost of S3 requests, which very quickly adds up and is often neglected.

It costs $1 for 2.5M GET requests to S3. They have 180 shards, in a general case query seems to fetch all of them. Presumably they don't download full shard per request, but download an index + some relevant ranges. Lets say that is 10 requests per shard. So that would be 1800 S3 GET requests = ~1400 search queries cost them $1.

Assuming their service is reasonably popular and serve 1 req/second on average, that would be $1,440 per 30 days in addition to advertised $1,000 spent on EC2 and S3 storage.

Seems comparable to AWS ElasticSearch service costs:

- 3 nodes m5.2xlarge.elasticsearch = $1,200

- 20TB EBS storage = $1,638

View on HN

I tend to agree :). If we get 1 req/s, even for a dataset of that size, this is not as cost efficient.

For that kind of use case, I'd probably start using minio.

> Seems comparable to AWS ElasticSearch service costs: > - 3 nodes m5.2xlarge.elasticsearch = $1,200 > - 20TB EBS storage = $1,638

Don't forget S3 includes replication. Also EBS throughput (even with SSD) is not good at all. Also our memory footprint is tiny. This is necessary to make it run on two servers.

Finally, cpu-wise, our search engine is almost 2x faster than lucene.

If you don't believe us, try to replicate our demo on an elastic search :D.

Chatnoir.eu is the only other common crawl cluster we know of. It consists of 120 nodes.

> If we get 1 req/s, even for a dataset of that size, this is not as cost efficient.

How many req/s do you have in mind for your system to be a viable option?

> Also EBS throughput (even with SSD) is not good at all.

It is not worse than S3 still, right?

> Chatnoir.eu is the only other common crawl cluster we know of. It consists of 120 nodes.

I have no deep ES experience. Are you saying, that to host 6TB of indexed data (before replication) you'd need 120 nodes ES cluster? If so, then reducing it to just 2 nodes is the real sales pitch, not S3 usage :)

What about d3en instances? Clustered, and together with minio you might reach similar performance. Only issue is the inter-region traffic, it would need to be inside the same AZ

EDIT: Realizing that d3 has just slow HDD

Have you checked out the new EBS gp3 disks? Throughout vs cost is much better on those than gp2, and also cheaper than Provisioned IOPS

I had the same feeling when reading the post. Their remark that they "estimated the cost" to be that low is in my experience a bad signal. Estimating costs on the cloud is really hard, there are so many (hidden) costs you may miss making it a lot more expensive.

For what it's worth, if you want to run ElasticSearch on AWS I would always go with local-NVMe instances from the i3 family, this is also what AWS and Elasticsearch themselves recommend.

4x i3en.2xlarge (64GB / 5TB NVMe) at $449 / month (1yr reserved) is $1796, or $2636 without reservation, but much better performance due to the NVMe drives.

It's easy to put a block cache in front of the index, and I'm sure they'll get to it sooner or later.

The benefit of using S3 in that case is that unlike e.g. Elastic, your block cache servers don't need replication, and you can tear them down when you're done. You can put them in a true autoscaling group as well.

For Digital Ocean object storage, data transfer to/from a Digital Ocean VM is free. You only pay for bytes-at-rest.

But it seems S3 doesn't have a similar offering. Data transfer is free between S3 and EC2 instances, but you still pay the per-request charge.

I wonder can you factor this into the pricing algorithm.

Digital Ocean has a limit of 750 requests per IP per second, so you wouldn't be able to complete a search query at all (1800 > 750).

https://docs.digitalocean.com/products/spaces/#limits

Obvious optimization would be to cache chunks locally on every worker nodes.

AWS is almost never cost efficient. Maybe if you stay in their free tier.

> AWS is almost never cost efficient.

A ridiculous blanket statement, despite the "almost never" cop-out...

It is cost-efficient in a wide array of scenarios. Many companies pay for it because they have calculated the different investment scenarios and AWS comes on top of alternatives such as owning the hardware or using competing cloud vendors.

I own a consultancy that builds complex web apps and while I appreciate how occasionally a dev has tried to save costs for me by cramming every piece of the stack (web server, cache, db, queue, etc.), in a single Docker image to host in a droplet, I'd much rather pay for separate services, as I consider it cheaper in the long run.

Many companies pay for it because they have spent an inordinate amount of time learning the ecosystem and know of nothing else.

Name an AWS tier that there's no cheaper alternative for. I'm only aware of Glacier, I haven't seen anything cheaper than it.

AWS is convenient and reliable, but it's not cheap.

Cheap is a whole lot different than cost efficient.

Convenience and reliability are deeply related with cost efficiency - that’s the point the example I made was meant to illustrate.

> I consider it cheaper in the long run.

Really?

Yes, I charge $60-$90 per hour of dev time to my customers and the time saved from using simple Elastic Beanstalk deployments pays for itself in saved dev time. The architecture is also infinitely easier to reason about and scale than cramming parts in a single image.

Reply on news.ycombinator.com