Comment by ajross | Hacker News Reader

Comment by ajross | Hacker News Reader

Parent Full thread

Reading the thread (I'm curious about GAE, not an expert), it seems like the details aren't clear at all. Neither the original poster nor the Google rep seem to have a clear idea of what I/O operations are being generated.

But this bit stood out: $0.10 per 100k writes. That price seems to be far too high. The poster is doing (something like) a reindex of 10M entries (that kind of data is pretty small really: it's the kind of database you might use as a test set on your laptop interactively). Figure each modification is atomic, and that the b-tree height of the storage is ~4. So that's 40M writes to create an index, or $400!

Seriously? Again, this is the kind of task you'd expect to do quickly and interactively on your development box, and it costs a price of the same order as your day's salary (!) to execute in the cloud?

Looking at this from the perspective of the underlying I/O device: this index consumes just a tiny, tiny fraction of a hard disk drive's capacity. Yet creating it costs enough to buy the device several times over?

Something is wrong. Is that a misquote or have I misunderstood?

$0.10 per 100k writes is just a way of them billing you. Sure it's a lot compared to your avarage HDD price, but the data is hosted on at least 3 datacenters at all times, running google's software stack & everything is being constantly monitored by Google's SRE team.

App Engine pricing might seem expensive if you try to do a simple table comparsion with alternatives, but when you get more deeply into it you'll find that a lot of stuff that is included in the service with GAE will cost you extra when you use the alternatives.

combine that with the fact that the High Replication Datastore will most probably never see a downtime and you tend to be ok with the cost (of course as a app engine customer, you would always want lower costs, but the current write cost is not a deal breaker).

The only problem here is that "delete" is considered a write and when you want to delete data you just cannot accept the fact that you need to pay for something you do not want to keep. I think GAE should definitely look into this aspect and try to get some cheaper alternatives for data deletion.

One of the things that is not included is any real support, unless you go for the $500/month Premium package, which then only buys you a 2-hour turnaround with 18h/5d support. For that much money you should at least be able to get some support on the weekend.

I believe that $0.10 per 100k writes is what both Windows Azure Storage and Amazon S3 cost. But honestly, the cloud is the future!

Disclaimer: I work at Microsoft and am required by the terms of my employment to believe in "the cloud".

I believe that $0.10 per 100k writes is what both Windows Azure Storage and Amazon S3 cost.

Correct. Of course, those are object writes -- Elastic Block Store disk I/O is 10x cheaper.

Right. The granularity to be expected with an S3 object (~= "a file") is much higher than with a database index of tree nodes of a few kilobytes each. And of course EBS is subject to caching on the host. Rolling the index into a transaction can eliminate almost all the duplicate I/O to the tree nodes.

> So that's 40M writes to create an index, or $400!

At $0.10 per 100k writes, 40M would be $40.

According to the billing page [1] writes are indeed billed $0.10 for 100k operations.

Also, reads, writes and small operations (which are the ones billed) are low level operations. An API operation actually translates into several low-level operations. And the way it is described [2] I think the poster is doing more writes to reindex 10M entries.

Considering that reindexing takes 1 write for the entity itself (existing put) + 4 writes for each element in the list property and considering that the poster has on average 18 elements in that list for each entity, then he's probably doing on average 73 writes per entity (I'm taking the "Existing Entity Put" scenario into account, otherwise for new entities it would be 2 + 2 per list element == 38 writes).

So by these numbers, that's 730,000,000 writes, or a cost of $730 -- if you go over them sequentially, only one time. But considering that he's doing manual full-text indexing, maybe he had to go over those items several times for the reindexing being done.

Maybe I'm missing something here. I don't know.

[1] http://code.google.com/appengine/docs/billing.html

[2] http://code.google.com/appengine/docs/python/datastore/entit...

Those are "writes" from the users perspective, it doesn't take into account writes due to index updates. So that's 1 write multiplied by however many indexes that need update.

AWS seems to be $0.11 per 1 million I/O requests to EBS/RDS.

Reply on news.ycombinator.com