Switched away from App Engine, couldn't be happier
war-worlds.com
war-worlds.com
With that being said, I'd really like to encourage the OP to store his data base (and boot his instance) from persistent disk. Running any database on scratch disk (without replication) is probably not a good idea. Even with hourly backups (make sure you are testing restores!) you still stand to have up to an hour of data loss and the pain of doing the restore if your instance should fail.
In addition, when using PD for all block storage you can start with a smaller instance. If you need more horsepower you can terminate that instance and boot from a larger instance with a minimal amount of downtime.
| Instance Type | MB-mem | $/hour | $/month | $/hour/GB |
|----------------------------+--------+--------+---------+-----------|
| GAE F1/B1 | 128 | 0.080 | 57.60 | 0.64 |
| GAE F1/B1 (reserved) | 128 | 0.050 | 36.00 | 0.40 |
| GAE F2/B2 | 256 | 0.160 | 115.20 | 0.64 |
| GAE F2/B2 (reserved) | 256 | 0.100 | 72.00 | 0.40 |
| GAE F4/B4 | 512 | 0.320 | 230.40 | 0.64 |
| GAE F4/B4 (reserved) | 512 | 0.200 | 144.00 | 0.40 |
| GAE F4_1G/B4_1G | 1024 | 0.480 | 345.60 | 0.48 |
| GAE F4_1G/B4_1G (reserved) | 1024 | 0.300 | 216.00 | 0.30 |
| GAE B8 | 1024 | 0.640 | 460.80 | 0.64 |
| GAE B8 (reserved) | 1024 | 0.400 | 288.00 | 0.40 |
|----------------------------+--------+--------+---------+-----------|
| GCE f1-micro | 629 | 0.019 | 13.68 | 0.03 |
| GCE g1-small | 1783 | 0.054 | 38.88 | 0.03 |
| GCE n1-standard-1 | 3932 | 0.115 | 82.80 | 0.03 |
| GCE n1-standard-2 | 7864 | 0.253 | 182.16 | 0.03 |
#+TBLFM: @I$5..@>$5=($3/$2)*1024; %.2f::@I$4..@>$4=$3*24*30; %.2f
For example there’s this old thing: https://docs.google.com/spreadsheet/viewform?formkey=dDRKNzd... (Google App Engine VM Runtime TT Sign‐up)[1]
Unfortunately, since all of this effort went into securing yet another runtime (PHP), I don’t think it will happen.
It would be great if at least it was possible to use Appscale over GCE or just the Datastore – which is by far the most attractive feature of App Engine, other than Task Queue – with custom deployment solution.
Google Cloud Datastore could deliver just that if you don’t mind paying twice:
You should be aware that Cloud Datastore has a serving component
that runs on Google App Engine, so there will be instance hour costs.
This is a very sensible limit. As your game is growing, more and more of these requests will pile up and will take resources away from quicker requests, making the game slower for everybody and crashing your architecture.
Adding limits like this is one of the very basic things you do when you need to scale. Changing platforms in order to get rid of limits like this will only mean that you'll have to re-add them at some point.
The correct fix for not running into such limits is to fix the code, not remove the limits.
Saying that a legacy product sucks, but that you still need it to run is not an effective mindset. These platforms have constraints. Work within them and the app will scale and perform very well.
Having an open connection for 30 seconds + is bad practice, and App Engine is a shared resource. Arbitrary limits are sometimes bad but in this case it will deter poorly designed APIs from starting in the first place.
We had earlier versions that required long HTTP calls and it was generally less reliable if something should happen during the call.
It has a cost (it takes time to fix it) and value (users won't have to wait). The developer has more information than Google to decide whether cost or value is higher.
EDIT: long-polling aside, and even then, 30 seconds is a perfectly acceptable limit.
While I agree that a 30 second limit is to be expected in order to conserve resources, saying long requests aren't needed shows a lack of imagination.
Streaming music and certainly video are generally delivered as chunks, with a few chunks per request that don't require long connections [2].
[1] https://github.com/LearnBoost/Socket.IO/wiki/Configuring-Soc... [2] http://en.wikipedia.org/wiki/HTTP_Live_Streaming
[1] http://googleappengine.blogspot.com/2011/10/app-engince-155-...
One of the overheads I did not realise that took a long with was using channels. We found them about 3x slower than a get or put. By parrallelising those with another thread we reduced our latency hugely. Our game went from taking 8 seconds to service a move, to 300ms.
Just my experience, did you work hard with a profiler before moving all the code?
Using memcache for store gameplay data is not the best idea. Memcache could be evicted at any time without any previous warning. Using a backend instance and storing the data in the ram is an order of magnitude faster, and thanks to the runtime environment api the service could stop gracefully.
The price hike completely killed me. I optimized what I could but there wasn't much to gain. Because of the way GAE is architected it forces you into efficient design.
I switched to AWS, and now with a lot more traffic across all the apps I am still paying ~20% of what my GAE bill was. I invested a lot of time into the platform, only some of which can be applied to other platforms, and fell for the bait and switch.
I really don't like what Larry has done with squeezing profit out of each business unit - the pricing just doesn't make sense for eg. implementing a simple spam filter for blog and forum comments using the Prediction API cost $30 per month alone for a site with thousands of visitors.
Completely killed a product that could be the center of making Google the best cloud platform for developers.
Personally I'm happy with Hetzner:
http://www.hetzner.de/
But I did stumble across http://serverbear.com/ which seems to have some useful information, though many of the plans they show aren't actually available (out of stock) at the providers listed...
However, I eventually needed an upgraded server, most importantly the ability to do an out-of-band remote reboot. I switched to Hivelocity for that.
To get the best deals on hosting, look carefully at the hosting company's website, search for announcements of sales on Webhostingtalk, and check their Twitter and other social media.
https://www.ovh.com/us/index.xml
It's important to note that their North American data centers are actually located in Canada although I do believe they are planning on eventually opening a data center in the US.
I am a bit surprised that 300 users costs him $10 a day though. There must surely be some giant inefficiencies in the code for that to happen.
for the front-end instances, i have no idea what can it be. I suspect the numbers are high because of "nice" global distribution of users.
Claiming to be better doesn't make you better
I see you are not one of those people.
What worries me with SQL is that queries can do just about anything, and it'll do it right next to the data. That means that you want your database machine(s) to be hulking monsters, and sharding / replication gets complicated.
In my personal experience, the App Engine datastore exposes fewer and simpler operations which scale horizontally more or less perfectly. It's harder to write for initially, but it scales up incredibly smoothly.
The GAE data store is really nice for persisting objects that come from users filling in HTML forms, or equivalent. One user can only type in data so fast and that kind of volume is fine on GAE. Plus you get great read scalability.
However, if you're creating data in some sort of simulation or loading it in from elsewhere, use a relational database (you can connect to external servers from your GAE app). Loading a few million rows into the GAE data store would cost you a fortune and the lack of joins or proper SQL probably means you have to get them all back out into memory again before you can do anything useful.
QueryTree uses the GAE data store as a central hub for user account info and settings, then shards lots of MySQL instances for actual data work, which seems to work well.
The secret of affordable app engine is the pending latency slider. The default is CRAZY FAST. I have it set at min 5 seconds and max automatic. This is about 10x price reduction over the default settings. This prevents new instances spinning up until a request has not been served for 5 seconds.
The client replicates all the game rules so as far as the player are concerned there is no latency at all.
The game is called Neptune's Pride 2: Triton if any of you would like to have a look.
As a single developer, more interested in the game rules and player interaction, I would not switch to trying to manage my own servers for all the money in the world.
For those interested, right now I have the entire games save file stored in a blob property which I unpickle for each request.
The big thing is that GAE is only suitable for certain types of apps, so make sure your's is one of those before even starting.
The one small, hobby app I wrote for GAE was unsuited and it was a painful process to port away. My company, with a real GAE app, has a cost threshold where we know it's cost effective to move from GAE; write heavy apps are expensive in the GAE environment. Each day we inch closer to that threshold.
At this point the headaches of using GAE are more than the headaches of not using GAE, but I won't miss it when we pull the trigger on moving away.
> My company, with a real GAE app, has a cost threshold where we know it's cost effective to move from GAE
These guys may help: http://www.appscale.com/
If your app is written in Java, I believe http://www.jboss.org/capedwarf is another option.
For example: http://fwenvi-idl.blogspot.com/2012/07/cross-validating-neur... and http://neuroscience.telenczuk.pl/?p=435
How would you have dealt with a data corrupting code bug? If you don't have real backups, wouldn't those screw you?
> I ended up coming up with a simple cron task that takes a full dump of the database every hour and copies it to a "permanent" storage device. In the event of catastrophic failure, I should be able to spin up a new instance, copy my image to it and get the last hour's backup copied over in fairly short order. The main issue is the fact that I sleep for ~8 hours a day and work for ~8 hours a day, meaning it could be some time between when a problem occurs and I'm aware of it.
If you are just keeping the latest backup, you are still vulnerable to data corrupting code bugs. Generally for this kind of application you'll want to keep several backups, from several different times. Say, hourly backups going back 24 hours, then daily backups going back a week, and so on. The details depend on what kind of data you are storing and how important it is to your users that you can recover from data corruption.
foreach for while if for if return