Python 3 on Google App Engine flexible environment now in beta
cloudplatform.googleblog.com
cloudplatform.googleblog.com
Anybody has a reasonable theory for why this took so long, despite users asking for it for years?
Well, initially, their investment in more runtimes was covering additional languages, not covering more versions of languages they already had. (IIRC, AE started with Python 2, added Java first, and then later Go and PHP.)
As that was happening, they gradually shifted focus from building out the old, fairly limited model of App Engine to building out other parts of the cloud platform, including (particularly relevant to future App Engine work) Compute Engine, to have a more robust cloud offering covering more use cases.
Then, they went back and returned to improving App Engine -- leveraging the now-broader cloud platform, and particularly Compute Engine -- shifting from the old model for new runtimes to the Flexible Environment (originally called "Managed VMs") model (built on Compute Engine, and which, in addition to including some additional Google-provided runtimes, means people can package up their own runtimes for App Engine).
Since doing so, they've released official Google runtimes for Python 3, Ruby (which people have been asking for longer than Python 3), and Node built on the Flexible Environment.
But then again the Flexible Environment seems to be a very different product with different features/characteristics and pricing than what I once knew as "App Engine". I would then argue that Python 3 never came to "App Engine", only to their new product. Which is a valid strategy for Google of course, but now stifled my enthusiasm. Probably I only need to learn more about Flexible Environment style App Engine to become hyped again.
(It wasn't obvious to me why, given a first read of the docs; the fact that applications run in VMs in Flex seems more like an implementation detail. Am I missing something?)
The standard runtimes are superior in a lot of ways, two big wins being lightning fast deployments (seconds vs. many minutes) and batteries included for all the app engine services. Logging in the flexible environment can also be a bit lacking. When you have something running in standard runtimes you basically don't have to think about many of the things that can normally go wrong on a server.
That said, many things about the flexible environment are rapidly improving, they've certainly come a long way since we first started playing with them around 6 months ago.
Flexible environment supports any language/environment you want as long as you implement the HTTP endpoints it expects. I would be much more impressed if they brought Python3 to the standard environment which seems to be much more cost effective for many types of web apps. Flexible environment essentially requires you to run a VM all the time, which can get pricey, versus a much more managed sandboxed environment that can sleep and be awaken quickly as needed to serve requests. Also, Python 3.5 would be awesome to leverage asyncio, but that's probably still a bit too bleeding edge.
For our our app (Streak) we run 100's of instances, we moved some of our services from standard GAE to flex GAE and saw a HUGE reduction in costs. Depending on workload we saw 5x-10x reduction in cost. To see why this is possible, look at the price of a standard GAE instance and the price of a default flex instance. For roughly the same price you get a way more capable machine.
You just have to have enough traffic where you'd like to keep at least one instance running all the time. Enough traffic in this case is a really small number. If you're traffic is lower than that number than who cares about optimizing costs, its gonna be cheap no matter what.
[1] https://cloud.google.com/appengine/pricing#standard_instance...
[2] https://cloud.google.com/compute/pricing#predefined_machine_...
Compare this too flex VMs where the smallest machine is $0.038/hr and have beefier CPU and 3.75Gb of ram.
Doesn't it always make sense to use the flex VMs? The only downside being min 1 instance and slower autoscaling.
https://cloud.google.com/appengine/sla
Beta products are also more willing to make backwards incompatible changes with shorter deprecation timelines. But overall it's certainly a lot more, ahem, flexible. Sneaky awesome feature is ability to install things like imagemagick , phantomjs, or ffmpeg, which to shamelessly self-promote I wrote about here:
https://medium.com/google-cloud/scalable-video-transcoding-w...
That's using a custom runtime, which won't be as officially blessed, but you can of course just extend from the official runtimes, since they're on Google Container Registry and open-source:
https://github.com/GoogleCloudPlatform/python-runtime
Just write a Dockerfile that starts with: FROM gcr.io/google_appengine/python
Obvious disclosure again I work for GCP.
There are pain points with the flexible environment that we don't have to worry about at all with standard runtimes. The two biggest ones for us are slower deployments and a lack of monitoring tools.
Just today we actually switched one of our App Engine modules to Flexible runtimes out of necessity as we we're hitting quotas for App Engine infrastructure, and in doing so we no longer have easy access to metrics on how many instances are running or memory usage for the service. The "Cloud Trace" tools are also impaired when you aren't using standard runtimes. In short, you lose visibility into your instances. It ends up being a tradeoff between developer time and cost.
With certain runtimes in the flexible environment, you also lose out on pieces of the App Engine infrastructure like memcache, which are invaluable in optimizing services.
Most of these have gotten a lot better since managed vm's first rolled out (3 years ago!?!?!?!?!)
https://github.com/GoogleCloudPlatform/appengine-sidecars-do...
In a nutshell, you can use a standard memcache library from your favorite language.
Disclosure: original author of this proxy when I was on the App Engine team.
I'm guessing the flex price will go up when it leaves Beta?
[1] https://cloud.google.com/appengine/docs/the-appengine-enviro...
But not Postgres or Redis?
Edit: OK, I'm seeing some references in the docs. Still seems limited to deployment on generic instances though.
You can host your own Postgres on Compute Engine [1] or use a hosted provider like ElephantSQL [2]. You can use whatever client library you want to connect to postgres in App Engine flexible.
Same story for redis - you can host it on Compute Engine [3] or use something like redislabs.
[1] https://cloud.google.com/solutions/set-up-postgres
[2] https://www.elephantsql.com/
[3] https://cloud.google.com/launcher/?q=redis
Note: I'm in Developer Relations for Google Cloud Platform
Edit:
I notice that ElephantSQL doesn't surface in a partner search but EnterpriseDB does.
https://cloud.google.com/partners/#?q=postgresql
EnterpriseDB seems relatively expensive, even when just getting started (Estimated costs: $891.00/month).
https://console.cloud.google.com/launcher/details/public-edb...
Redis is provided by Click to Deploy, i.e. "Popular open stacks on Google Compute Engine packaged by Google", and is in beta (Estimated costs: $307.80/month).
https://console.cloud.google.com/launcher/details/click-to-d...
Looks like Redis Labs has a trivial free tier, and the 1GB plan is cheaper on GCE than AWS ($57/mo vs. $71/mo). I'm pretty annoyed that I had to create a Redis Labs account to see that though.
Disclosure: Also GCP Developer Relations
If you wanted to have some instances that couldn't talk to your database, easiest thing to do is probably throw them in a separate project or create a new subnet that you firewall off and create your non-App Engine instances in that subnet.
Just comparing the smallest instances, elephantsql:
- db.t2.micro 1 year 20 USD / month (250 USD upfront) - 24.82 USD monthly (that's a multi-slave with auto failover) compared (with 30gb (SSD) storage it's at 30 USD) to elephantsql only 2GB data on a shared database server 20 USD
- db.t2.small 1 year 41 USD / month (500 USD upfront) - 50 USD monthly (that's a multi-slave with auto failover) (with storage around ~60 USD) vs elephantsql Happy Hippo single dedicated database 99 USD 100gb data
you kidding right? I would need to pay twice the price.
Just kidding. Python 3 adoption will continue to be slow, and judging from PyPI stats, nothing to write home about even atm.
I agree with the parent post: now is the time. Open source projects should to stop supporting py2 in their new major release and cleanup their codebase from the hacks made to support both languages.
The danger with this for Python is that it can give the impetus (is that the word?) and motivation for people with py2 legacy projects (which is the majority of people) to say "fuck this" and go try another language that doesn't pull this shit on them. Go, JS with Node, Ruby, whatever...
If they're not gonna get compatibility anyway, they might jump ship altogether. At least for green-field projects...
Ruby (and Node) are supported on the Flex Env the same way Python 3 is. JRuby may also be usable on the standard environment's Java runtime (ISTR seeing a how-to in the past, but I also seem to recall at one point seeing that there were some issues.)
More info on the different Django+AppEngine options here: https://cloud.google.com/python/django/ https://code.djangoproject.com/wiki/AppEngine
Disclosure: I'm an engineer at Google.
Super slow and hard to navigate. Stupid "tutorial". I tried for a good two hours, tried to accept it but finally gave up on it.
Google, cut down on your Angular and material design kool aid and try harder next time.
cloud.google.com/python
You'll see we have a lot of tutorials that link to Github repos . If you file issues there you are more likely to get help. The caveat is we are not a support team and have no guarantees (just a likelihood), so if you need paid support look into our options for that instead.
Google Cloud Platform UI (and speed) was just actively hostile towards me.
For example I went to the App Engine section to create a simple flask app. It keeps forcing you through a stupid tutorial trying to guide you around the site. Believe me being a developer myself I'm no stranger to web and its interfaces and can quickly navigate my way around this stuff but this had me hitting my head on the desk.