Hacker News Reader

Comment by stewart27 | Hacker News Reader

Hi all,

Product Manager for App Engine here. Happy to answer any questions about Python 3 on the App Engine standard environment.

View on HN

Is GRPC supported? My gut feelign is no, but I hope I am wrong

Edit: Holy shit this is so bittersweet. I was relying heavily on the batteries included user management, logging, admin only handlers, and especially the super easy cron job definitions.

IDK what to think about this. It kinda takes away all the things that made App Engine worth it for me as a programmer who doesnt want to do dev ops.

AFAIK you can't run a gRPC server on App Engine, but you can access other gRPC services (like all the GCP APIs) as a client. I might be wrong about the server part.

All those things you mentioned in the edit should be supported in the new runtime.

Edit: I see what you mean. The deeply integrated services are removed in favor of more portable solutions, like Logging moving to Stackdriver Logging (though stdout is still logged automatically), Users moving to Firebase Auth, etc. Cron should still work fine though.

(I work for GCP)

Thanks for the reply. Yeah I'm bummed about that because id rather use GRPC client libraries for my apps than the endpoints discovery doc generated ones.

Also according to this they are not supported directly anymore except logging might still work but the wording is unclear. https://cloud.google.com/appengine/docs/standard/python3/pyt... .

That page mentions that it supports the google-cloud-python libraries, which are the primary GCP gRPC client libraries for Python. The Apiary service-discovery library you are referring to is called google-api. The new GAE version doesn't support the old GAE-only libraries, like memcache, datastore, etc. These older services only worked with GAE, and therefore weren't portable to other services, which they are in the process of replacing. If I recall, the old version didn't support the google-cloud-python libraries either, so it's good to see that they are moving in that direction. I could be wrong about that though.

In short, it looks like you have access to the main GCP gRPC libraries. You don't have access to the GAE-only service libraries.

Disclaimer: was former GCP customer support.

The Google Cloud client libraries [1] are fully supported. You can use these to access services like Datastore, Spanner, Natural Language, and many other Cloud services. They work on this new runtime, on the flexible environment, on a VM, your local machine, etc. In my opinion, this is the most Pythonic way to integrate Cloud services in your app while maintaining portability.

[1] https://github.com/GoogleCloudPlatform/google-cloud-python -- small caveat, the repo currently states that they're not supported on App Engine standard environment. That will be updated soon to clarify that the Python 3.7 runtime on the standard environment is supported.

Would the python3 runtime support things like ndb, taskqueue, memcache modules?

We have a python2.7 standard app in production. It would be a considerable re-write if we had to move away from ndb and taskqueue.

https://cloud.google.com/appengine/docs/standard/python3/pyt...

it doesn't.

"The ndb ORM library is not available for Python 3. You can access Cloud Datastore through the Cloud Datastore API. You can use the Google Cloud client libraries to store and retrieve data from Cloud Datastore."

We're working on suitable replacements for these services. In some cases, e.g., Cloud Tasks, we're pretty close. Others will take a little longer. We'll share updates as we make more of these services available.

Great! Thanks for your teams work so far.

Also I did find that someone wrote a ndb-like wrapper around the cloud datastore API. https://github.com/Bogdanp/anom-py I haven't tried it out yet.

Although i think any reasonable path to upgrading to 3.7 might need atleast some wrapper like that around the datastore(even if it's not 100% compatible), otherwise forcing people relying on that to re-write the entire data layer. At this point i might just start re-writing the app, to move to something with less lock-in like Mongodb. But i'd love to hear what other people's plans are for this.

What is the access to your hosted postgres like? Do you still have to use that proxy thing? I'm wondering how much effort is require to move a sqlalchemy based app over to this.

You would use Cloud SQL to access PostgreSQL.

Here's how you configure your App Engine standard environment project to use Cloud SQL: https://cloud.google.com/appengine/docs/standard/python3/usi...

The Cloud SQL for PostgreSQL docs contain code samples to get you started: https://cloud.google.com/sql/docs/postgres/

This should all work with SQLAlchemy.

It is not possible to access Postgres based CloudSQL from Python on AppEngine standard. See here:

https://cloud.google.com/sql/docs/postgres/connect-app-engin...

Note that only node and Java are listed under non-Flex.

I got burned by this and had to start over with MySQL.

> It is not possible to access Postgres based CloudSQL from Python on AppEngine standard.

It is possible on the new (beta) Python 3. 7 runtime on AppEngine standard:

https://cloud.google.com/appengine/docs/standard/python3/usi...

Then they should update their docs.

Thanks for noticing this. We'll get this updated.

If you're an existing AppEngine standard user/customer, it's pretty likely that that means you will not be able to take advantage of PostgreSQL, as the new runtime does NOT give you access to a ton of their existing services that you are probably already using (such as datastore.)

Hi optimusclimb -- just want to make clear that using Cloud SQL to connect to PostgreSQL should work using Python 3.7 on the App Engine standard environment. Your earlier comment accurately pointed out a gap in our docs. We're going to address that -- thank you.

If you try Cloud SQL and find that it doesn't work with PostgreSQL on this new runtime, that's a bug and we need to fix it.

I think what he's saying is you're currently using Python 2.7 on Standard and using existing built-in services like task queue, memcache, ndb, etc then switching to the new Python 3.7 runtime (to be able to connect to Postgres) runtime really isn't possible since there aren't non-rewrite solutions for those missing pieces.

The (datastore) client libraries take up the http request quota and the build-in mechanism of the Standard Environment is not. Google recommends going forward with client libraries everywhere in the documentation but what's the solution on the introduced request limitation which isn't mentioned anywhere? (Or is this finally fixed in the meanwhile?)

I'm not aware of this issue. Can you point me to a public bug/GitHub issue?

We hit that HTTP request limit as well, and (driven by a discussion with Google support) migrated our backend away from the REST-based Google Cloud Datastore API to the internal (ApiProxy-based) Appengine API. Reduced the latency as well. I can send you the internal support ticket nr if interested.

Sure, feel free to send it to me directly. stewartr at google.com.

Nice launch! I have some Python code which I've been meaning to convert to App Engine standard to take advantage of the free tier, and I hadn't yet adapted it to the proprietary APIs. Now I don't have to. Thanks!

One Q: Does Google Cloud IAP work with this as an option for user auth? Would one follow the App Engine flexible environment pattern in their docs? The Cloud IAP docs still mention the Users API as the recommended approach for the App Engine standard environment without an exception for the Python 3.7 runtime.

Which were the technological problems that you had to update App Engine to Python 3? I thought it were upgraded a lot of time ago.

Just FYI - this page still only mentions Python 2.7, presumably it needs updating: https://cloud.google.com/appengine/docs/standard/

Are there any billing changes for apps previously running within the free tier in standard environment?

There are no changes to the free tier[1] or to apps running using the existing Python 2.7 runtime in the standard environment.

[1] You can use the free tier with the new Python 3.7 runtime.

Can I run pandas in it?

May I follow up with:

pandas and numpy

Yup, both should work fine. Let us know if you run into issues.

(I work for GCP)

Article says "YES!"

> This new runtime allows you to take advantage of Python's vibrant ecosystem of open-source libraries and frameworks. While the Python 2 runtime only allowed the use of specific versions of whitelisted libraries, Python 3 supports arbitrary third-party libraries, including those that rely on C code and native extensions. Just add Django 2.0, NumPy, scikit-learn or your library of choice to a requirements.txt file. App Engine will install these libraries in the cloud when you deploy your app.

Yes, this runtime supports arbitrary dependencies specified via a `requirements.txt` file. If you find a library that doesn't work, please let us know. In general, anything that works using an open source Python 3.7 distribution should be supported.

Awesome!

Incidentally, the best way to "let us know" about bugs in GCP (App Engine or whatever else) is via the Issue Trackers, as per https://cloud.google.com/support/docs/issue-trackers (disclaimer: making this and similar processes work is my current job at Google).

Does this means, DJango can be supported natively ?

Yes, Django and Django 2.0 (which only supports Python 3) should work.

Reply on news.ycombinator.com