Hacker News Reader

Comment by empath75 | Hacker News Reader

the scaled from nothing to the size of facebook almost in a few weeks... i'd say it's pretty impressive even with the speedbumps.

View on HN

Not from nothing, they had years of experience and infrastructure from Ingress.

Ingress ran on App Engine. I was one of several people working with Niantic at the time to address application architecture issues to avoid App Engine anti-patterns.

Interesting, can you share why it would not make sense to use App Engine here (or why Ingress ran into anti-patterns)?

Not GP but I recently assessed GAE for my company and ultimately decided to go with GKE instead. The reason was GAE had no support for websockets [1] and had a weird way of influencing almost all implementation details in your code base. It pushes towards full adoption of GCP's other products like CloudSQL, Memcache, Cloud Datastore, etc, because those products run so bloodly performantly you feel like a poor engineer not taking advantage of them.

In short those "negatives" are actually perfectly reasonable when you consider what GAE was designed to be: A Google managed PaaS that handles all the hard parts for you IFF you follow the platforms conventions. The problem is that others see it as a drop-in for EC2 and get burned as they throw away their previous tool chains like Postgres, Redis, and the entire networking stack.

[1]: http://stackoverflow.com/questions/4114902/how-can-i-run-a-w...

That Q&A is from 2010 (!). GAE Standard has the Sockets API, and GAE Flexible runs in a VM instead of a sandbox, so it supports sockets out of the box.

[Edit] Disclaimer: I haven't used App Engine in a while now.

I don't recall whether Ingress specifically ran into anti-patterns (or we advised them early enough to avoid them). As I left Google in 2013 I am not sure what the experience of Niantic was with regards to Pokemon Go.

Before App Engine used snowflake IDs for the App Engine datastore IDs were allocated sequentially. The App Engine datastore is based on Megastore [1] which is based on BigTable [2].

If you read massive amounts of sequential keys you could potentially run into issues with the automatic management of tablets. Compaction/splits could introduce terrible performance.

Since Niantic Labs operated within Google there was more flexibility around budget, but with other customers (for example Snapchat), you also needed to carefully control your cost. Perform projection queries (always a good idea) to only return the data you absolutely need and many other optimizations. There is a lot to learn that is specific to App Engine, for example [3] (this is a very old and now outdated SO answer I gave).

From my perspective, designing a high performance and scale, cost-effective application on App Engine is very difficult, but not impossible. This is of course only the case if your application is a standard web application (or REST API). A lot of functionality you may want (support for certain system libraries or connection protocols) may be unavailable.

IMHO, Google Container Engine (and similar services) eliminates many of the use cases for which I previously would have chosen a PaaS such as App Engine. I am in full control of my runtime and application environment without the need to learn proprietary APIs or system behavior, while not having to worry about managing the server infrastructure.

[1]: http://research.google.com/pubs/pub36971.html [2]: http://research.google.com/archive/bigtable.html [3]: http://stackoverflow.com/questions/14698119/httpexception-de...

Ingress, at peak, probably had only a small fraction the users Pokemon Go had in the first month of launch. So their experience of working at that scale was basically nonexistent. It's a whole other ball parks when you increase your player base by 2-3 orders of magnitude.

That being said, they really should've done better at predicting hype, especially since they had beta sign up and all.

Reply on news.ycombinator.com