Bringing Pokémon GO to life on Google Cloud
cloudplatform.googleblog.com
cloudplatform.googleblog.com
It would be naive for everyone to assume that a high traffic launch is all about the cloud underneath and only that.
The article didn't mention any of the technical details of the Pokemon application itself, for all we know the infrastructure was humming nicely and the application itself didn't scale. Or the other way around, or a combination of both or one of the other of thousands of moving pieces it takes to launch something.
http://www.businessinsider.com/amazon-cto-makes-dig-at-pokem...
https://www.reddit.com/r/pokemongo/comments/4rwrhf/amazons_c...
Werner actually posted as himself in that Reddit thread ^
"Guys, this was not intended as a snark, joke, rip or self-promotion. I just want people (including myself) to play given how much fun it is. We have some experience in running big software systems that need to scale and just want to help.
Update for the skeptics: yes I am @werner, not Amazon PR. I have the same username on Instagram."
EDIT: Unless you can prove the snark or malicious intent was there, to say otherwise is disingenuous.
This is called back-peddling.
Shit is complicated. Shit breaks. No one is infallible.
That's pretty low. FWIW Simcity 2013 is on AWS and the launch was far more disastrous. Doesn't prove anything.
I think Pokemon scaled remarkably better than any app many of us have ever dealt with, seeing the crazy demand spikes. You would run into a lot of app issues, data choices, a lot of things that don't just get solved from simple uses of autoscaling.
Pokemon Go is less complicated in terms of the conceptual model. There are only a few pretty simple ways that different players can interact (gyms, lures on pokestops, that's about it).
None of Twitter can be partitioned like that because every user follows different people.
Like contending with Team Rocket
In short those "negatives" are actually perfectly reasonable when you consider what GAE was designed to be: A Google managed PaaS that handles all the hard parts for you IFF you follow the platforms conventions. The problem is that others see it as a drop-in for EC2 and get burned as they throw away their previous tool chains like Postgres, Redis, and the entire networking stack.
[1]: http://stackoverflow.com/questions/4114902/how-can-i-run-a-w...
I don't recall whether Ingress specifically ran into anti-patterns (or we advised them early enough to avoid them). As I left Google in 2013 I am not sure what the experience of Niantic was with regards to Pokemon Go.
Before App Engine used snowflake IDs for the App Engine datastore IDs were allocated sequentially. The App Engine datastore is based on Megastore [1] which is based on BigTable [2].
If you read massive amounts of sequential keys you could potentially run into issues with the automatic management of tablets. Compaction/splits could introduce terrible performance.
Since Niantic Labs operated within Google there was more flexibility around budget, but with other customers (for example Snapchat), you also needed to carefully control your cost. Perform projection queries (always a good idea) to only return the data you absolutely need and many other optimizations. There is a lot to learn that is specific to App Engine, for example [3] (this is a very old and now outdated SO answer I gave).
From my perspective, designing a high performance and scale, cost-effective application on App Engine is very difficult, but not impossible. This is of course only the case if your application is a standard web application (or REST API). A lot of functionality you may want (support for certain system libraries or connection protocols) may be unavailable.
IMHO, Google Container Engine (and similar services) eliminates many of the use cases for which I previously would have chosen a PaaS such as App Engine. I am in full control of my runtime and application environment without the need to learn proprietary APIs or system behavior, while not having to worry about managing the server infrastructure.
[1]: http://research.google.com/pubs/pub36971.html [2]: http://research.google.com/archive/bigtable.html [3]: http://stackoverflow.com/questions/14698119/httpexception-de...
That being said, they really should've done better at predicting hype, especially since they had beta sign up and all.
"Google CRE worked hand-in-hand with Niantic to review every part of their architecture, tapping the expertise of core Google Cloud engineers and product managers"
Architecture analysis may or may not be standard (I don't actually know since I've never had to deal with something like this) but that sounds great to me.
Gold and Platinum support can help with architectural analysis. I'm also guessing Google had a vested interest to help Pokemon Go do an analysis of their architecture. Doing so could help find any inefficiencies to lighten the load on their cloud servers.
The omission of that fact may cause the reader to infer that Google Cloud Platform was chosen for it's technical merits, and not because the company was actually a Google team that spun out.
That said... in reality, they likely had real reasons to use Google since they probably have a higher expertise with it that AWS.
IMO This is the most valuable thing in this article. It essentially says what others are pointing out. You can't just press a button and have scale. It's not that easy. You have to tackle many layers. Considering they 50x'd their worst case scenario it would have only taken a few bad queries to fuck shit up.
I wouldn't expect any service to survive that much unplanned load. Maybe they could've estimated better, but how likely was it for the game to go viral? Worth sinking lots of dev time into and delaying launch? That's a hard question to answer without the benefit of hindsight.
The hard part about scaling is knowing when to invest in scale. The binomial choice is to build and throw away, or build so as to be scalable. So hard to predict.
Actually you can... if the system was designed for this purpose.
It's known that some things will break from time to time (e.g. servers, hard drives, networking). If you've designed the application poorly so that a single minor failure can snowball to the entire system, it's guaranteed that your site will fall apart periodically, costing you lots of money each time.
At this point, the best technical AND business converge to "pick an option that can sustain [minor] breakages". Either you've done it like that from the start, or you're in the process of rewriting/redesigning your site.
Not really. In the case described they had accounted for some maximum number of users and instead got 50x that number. It is reasonable to design a system which handles just some planned number as long as there is a strategy in place to manage a situation beyond those limits. In this case the strategy was to apply engineering effort. That seems like a valid risk management plan to me.
Rrrrrright?
Queues, buffers, locks all have ways of interacting at high volume that defy prediction. And if you think your system has no queues or locks then you just haven't looked deeply enough.
Nitpick: they 10x'd their worst case scenario
Just because you have the resources does not make for a well scaling service. As outlined in the post, "Google CRE worked hand-in-hand with Niantic to review every part of their architecture, tapping the expertise of core Google Cloud engineers and product managers". Look past the chart, this wasn't just an estimate for Google, it was also for Niantic. You don't have unlimited development resources, not every aspect of an application may have had the time to flesh out a scale-able approach.
Netflix didn't scale out overnight. I'm sure we've seen their techblog about X extremely specialized framework/tool they've built out over the years. I'm impressed with how quickly Niantic achieved a playable experience.
I have mixed feelings about this, didn't they drop more than half their peak user base to achieve playable experience?
However, it became perfectly playable after their "tracker patch" (which was the trigger for the massive exodus that followed).
When the game experienced the most instability it hadn't even been launched in most countries.
It's purely anecdotic, but after that release, popular spots (with multiple pokestops and arena in one place) usually crowded with more than 50 people playing actively were down to one dude on a bench.
Sure the "loud minority" is not representative of the whole game's community, but they were it's core players. They shaped the game's community (and niantic didn't helped) from the beginning. And it didn't help when those people decided to stop playing.
Back to server's issues, Google CREs did a very good job during these times. They can't be blamed for trying to scale an app that was absolutely not designed to get this high.
No doubt, the CRE program could prove valuable. But in this case, they are congratulating themselves on a rocky and widely panned launch of a product on their platform. One might wonder, "If this is what deploying a viral app on Google Cloud Platform looks like when you have help from Google engineers, what chance does anyone else have of getting something right on their platform?"
I think that's probably the wrong takeaway, but it's not difficult for me to imagine that being the only conclusion one has.
If I linked an old style "HP Whitepaper about Success with Customer X" it would be downvoted to hell - but that's exactly what this article is, but by Google instead of HP.
> In response, Google CRE seamlessly provisioned extra capacity on behalf of Niantic to stay well ahead of their record-setting growth.
Anyone who played that game within the first week of launch would certainly not use "seamless" to describe the experience.
You could argue that Google was providing GCE and therefore GCE couldn't scale, but GCE is really just hosted Kubernetes and it's scaling limits were known in advance. Luckily GCP was able to push a quick version update and migrate the cluster but that took considerable time, coordination and engineering effort that couldn't be done "seamlessly".
This is a clear situation where the software Niantic chose couldn't handle the resources that were available. It turns out a lot of the choices you made for your worst case capacity aren't necessarily adequate for 50x that amount :)
(disclaimer I work at Google but not on GCP)
GCE - Google Compute Engine. VMs in the Cloud.
GKE - Google Container Engine. Hosted Kubernetes. Runs on top of GCE.
(I work on GCE)
"reliable"
"low-latency"
I think a large part of that was due to the non-Google login from Pokemon Trainer Club (which must have been handled by Nintendo's servers, AFAIK), as opposed to the Google OAuth. Both groups of users had problems, but people who used their pre-existing, Nintendo-issued logins had much more problems (and it took a lot longer for those to get fixed).
There was a period of about a week or so during which time people who used the Pokemon Trainer Club accounts were still having just as much trouble logging in and staying logged in, but people who used their Google accounts were fine.
I wouldn't be surprised if providing multiple login options made it harder for them to properly separate their login servers from their game servers, and if this coupling meant that millions of users (effectively) DDoSing the PTC servers ended up impacting the game's uptime more than simply doing all authentication themselves would have.
I can't think of anything further from the problems this game had at launch than the login. So few people actually had pokemon logins this comment is honestly comical to me.
https://4.bp.blogspot.com/-GLc5ve5_djc/V-ysZgW6uDI/AAAAAAAAD...
The worst case scenario was 5x, which was a factor of 10 off. If you are that far off when you do your capacity planning, you can be pretty sure you've got problems throughout your entire stack.
That said, what you said is still true. If you're thinking you're going to have 100k users, you might be willing to allow a lot more data to be transmitted and/or processed than if you had 10m users. Just looking at tracker alone, having to transmit and measure distance between a dozen points and ordering them is a lot more work than checking whether a dozen coordinates are within a range and listing them in any order.
I don't believe this for a second. Nintendo knows where its goldmines are. IMHO it is far more likely that they steered clear of mobile because they didn't want to cannibalize their Game Boy sales. It's a classic case of a big corporation being slow to react to change or even trying to stop it because they hold a dominant position in the old system.
Nintendo would much prefer if "mobile gaming" still meant Game Boys but the world has already made their decision and they can't stop it. They can't ignore the sales figures on what would have otherwise been an obscure goofy spinoff fitness app on the Game Boy. They've ignored the mobile market for too long and now there is pent up demand. Without Pokemon Go I doubt you would have seen the sudden scramble to develop and release a Mario game for iPhone.
I'm not sure this is entirely accurate. The Nintendo 3DS (I assume what you mean by "game boy"), has been a very successful product for Nintendo. And sales of the 3DS have actually spiked as a result of Pokemon GO.
The reason Nintendo has avoided mobile gaming is probably down to fear of piracy. The DS was ruined (from Nintendo's POV) by flash carts, and with the changes they made on the next generation the barrier to entry for copying games is now lower on Android/iOS than on the 3DS. I think that when they do finally go mobile, the games will have a lot of server-side checks going on. The games will be a pain in the neck to play even if you are paying. For example, always-on internet required will only work on non-rooted devices, that sort of thing.
The Niantic team did incredible things given the instant historic success that became Pokemon Go.
The generic 1X, 5X and 50X are hard to understand in this context I think. Pokemon is so popular it's very difficult in imaging what the real numbers actually are. For instance on launch day in America and Asia I would have expected insane numbers (many hundreds of million).
I also feel like the 1X, 5X and 50X number placeholders are useless in this conversation because it doesn't give a sense of scale at all.
Given no app had ever done this in history over the first few weeks, yet alone on launch day with no marketing, I doubt it.
But in all seriousness this is Pokemon; they sold over 3 million copies of a remake of a game on a niche portable console in just 3 days; elevating its brand to an open platform for FREE that can be downloaded and installed on theoretically, what, a billion or more devices? Seems incredibly doable for Pokemon. Few other brands could do the same. Even Mario wouldn't be able to come close to competing with Pokemon's brand power.
I would be very interested in the real numbers :)
If this doesn't clearly show how crazy this growth was compared to what anyone was expecting, I'm not sure we're operating in the same plane.
"Many hundreds of millions" implies at least three of them.
I would not expect the entire population of the US to be playing on day one, especially the roughly 50% of them that don't even own a phone.
Sigh. This would have been easier if we had real numbers. Let us all test our hypothesis. Oh well.
The 2 lines can be thought of as ceilings or upper bounds, hence why they are static - this are the numbers that traffic was expected to eventually reach at peak.So you can think of it as, "we thought we'd be looking at graphs that had this line as the top and traffic would be some curve underneath.
Obviously from the graph shown here, we/they needed a tall graph.
I think this is the disconnect. This wasn't not obvious to me. It looked like everyone was expecting flat growth at either 1X or 5X. If there was a line showing what they thought would be the traffic that goes up (which would be expected) in addition to the ceiling lines then I think there would have been a lot less confusion.
I wonder if a single bar graph would have illustrated it better with overlaying colors for each ceiling.
Question: When should we start checking around for posts containing
A) High-level technical overviews with some basic implementational detail
B) In-depth analyses of the stack you built, the challenges you faced, what improvements you folded back into various open-source components, what you'd have done differently, etc etc
?
I'm thinking in terms of timescales - like n months or so. I suspect (A) will be a little easier (and quicker?) to publish than (B).
The article says (in one of the only bits of real information) that they blasted past their estimates with only Australia and NZ just 15m after launch.
Whoever came up with those numbers must have had some serious methodology flaws. I know they couldn't predict that it would become the biggest online game ever for a while, but the initial demand prediction was clearly way off even before it started growing like a rocket.
Pokemon Go became the most-downloaded app in history within the first few days.
Are you saying that's the bar that people should estimate at launch?
>Throughout my career as an engineer, I’ve had a hand in numerous product launches that grew to millions of users. User adoption typically happens gradually over several months, with new features and architectural changes scheduled over relatively long periods of time.
When it comes down to it, the game is closer to a fitness app than a Pokemon game and can be described as bare bones. The fancy accessory wasn't even available at launch. Many of the players are people who have not bought a Pokemon game in over a decade, or ever in some cases.
If there is one area I can solidly criticize Niantic for it is rushing the other region releases so quickly. It's clear the servers were already badly overloaded and they just started adding countries left and right. I know it sucks for people who live in those countries to have to sit out while the rest of the world is having fun, but it's really not much more fun to sit at the eternal loading screen because the backend is entirely on fire.
And you're sitting there as the engineering lead or staff going, "How do I even feel about this? Fortunately I have no time to feel because I am off to fight fires." I didn't go home for 2 days, I worked 82 hours that week and >70 the next.
Complaining that the estimates are bad for a product that literally broke everything we know about how to build a successful mobile game and has scaled to a truly unprecedented level is meaningless. Obviously no one expected this. Obviously the engineers wouldn't have wanted it. Obviously the world will respond the way it will to our work.
Show some compassion. But also some humility. None of us are qualified to make projections in the face of phenomena like this.
You can't predict it will be the biggest game ever, that's not possible. But I feel like their initial estimates were still too low and you could predict it would have been higher. I don't know if it was based on how well Ingres did but...
Pokemon is a huge property. It's had tons of games, movies, sleeping bags, an incredibly successful trading card game, etc. Just having the Pokemon brand on something makes it VERY big.
In the game, you live out the Pokemon dream. This isn't just Pokemon Puzzle League. This isn't Pokemon Mystery Dungeon where you navigate cute little Pokemon around and play a top-down rogue like. You FIND AND CATCH Pokemon in the wild. It's exactly what Ash did in the TV show or comics.
Also, Pokemon are cute as hell. That plus the novelty of the AR stuff meant this had a lot of potential. "Look, I found a cute Eevee over here on my potted plant!" Those pictures were EVERYWHERE. That's tons of viral advertising.
But there is also the in-person effect. You want to compare what Pokemon you have with other people, and that encourages you to get your friends into it. But people were walking outside with their phones playing the game, and they quickly got spotted by people asking "What are you doing?"
All these things make it clear to me the this game had a high chance of success.
There's no way to know it would go to 50x what they guessed or would top the charts. Given their numbers I wonder if the expected should have been closer to 7x and the worst case at 20-25x.
The popularity they got would have taken basically anyone down. That was going to happen. I'm just surprised the estimates weren't much higher.
What numbers? We know that their estimates and their realities were quite different, but not knowing the real numbers we have no way of even beginning to judge what's reasonable and unreasonable here. For all we know, they modeled directly after the most successful game in the Android market at that date as a baseline and then said, "At the worst case we'd expect 5x THAT."
You can write a ton of paragraphs about how cute Pokemon are, but the truth is that the Pokemon AR game was a massive risk. AR games have had extremely limited update. It seems incredibly unreasonable for me to expect that those folks should have realized apriori that they were about to release the most popular mobile game ever created.
I, for one, will not throw stones. I don't get why you feel the need to assert that you (or anyone) could have done a better job by setting a 20x or 25x target. Or that you could have not only forseen it was necessary, but convince everyone around you that the capex was justified.
Why are you so keen on assigning blame and shame in this scenario? Some of our peers made history. Can we be happy for them for 6 months before immediately backseat driving about how much better we all are in our armchairs?
A mobile game, available to be downloaded onto hundreds of millions of phones, that is also FREE? I feel like they broke 100 million in the first day at the latest (counting totals from each region's first day). It wouldn't surprise me if it's significantly higher than that.
Media, users; everyone has been BEGGING Nintendo to release IP to mobile devices but they have kept it locked up in their own hardware. If no one had even a rough idea of the possible popularity they most certainly had a very, very wrong methodology.
However, the Pokemon Go player in me says "Wow, even with all of Google's resources, they still couldn't manage to get this remotely stable for several weeks?".
I'm sure there was many amazing technical feats that occurred, and from a deeply technical level this is a good sales pitch. I'm sure a good sales person could spin it even better "50x your expected traffic? Google Cloud can do that!". But beyond that... most people will probably see this as a failure.
From watching my partner play the game (from Canada), the scaling issues went on for days and it serverly hampered her enjoyment of the game. This should be a cautionary tale of having worse case scenario planning at launch time, which a large organization like Nintendo should have factored in (when planning with Niantic).
This article doesn't fairly reflect that. But maybe they are acting confident now because they will be better prepared next time? They can point to Pokemon Go to why you need experts who have been through the trenches of a rocky high-traffic launch.
The "without incident" part is hilarious. The game was unusable for over a week when they added more countries. There were memes all over the place about Niantic execs ignoring the burning server and pushing to launch in more countries anyway. I wonder if any of them actually tried to play as a user on the public servers and spent hours trying to logon and it failing, or locking up soon after for a week.
Not to mention they never even got the original tracker functionality (1 footstep, 2 footstep, 3 footstep for anything nearby) working again after that, they had to replace it with a lower load knock off where you just see what is around a certain location that isn't very popular. So not only did they not even keep login working, they cut features too.
...But, according to the nightly news, it was a tremendous success. The word 'ever' came up a lot.
PMG was the first successful overnight/viral planet-wide/client-side launch. People who had never heard of it saw it on the news and then visited their local app store in response.
And according to the googleblog, it took a tremendous expenditure of money, hardware, electricity, skills, and knowledge to pull it off.
Makes me wonder... Did some other game/app go almost global, but fall short for the want of those very resources described in the blog post?
Something for app devs to think about.
Google's post is weird because they seem to think the game was a technical success. Google may have done great, it's impossible to tell from the outside, but the actual user experience is - or at least was when I played it - awful.
So not total-success, but very successful for the situation at hand.
(disclaimer: I work at Google but not on GCP)
I don't want to criticize their model too much, because it's obviously simplified for our benefit. However, it appears that their worst-case scenario was "What if we become the next bejeweled or [insert popular F2P game here]?" It's a ridiculous assumption, because Pokemon has a much broader appeal than any other casual game, cause the IP is so insanely popular, and the game still appeals to people who just want a casual game. I know it is a lot easier to get fired for spending too much money than it is for not spending enough, but it's a stretch to say their launch traffic was beyond imagination. Niantic should start looking for new analysts now if their current analysts honestly thought this traffic was outside the realm of possibility.
I don't consider the server issues to be much of a problem though. It's hard to ensure everything will work perfectly under that kind of load. You have to accurately predict who will be playing, how much they will be playing, how they will interact with the game, and so much more. However, I do think they need to figure out their communication with the fan base. I know that there will be a vocal portion of any constituency that hates everything. That isn't a good excuse for communicating poorly. Good communication will help almost every relationship.
There's no Y axis on the chart, so we don't know exactly what their estimate actually was. Regardless, I'm pretty sure Pokemon GO exceeded any reasonable expectations of popularity, even accounting for the brand and marketing efforts behind it.
From my own experience, lots of people that never engaged with mobile games before started playing Pokemon GO within days of its release. My entire extending family was playing the game. Local bars have become arenas for Pokemon fights. The adoption of this game was absolutely crazy.
So even given the scaling problems, the features they had to remove from the game, and the bugs they introduced, I think this is still a solid win for Google CRE.
For example, one of the highest estimates of daily US users I saw was 25 million. That's way more than I would have guessed off the top of my head, but it's no where near 10 times what I would have guessed the "worst case" scenario would have been. Pokemon Red and Blue sold 9.85 million copies on gameboy alone. My worst case scenario would assume that literally everyone who played the game as a kid would want to check out the game as an adult. Our worse case is massively more accurate than theirs just using the Gameboy numbers alone, with the assumption that each cart sold was only played by one person.
The assumption that people who have never played mobile games before wouldn't be interested in Pokemon is kind of crazy as well. I can't think of another mobile game that has as much name recognition before it released as Pokemon go. It's a beloved franchise, whose popularity spans generations. On top of that, cellphones are far more ubiquitous than the gameboy ever was. There are people who are playing Pokemon go who have never been in to mobile gaming, but I doubt the number of people playing Pokemon Go who have never played another video game in their life is much, much, much smaller.
We had our team building event for my company at an barcade last year. Every single high score was set by someone older than 50. Literally every single person at the company had played Ms. Pac Man before, including some people who literally have never owned a cell phone. The people who don't own a cellphone are obviously not playing Pokemon Go, but video games are not niche, and have not been niche for a long time now.
I will give Niantic massive credit for recognizing how much of a problem this was going to be immediately. I'm sure I would have under-allocated resources on launch day too, and I don't see myself handling that issue as well. Still, there's no way in hell what actually happened would be 10x worse than my actual worse case.
That was two games, and that was "by the end of their run".
>The assumption that people who have never played mobile games before wouldn't be interested in Pokemon is kind of crazy as well. I can't think of another mobile game that has as much name recognition before it released as Pokemon go. It's a beloved franchise, whose popularity spans generations. On top of that, cellphones are far more ubiquitous than the gameboy ever was. There are people who are playing Pokemon go who have never been in to mobile gaming, but I doubt the number of people playing Pokemon Go who have never played another video game in their life is much, much, much smaller.
You're aided by hindsight. PKGO grew faster than any prior mobile game or any previous pokemon IP, by a large margin.
That makes the article a lot less interesting and worthwhile.
So yes this is a PR piece. Not very technical at all.
That's pretty misleading - I believe Google's parent company still own part of Niantic. So other customers shouldn't expect the (implied) same access to Google resources.
Alsop Louie Partners
Cyan Banister
Lucas Nealan
Nintendo
Pokémon
Scott Banister
You & Mr Jones Brandtech Ventures
Or in the vernacular of youth: "This is fine. Everything is fine" as a scaling graph.
http://www.gamasutra.com/view/news/168799/Scale_Something_Ho...
The launch in Europe was a catastrophe imo (constant crashes and freezes). I don't know how much of this is to blame on the cloud infrastructure but I suspect it's not nothing. I feel they didn't provide nearly enough infrastructure given the data they should have had from Australia/USA.
All that being said I think they smoothed out everything and the system seems to be running very nicely now given the scale. It's certainly a positive engineering tale overall.
[1] https://www.quora.com/What-server-infrastructure-is-Pok%C3%A...
Interestingly, the one time they did decide to communicate was when they announced that they banned a bunch of third parties from accessing their server[1]. Of course, just like in this post, they show a graph with a missing y-axis which tells you very little about the traffic they actually received.
It's surprising that this wasn't mentioned in the Google blog, since according to Niantic it was thanks to this ban that they were able to launch in more regions.
Being this type of site I wonder if a lot of the employee's at Niantic have accounts and could be down voting?
Either way it would be nice to hear why other users are down voting these because the users comment is within the guidelines.
I'm most curious about how they identified issues. Obviously crash reports are a thing, maybe traffic to a login error page, twitter complaints etc...
I wonder how they managed prioritizing issues.
I was previously interested in the game but Niantic has shown zero interest in being responsive to the community's concerns.
Especially a success that they went out of their way to avoid by creating a new battle system instead of using what's already there.
Even with the servers stabilized the game has plenty of other issues that need addressing. The "nearby Pokemon" window is still broken and shows no sign of ever getting fixed. People in suburbs and rural environments are still stuck playing in extra grindy hard mode. The distance tracking has a speed limit that is way too slow (10.5kph) and noisy so it largely fails if you jog and or get on a bike. The gym battles are comically out of balance (slow defensive Pokemon are blatantly overpowered).
They shouldn't be hard problems to fix but Niantic doesn't seem interested. They toyed with a half-fix to the nearby Pokemon window, but only deployed it in their hometown and then seemingly forgot about it. It's clear the client has enough data to do the launch style footsteps (see: all of the PokeRadar apps on the store), but for some reason Niantic has no interest in re-enabling it.
I've always suspected that was because they quickly realized that getting pokemon to not spawn in dangerous spots was an insurmountable task, so having tracking that encourages players to go to those locations was asking for even more lawsuits that what they face currently for that issue.
edit: added quote I was responding to
However this is all irrelevant considering they spent a month or two deciding to ban these devices and it only took 3 weeks for new scanner apps to pop up that potentially will never be brought down [1] the owner claims they will never be taken down now in his posts which makes me wonder if he is balance loading thousands of accounts to prevent Niantic from banning his application from working - which really it was bound to happen eventually just needed the time for some to write out the necessary code and figure out a quick way to authorize and potentially keep creating accounts.
In all honesty I believe the ban on rooted/jailbroken devices was really to remove people who were not contributing as much money to the game. In my experience a rooted device usually signifies the user wants to pay less for/in apps.
Of course many users need rooting for backing up, tweaking the interface for better performance, but in the end most rooted users did this method to install hacked apps or circumvent some restriction that they would normally have to pay for (such as wifi tethering).
It is silly and I am more then just frustrated that I had to spend hours getting Magisk installed to keep playing but in the end they are a business and the main focus is making money.
edit Apologies I must have read over you mentioning the Magisk method!
edit2 also I have a lot of friends who bought phone's pre-rooted so it goes to say it's not really a smart maneuver on Niantics part to just assume people are rooting devices purposely.
Eh, it's too much of a pain to hide my root. I'll never play this game again.
You could make an offline Pokemon Go version and not notice any difference.
Blizzard scaled using appropriate technologies for its time. i.e. sharding basted on "Realm" and enforcing active player limits. This scaling mechanism was used by every MMO I knew about until EVE.
EVE pioneered the modern MMO. It is no longer acceptable for an MMO to be sharded by server. Even WoW has been aggressively trying to scale its architecture and infra such that it can re-connect these shards into more cohesive worlds. With the most credit to them, they have a lot of technical debt in this regard, but the fact that some servers are labeled "full" with wait queues to join, implies that they still don't have the technology to scale let alone connect all their servers[0].
[0] Note, there are also other reasons (i.e. in-game economical and political reasons) why they are not aggressively connecting more shards.
I'd wager that at any point in time WoW dealt with higher player densities than Pokemon Go, realms or no realms. Certainly higher data densities with VASTLY higher latency requirements.
Of course in EVE, when the whole Python shambles eventually crumble under the load, they just slow down time. I also think there is some Excel spreadsheet you are supposed to fill out prior to a battle so they can deploy some more Cython before that epic MMO experience.
(I think EVE is great, it's just also a hilariously convoluted Python hack sending serialized objects rivaling XML in complexity over TCP. Not much leverage in that for other game types.)
How much are you willing to bet? I'd definitely take that bet.
> One system cannot scale over more than one server.
In this regard, Blizzard's dynamic realms are really interesting. Ideally, we can move to a world, where even single "systems"/"zones"/"areas" are able to scale horizontally.
I have zero experience with it, so I don't know if it can practically scale to solve this problem, but http://www.paralleluniverse.co/spacebase/ has a really cool set of features that MMOs could take advantage of.
What it probably shows is that using the cloud isn't a magic bullet to making you scale indefinitely, you need to work on your code to decouple from the infrastructure too.