Update on Maintaining and Running the Pokémon GO Service | Hacker News Reader

Update on Maintaining and Running the Pokémon GO Service

pokemongo.nianticlabs.com

Looking at the title, I was expecting them to share the juicy details of how they handled the huge amount of traffic, the hacking attempts and the engineering problems they are facing. Instead I was met with some sort of PR attempt to explain the downtime.

Not necessarily downtime, but why they "killed" a very popular third-party tool to show real-time locations of Pokemon.

I mean the tool basically defeated the point of the game.

Really? You think wandering around was the point of the game, as opposed to actually catching the Pokémon?

Depends on how you look at the game. The point of the game for users is to catch pokemon. The point of the game for Niantic is to have users wander around, so they can make money via sponsored locations.

A winning strategy must accommodate both.

I mean, if you reverse engineer an api for "where is pokemon X", you may as well just reverse engineer your GPS device for "where am I standing" to move your in game character to right where the pokemon is.

Then you can do a for loop and catch all the pokemon in 30 seconds.

Actually catching the pokemon is a 0 skill item. Its literally dragging your finger across the screen.

The entire game is the exploring the real world part.

The point of any game is enjoyment. I found Pokemon Go much more enjoyable with the aid of Pokevision than I do without, since at least then I could find interesting pokemon. Since the last update, I chose to stop playing entirely. It's just not very fun.

The tools gained so much popularity because the in-game tracker stopped working. That was part of why players were upset with killing the third party tools too. There was then no way to track pokemon at all.

Godspeed Niantic dev-team!!! Really despise the negative press you're getting. Don't think anyone in their wildest dreams would have thought PoGo would have taken off as well as it has and now they are stuck between a rock and a hard place:

Do they suffer the rage of entire countries, and potentially their backers by delaying launching in territories to fix bugs?

OR do they suffer the rage of existing users by delaying bug fixes to keep to their original schedule by launching new territories?

All the while, they are crippled by scaling issues they never dreamt of having and managing cheating/additional traffic from the meta-ecosystem of trackers.

Have no doubt the team are working doubletime and overtime as it is, a shame that redditers are unable to see this. So keep it up Niantic - thanks for bringing us the AR killer app and for showing that there are still opportunities for huge apps to launch![0]

[0] https://www.appannie.com/insights/mobile-strategy/pokemon-go...

PS if you guys need more devs, do let me know :P

EDIT: downvotes - really? The post scriptum was obviously a cute joke. Or are the downvotes towards the Niantic support?

Uhmmm I guess the downvotes are coming because no one believes that Niantic couldn't predict that the first mobile game(if we forget Pokemon Shuffle for a second) in one of the biggest franchises in the world would be this popular, and I feel like it's quite a popular opinion that the app was released at least 6 months too early, it's poorly optimized and full of bugs. Not to mention that Niantic cannot handle PR, they don't even have a Pokemon GO-dedicated rep yet, they are looking for one. Personally I know several people who played this game every day after it came out, now this number fell down to zero, after a while there isn't much to do and the recent updates made the game worse. I have stopped playing after few days because every other catch the app would freeze and I would have to restart the whole thing. Plus the fact that it's obliterating the battery just by being open is unacceptable - obviously it's just a personal opinion so take it as you will.

It's also unfortunate that the game handles failures so badly. It hangs and behaves strangely in a number of ways instead of showing an error message.

Yup. For instance, the whole catching interaction part behaves as if the code was full of shitty hacks that couple UI and local game state with successful and timely API responses. The kind of hangs you get with e.g. your Pokéball stuck mid-air because of API response problem seems like a type of bug you'd actually have to go out of your way to make in any kind of sane design.

Fine - that's fair enough, there have been some questionable actions (or lack of actions) especially from the PR front, but these aren't things the average dev can influence and the lack of a PR head is absolutely nothing the devs can do anything about.

That said, however, the devs would still be affected by the feedback, they are almost certainly are working crazy stressful hours and the general backlash towards them is directed towards engineering.

My point being that we have no idea what it's like whatsoever inside Niantic, but you can be pretty sure their employees are working frantically right now, and the flak directed in their general direction is pretty unwarranted considering they've brought out a free video game that loads of people are enjoying.

The product might well have been released 6 months early, but isn't a more positive viewpoint that we should be grateful for having a version of Pokemon Go available early? 1 month ago, we didn't have pokemon go and our lives were essentially the same. Nothing is forcing us to play this game, it's bad, don't play it. If you have a problem with the management of the product then explicitly call out the management team but let's not forget that humans built and are building this product and are mostly following the instructions of their corporate overlords and take a moment to thank them for their hard work.

How about putting some actual efforts into proper engineering before launch, stop blaming third parties for your 6h+ downtime and proving ban effectiveness with a graph missing its Y axis?

The only edge Niantic have over their fanbase is their gis database. Lack of communication was appalling, execution and reactivity too. I'm surprised nobody started a "private server" implementation of the game, yet.

You seem borderline vitriolic about this.

Nobody including Niantic would ever have dreamed that Pokemon Go would be as huge a sensation as it has been. Nobody. And they have smartly realised that this opportunity is potentially fleeting and so they need to capitalise as quick as possible i.e. grow into other regions.

To expect them to do accurate capacity planning is ridiculous. Likewise it is ridiculous to assume that they can react any quicker than they can. It's very hard to scale quickly whilst still maintaining a high level of uptime.

This still doesn't explain the graph without scale on Y axis and especially one without a clear 0 label. This is like stats 101, and not doing so implies either utter incompetence or intentional attempt to mislead the audience. There is no reason whatsoever to omit the scale and 0 label on a graph if the data actually supports the argued thesis, so most of the time lack of those elements means you're being lied to.

More likely, the nominal values of the y-axis are confidential, and they just want to show the % reduction in traffic. I do assume the crossing with the x-axis is actually at 0.

You assume. The graph itself doesn't justify that assumption. If you want to lie to someone using a graph, this is exactly the assumption you're hoping they'll make while reading it. Not labeling axis and not starting the graph at 0 are two most basic techniques from the book of Dark Patterns.

Of course this could be an innocent mistake made by Niantic. Could be. But given that PR statements in general tend to be manipulative, I'm not willing to trust that. If they didn't want to reveal their resource capacity, they could have labeled the Y-axis with percentages.

They could show that by placing a 100% at the peak, and marking the axis relative to it.

Given that the "gameplay"/code of the game itself also implies either utter incompetence or intentional attempts to mislead the audience, I find myself unsurprised by this. :-)

Not every person who works at Niantic is working on capacity planning and programming. There are non-tech and marginally-tech savvy people who could... Oh, I don't know, draft reasonable press releases? Post-mortems when the service is unavailable or broken? Actual explanations of their actions?

Like all good companies do in the tech/gaming space?

I am, indeed, quite angry towards Niantic. They made a very fun and addictive game, I'm not denying that.

However, they behaved in a way that really makes it difficult for us, players, to support them. They weren't ready for their success, sure, but they could at least bring some explanations on why would they ship without a trade system, why are they struggling with traffic and downtime, why would they delay bug fixes for more than 3 weeks, why they decided to remove one of the key feature of the game, and why they decided to ban third party apps made by people trying to fix their mistakes (and not with a broken graph!). I remember terrific launches from Blizzard, but even with all issues combined from Wow, Diablo and Hearthstone, this is nothing compared to Pogo's. I care about this game, because it's different from the long trail of duplicates we've seen for a long time; but, I really wish, Niantic, as a company, was more mature.

I don't think these are very interesting questions.

Why did a product ship without features? Not enough time to make them.

Why are their servers struggling? The game is significantly more popular than they expected.

How much did you pay for the game?

Pokemon Go does have ingame micro-transactions.

Every update of the game I've encountered seems to make things worse. With the tracker debacle and a few new bugs that come up when come up when catching pokemon, I've almost stopped playing. Really sad where this game could have been as opposed to where it is. Also, it seems these guys are putting more effort stopping third parties fill in holes in their game rather than actually fix/improve the game itself.

They banned third party clients from accessing their API.

From the chart it looks like third party API clients were generating two thirds of total traffic. That is a lot.

On the other hand, this has sparkled a huge community effort to reverse, understand and bypass the implemented protection/fingerprinting. https://www.reddit.com/r/pokemongodev/comments/4w1cvr/pokemo...

I'd be really interesting the see that graph extended to to left - what was the traffic like before the 3 paws "bug" was first introduced.

I definitely feel that a significant amount of the drive to use the 3rd party services was due to the fact that you couldn't realistically hunt down any of the pokemon that show up on your radar.

Yes, the lack of a functional Nearby feature - either integrated or with a third party app - really affects motivation while playing.

I was out walking today and saw an unknown Pokemon silhouette in my Nearby list. All I could do was shrug and hope I'd run into it (I didn't). Previously I would have gone significantly out of my way to find it, adding mileage to my walk - which I understood to be one of the design objectives of the game.

Not only must the three steps have been buggy, the whole "nearby" list is useless and/or wrong. Two friends walking together, with the same exact GPS position on the maps, can have vastly different nearby-lists, sometimes without even one entry in common. How is that even possible?

It could be that, since the nearby listing holds only 9 entries, that there are more Pokemon nearby and the query to display them is unordered.

http://i.imgur.com/RLc4i9V.png

That's assuming the the y-axis starts at zero, it also lacks numbers for the "spatial queries". This graph doesn't actually tell us anything, besides that there was a drop in traffic after blocking third party access.

> From the chart it looks like third party API clients were generating two thirds of total traffic. That is a lot.

Around the time they banned API access they removed distance evaluation from the software (after having had it broken for a few weeks), initially the game included three distance ranges shown as steps in the UI, there's no distance information left whatsoever now.

> From the chart it looks like third party API clients were generating two thirds of total traffic. That is a lot.

True, but I don't know if that's a fair metric. The Third Party API clients were the only effective way to play the game (because unlike real Pokemon games, "Go" focuses almost exclusively on the act of finding and catching, instead of training. And finding and catching doesn't actually work in their own client)

Effectively, Ninantic has "gotten traffic under control" by banning all non-beginner-level players and play strategies. It's like if YouTube decided to "manage traffic" by removing every video from PewDiePie.

Your analogy might be fair, but to continue it all the way, it would be like them turning off ability for others to upload pewdiepie videos so that they have the resources to give him his own you tube branded channel.

Which we know, hasn't happened yet, but it's an important part of the direction they are going.

They don't cache images (of Pokestops) client-side. This has got to be one of the bigger inefficiencies of this client. Aggressive client-side caching can relieve a lot of load on the server.

Well, perhaps. But presumably the images from the pokestops can also be delivered by a CDN with a simple key lookup, and can be outside all of their spatial databases and primary logic servers.

The graph they showed was of spatial queries. That's not going to be affected by lookups of images for pokestops.

This was my biggest surprise when I started playing the game. There are two pokestops outside my office which were the ones I most often visited. And every single time, it had to load from scratch despite absolutely zero changes in image or information. Can't understand why they wouldn't cache that on my phone. Little things like that add up.

That's only on the network side. Playing the game you get a feel that the entire codebase is a cesspool.

Everyone I talk to who's also played Ingress sees history repeating itself. They've experienced the same bugs, lack of communication, shutting down services making the game better instead of improving said game and whatnot.

I really, really wish Niantic fixes its technical, management and communication problems. The game has so much potential but at this point I lost almost all hope they'll be able fully exploit it.

Seems like they started authenticating a key (unknown6) that was previously being sent but not getting authenticated against.

A lot of bot writers are trying to break that authentication. Given that people are paying others to 'walk' their pokemon there is real money in faking GPS movement in the API.

Or go middle way and spoof the GPS radio signal near the device: http://www.rtl-sdr.com/cheating-at-pokemon-go-with-a-hackrf-...

That is extremely overpowered, all you need is a rooted device with Xposed framework and a GPS spoofing app.

I was (half-)joking :)

This is illegal in most countries and may end up in a criminal court.

I'd like to see some numbers in that graph.

That graph is a joke.

http://i.imgur.com/HC9PEx0.png

Spatial graph, no scale

Reply on news.ycombinator.com