Comment by fencepost | Hacker News Reader

Comment by fencepost | Hacker News Reader

Just as glad I didn't get a ticket ($20? sure. Scalper prices? Heck no) and waste the time. This whole thing is going to turn into a real fiasco for some time due to the number of people who bought resold passes (so the original seller sold it and now gets their original $20 back), people who traveled (I know many got hotels, not sure how many flew in), and questions of how they're going to get that in-game currency to people (reportedly some are still in line to get in, while others are waiting in a not-quite-as-long line just to get out).

Each person attending has a uniquely-numbered wristband pass, and apparently on entry they're getting an envelope with a patch and a unique-to-them QR code which needs to be scanned after spinning a special "Pokestop" and effectively checks you into the event. Since the game is crashing basically at startup for most folks, that makes it a bit of a challenge just to get in. Still, I suspect the in-game credit is going to be based on people who checked in with that code since the ordering process didn't involve providing a game login.

Also, I've seen some folks questioning why they didn't have WiFi set up, but I'm pretty sure that the way WiFi works would make that almost impossible anyway (2.4 would be a bad joke no matter what, 5GHz might be possible over most/all available channels, APs that can handle hundreds or thousands of simultaneous connections are rare and probably require major advance planning, AP broadcast power would probably have to be quite low to allow large numbers of APs in a physically small area, etc.).

A company I used to work for specialized in exactly this case - high density WiFi. They have an 8 radio / 24 antenna AP: https://www.xirrus.com/product/xr-4000-series/

You can use 6 20 MHz channels in the 5GHz band and 1 or 2 channels in the 2.4 GHz band, on each of these multi-APs, and you can get much smaller "cell sizes" (low power signal, smaller signal area) than cellular providers. You can bump clients from busier channels on to less busy ones to force better balance.

WiFi drivers for clients and APs are a big pile of bugs - the resilience (retries at lower rates, chip-timeout-resets, etc) covers a lot of them up. You can get more consistent throughput and latency by digging in and fixing these bugs (and working around bugs in most clients). Most good WiFi APs are debugged until they work pretty well for the "typical" home or office use-cases. But the Xirrus APs (and some competitors to some degree) are debugged for the "high density" use-case - multiple hundreds of users per multi-AP, and maybe 8 to 30 of these multi-APs in a large conference hall or stadium.

You have to know what you're doing. It is expensive. Niantic has a hugely popular service. They're making money. But they never seem to know how to handle the massive scale that was predictable after the first month or so. They sold tickets, they knew how many were coming. There are people who know how to handle that. Niantic just didn't feel like doing the research to find out how or who to handle it. Their systems fall over again and again and again.

There are definitely companies have the kind of wifi infrastructure you are talking about who offer rentals to the people who organize these kinds of events. Here is an example of one[0].

Wifi is often deployed at large events (PAX East for example) and it is more then feasible if the company is willing to pay the money for another company to do it.

However, their server infrastructure is another question entirely. In today's age, making scale-able network services with very little downtime is not something that requires a dedicated networking team and hundreds of millions of dollars. I can't imagine they have very many good excuses to why they continue to allow this to happen other than that they don't value investment into their infrastructure enough (despite being a company completely dependent on the internet)

The problems that they are experiencing are ones that have caused problems and had solutions created for them for years. The just seem to be unwilling to properly invest in the solutions to fix them, which are cheaper than ever (considering the complexity of the issues in the grand scheme of things).

[0] https://tradeshowinternet.com/solutions/event-organizers

They may also be having load-balancer issues assuming they have front-end servers that are checking whether accounts are supposed to be directed to a specific set of servers. Assuming that they normally load-balance based on geographic location, the check for specialized server connections may be something that they never actually load-tested before after confirming that it worked for their staff in testing.

I asked earlier and have asked again whether anyone at the location but not officially attending is having better luck connecting, that would help to narrow it down between carrier issues, Niantic server issues and Niantic server-for-this-event issues

Edit: They expanded the area for special/regional spawns to 2 miles around Grant Park but still only for people who "checked in" at the park, and people who've left are reporting much better luck when they're away from the park. That may indicate that most of the issue was insufficient networking.

People in a stadium aren't going to be quite as focused on their phones, but lots of people in a tight space isn't an unusual occurrence for networks.

~15k people in a small section of Grant Park all trying to play a somewhat data-intensive online game that requires constant communication with servers is a VERY different activity stream than people in a stadium. Might be closer to think of those people in a stadium all trying to stream video of the event.

Is the game that data-intensive? What data is there to send/receive that you're comparing it with streaming video?

how many people are live streaming a concert at a given moment? maybe 5-10 on periscope, in addition to maybe 2x that number on facebook. definitely not 15k.

The game uses google's protobuf protocol which makes communication more efficient and the game updates (I can only tell from <= 0.35 analysis) all 10-30 seconds like loading pokemon spawns and map entities of the current map area.

The most expensive data in 0.35 was:

- Pokemon models (They were dynamically loaded but cached then)

- Pokestop images (Server sends image url and client searches for a chached version)

- Syncing the player state with the client (Inventory, rewards, egg states..)

- Updating the map (Future pokemon spawns, pokestops/gyms in view..)

If a player is idle and doesn't move, just keeps farming a single pokestop and battles sometimes, then it should be relatively low cost since the game doesn't use a realtime communication (It's actually HTTPS based lol).

The pokemon models are loaded over the network? Whatever for? Seems like a big waste of precious data

So you can add/update them without doing an .ipa/.apk update ? Seems really convenient, and a one time deal if it's cached.

Last year I developed a server emulator for the game - On the server's first run, it cloned all models from the official servers to a local folder. The download urls and the models itself were encrypted, also their urls got disabled after ~24 hours. Other informations like how big a model gets displayed inside the game world was declared inside a GAME_MASTER file. When connecting to the custom server, the client asks for the GAME_MASTER file and the models as needed. It was pretty fun to edit these and send the players custom models like a GIANT snorlax or a red pikachu. I'm not 100% sure, but I think the GAME_MASTER file even contained a "model signature" to trigger a forced model update, so it's possible to overwrite already cached models. In the first weeks Niantic broke the model of Starmie (it turned full black) and I'm pretty sure they fixed it without a full game update. You can checkout the emulator project here: https://github.com/maierfelix/POGOserver

I haven't actually measured anything, I know when it first came out there was some concern about people possibly hitting bandwidth caps. I think part of the issue may be overhead - it's regularly communicating back to the servers with current location, and getting back a list of coordinates for stops within a certain distance (with a bit of other data), coordinates and other data for Pokemon spawns, etc. It's also using Google Maps data to draw a map of the terrain including roads, water and building outlines, so all of that is also something that changes as you move around. Finally, people are also supposed to "spin" Pokestops and gyms, each of which also has a photo of the piece of artwork, etc. that the stop is located at. Along with the map data, those photos are not preloaded in the app so they have to be downloaded at least once. I don't know how much caching of those images is done - hopefully some, but if they're doing little or no caching then that's yet another chunk of data to pull down.

Looking at the screen and a map, it looks like the game renders an outline for every structure within 600-650 meters of me, so for an area with a lot of structures that could be a fair amount of downloaded data depending on how good their caching is. If their download process isn't good at handling missing data or transmission errors, it could also be pretty vulnerable to problems on a congested network.

Slight correction, as far as I know they were using OSM, not Google Maps.

They use Google for the base map everywhere except South Korea, where they do use OpenStreetMap.

There are rumors that some game features (biomes and such) are influenced by features in OpenStreetMap, but no clear statement from Niantic, just a bunch of weak correlations (like OSM having a path where a bunch of people go and try to play the game...).

There was a big problem where people were manipulating biomes in the game by messing with OSM, so unless there's some sharing going on with Google..?

Base map vs hidden game features. Outside of Korea the base map is 100% Google.

The hidden game features, who knows. Lots of people are very convinced that OpenStreetMap features have an impact. I haven't really seen anything all that convincing, just lots of correlations that could be have other explanations (game analytics, other datasets, etc).

Apparently the game is a few megabytes per hour (this makes sense to me, the map isn't very detailed, a lot of other stuff is very cache-able).

So I think the experience in stadiums will translate okay. Of course it won't be the same, but it's tens of thousands of radios in high proximity.

Stadiums have issues with this too. I was at Marlins Park for the All-Star series. During the Home Run Derby I had press access, which had special Wifi credentials on a completely separate network.

It was down for the entire game. A lot of hotspots popped up in the press area.

The during the actual All-Star game I tried the regular stadium Wifi. It was down for most of the game.

T-Mobile was up for most of the game, so I was still able to publish photos and read people's reactions. A good thing as T-Mo was the main sponsor for the games.

It's a challenge, but a very well-characterized one that all enterprise-grade access points are built to handle. Lots of access points, with radio power turned way down to prevent neighbor interference, and channels selected to avoid overlap with neighboring APs.

Shameless plug - my ex-employer Meraki is particularly good at supplying these systems for events, since they have automatic channel-selection and good tooling for selecting power levels, and let you set up configurations at your leisure that the APs will use as soon as they get an internet connection and fetch the configs from Meraki servers.

Reply on news.ycombinator.com