Pokémon Go Fest attendees to get refunds as technical issues break the event
techcrunch.com
techcrunch.com
I used to be angry at them for ruining Ingress (one of the best games ever made) but nowadays I am just amazed at the level of sheer incompetence.
They managed to learn _nothing_ from 4 years of Ingress and I think now it's time that the whole upper management is fired so that the company can start fresh.
Looking at my history (pretty active player) I've used about 184 MB since June 24th (I am not as active on weekends so for the sake of this example I actually played 18 days out of the month averaged about the same per day). On average that means about 10.2 MB per day of playing (looking at the graph, however, I would say I used about 20 MB the day of the event as I was out for about twice as much time that day).
I don't think that is considered a lot
There's something different about what they were trying to do with Pokemon Go that made it suck. They don't have experience with these yet. And if someone made long distance travel plans, just getting your entry ticket refunded doesn't begin to make up for your loss.
First off, they had everything in one place. For those who haven't played Ingress to that point, anomalies in Ingress usually take place with a primary and two secondary locations in, roughly, Asia, the Western Hemisphere, and Europe. This means that people aren't usually traveling from far away (which probably would increase overall participation) but also that each site's numbers are lower. I figure overall anomaly attendance on a given anomaly weekend might well rival that of this event -- but they're spread out in geography and time.
The second aspect is that Ingress has much more encouragement in-game of local organization. I understand quite well why Pokemon Go lacks in-game chat, but it saps local organization. Locals play an important, perhaps critical, role in Ingress event organization, and provide Niantic with free boots on the ground to figure things out. I know there is some local organization going on in Pokemon Go, but it's forced to be based on random encounters, and folks are still unsure whether to organize on team lines or just geographically. (With the current setup, I'd say geographically -- if there's anything to be learned from Ingress, it's to tamp down factionalism.)
The last thing, the one that I think would hurt Ingress equally? It seems there wasn't a limit on ticket sales. In Ingress some special extra-value packs that get you extra tchotchkes both virtual and physical exist, but the base 'free' ticket is unlimited. Similarly, here. They probably wanted to avoid any chance of scalping or the like, but imagine if they limited it to, say, a quarter the attendance?
There are people who understand event planning, there are people who understand how to build F2P mobile games, and I get the impression that exactly zero of these people were consulted, or, if they were, their advice was immediately ignored.
No, I'd argue Niantic may originally have been engineering driven, but once Ruthless cut them from Alphabet, they went straight for the greed play, developed with an IP that would sell billions with next to no effort, and are now tightening the screws on the player-base with Pay-to-Win gym tickets and events requiring as many Lures and Incenses as you can buy to continue extracting income.
UI developers? Testing? You have apparently worked with very different software engineers than I have! Most I've worked with have followed the "if users don't understand the UI it's their fault" UI design philosophy, and all testing is handled in production because no one wants to put emphasis on testing when they could be working on the next feature.
Of course, they should have been better prepared for the event, but they did seem to do better at the cleanup this time. So... baby steps?
Each person attending has a uniquely-numbered wristband pass, and apparently on entry they're getting an envelope with a patch and a unique-to-them QR code which needs to be scanned after spinning a special "Pokestop" and effectively checks you into the event. Since the game is crashing basically at startup for most folks, that makes it a bit of a challenge just to get in. Still, I suspect the in-game credit is going to be based on people who checked in with that code since the ordering process didn't involve providing a game login.
Also, I've seen some folks questioning why they didn't have WiFi set up, but I'm pretty sure that the way WiFi works would make that almost impossible anyway (2.4 would be a bad joke no matter what, 5GHz might be possible over most/all available channels, APs that can handle hundreds or thousands of simultaneous connections are rare and probably require major advance planning, AP broadcast power would probably have to be quite low to allow large numbers of APs in a physically small area, etc.).
The most expensive data in 0.35 was:
- Pokemon models (They were dynamically loaded but cached then)
- Pokestop images (Server sends image url and client searches for a chached version)
- Syncing the player state with the client (Inventory, rewards, egg states..)
- Updating the map (Future pokemon spawns, pokestops/gyms in view..)
If a player is idle and doesn't move, just keeps farming a single pokestop and battles sometimes, then it should be relatively low cost since the game doesn't use a realtime communication (It's actually HTTPS based lol).
Looking at the screen and a map, it looks like the game renders an outline for every structure within 600-650 meters of me, so for an area with a lot of structures that could be a fair amount of downloaded data depending on how good their caching is. If their download process isn't good at handling missing data or transmission errors, it could also be pretty vulnerable to problems on a congested network.
There are rumors that some game features (biomes and such) are influenced by features in OpenStreetMap, but no clear statement from Niantic, just a bunch of weak correlations (like OSM having a path where a bunch of people go and try to play the game...).
The hidden game features, who knows. Lots of people are very convinced that OpenStreetMap features have an impact. I haven't really seen anything all that convincing, just lots of correlations that could be have other explanations (game analytics, other datasets, etc).
So I think the experience in stadiums will translate okay. Of course it won't be the same, but it's tens of thousands of radios in high proximity.
It was down for the entire game. A lot of hotspots popped up in the press area.
The during the actual All-Star game I tried the regular stadium Wifi. It was down for most of the game.
T-Mobile was up for most of the game, so I was still able to publish photos and read people's reactions. A good thing as T-Mo was the main sponsor for the games.
Shameless plug - my ex-employer Meraki is particularly good at supplying these systems for events, since they have automatic channel-selection and good tooling for selecting power levels, and let you set up configurations at your leisure that the APs will use as soon as they get an internet connection and fetch the configs from Meraki servers.
Wifi is often deployed at large events (PAX East for example) and it is more then feasible if the company is willing to pay the money for another company to do it.
However, their server infrastructure is another question entirely. In today's age, making scale-able network services with very little downtime is not something that requires a dedicated networking team and hundreds of millions of dollars. I can't imagine they have very many good excuses to why they continue to allow this to happen other than that they don't value investment into their infrastructure enough (despite being a company completely dependent on the internet)
The problems that they are experiencing are ones that have caused problems and had solutions created for them for years. The just seem to be unwilling to properly invest in the solutions to fix them, which are cheaper than ever (considering the complexity of the issues in the grand scheme of things).
[0] https://tradeshowinternet.com/solutions/event-organizers
I asked earlier and have asked again whether anyone at the location but not officially attending is having better luck connecting, that would help to narrow it down between carrier issues, Niantic server issues and Niantic server-for-this-event issues
Edit: They expanded the area for special/regional spawns to 2 miles around Grant Park but still only for people who "checked in" at the park, and people who've left are reporting much better luck when they're away from the park. That may indicate that most of the issue was insufficient networking.
You can use 6 20 MHz channels in the 5GHz band and 1 or 2 channels in the 2.4 GHz band, on each of these multi-APs, and you can get much smaller "cell sizes" (low power signal, smaller signal area) than cellular providers. You can bump clients from busier channels on to less busy ones to force better balance.
WiFi drivers for clients and APs are a big pile of bugs - the resilience (retries at lower rates, chip-timeout-resets, etc) covers a lot of them up. You can get more consistent throughput and latency by digging in and fixing these bugs (and working around bugs in most clients). Most good WiFi APs are debugged until they work pretty well for the "typical" home or office use-cases. But the Xirrus APs (and some competitors to some degree) are debugged for the "high density" use-case - multiple hundreds of users per multi-AP, and maybe 8 to 30 of these multi-APs in a large conference hall or stadium.
You have to know what you're doing. It is expensive. Niantic has a hugely popular service. They're making money. But they never seem to know how to handle the massive scale that was predictable after the first month or so. They sold tickets, they knew how many were coming. There are people who know how to handle that. Niantic just didn't feel like doing the research to find out how or who to handle it. Their systems fall over again and again and again.
Can you imagine how stressful this would be? It's one thing to get disappointing feedback data on bugs, UX, etc. Standing in front of thousands of angry people screaming at you to fix your software this instant? Yikes!!
There are 2 solutions to this problem.
1) Distributing a web of much smaller cells, lower power so much less devices are connected per node.
2) DIDO pCell technology developed by Artemis. All antennas transmit and the waveforms from each broadcast interfere around each receiver to create a valid signal for that signal device.
Both systems require loads of calculations by the broadcasters to either manage handoffs from each mini cell, or to calalate valid distributed broadcasts per each receiver.
This type of event would have been a great opportunity for Artemis to show off their tech.
That said there's still a bunch of other stuff you have to sort out if you want to support that many devices in a small space.
Really? I thought CDMA was something only Verizon used for their network. I tried looking into it a bit prior to writing the original answer but I couldn't find anything other than a mention on the TDMA Wiki [1] which has a [citation needed]. Do 3G/4G networks across the world use CDMA as well? Because I know that in Sweden carriers are buying up large chunks of frequency bands to be able to increase 4G coverage, that shouldn't be needed with CDMA, right?
[1]: https://en.wikipedia.org/wiki/Time-division_multiple_access
Cheap-ish providers are worse - they usually don't run fiber to every tower because laying fiber is expensive, but instead only to one tower which then distributes uplink via directional microwave to nearby towers. Once that link gets saturated or has issues, the whole area is screwed.
Providers like Deutsche Telekom (or other former monopolists) usually have it easier as they have huuuge amounts of fiber and especially conduits in the ground, which means it's easier for them to wire up even small towers with real dedicated fiber uplink.
Cell towers get saturated pretty easily by crowds. Network providers probably did not account for the crowd, or even if they did, it wasn't enough.
I would assume a company like Niantic asked network providers to install additional cell towers beforehand, so my best guess is they tried their best but still failed.
The technical details are quite complex, and you would need to read up on GSM,UMTS,LTE,TDMA,CDMA, but the bottom-line is, the overall technology isn't good enough yet.
Edit: Expanding on this a little while I still have time to edit, and all of this is from a non-telecom person so it's likely I have errors in here. Still, this should provide a starting point for the history and what to look for to get more current/accurate info.
Most 2G was GSM which used TDMA (Time Division) - basically, each device connecting to the tower was assigned a "slice" of the available time and had to broadcast within that time slice. When the tower ran out of slices, some devices simply couldn't connect to it. Sprint and Verizon were the exceptions, they used CDMA which allowed many devices to broadcast at the same time and was able to decode data for/from each device based on the Code Division used to encode it. CDMA towers "breathed" which meant that their coverage area could vary based on how many devices were connected because the devices and towers had to keep broadcasts within particular power levels at the endpoints. CDMA allowed more devices, TDMA could allow larger coverage from each tower by reducing the number of time slices (which also reduced the number of devices that could connect). Notably, CDMA allowed devices to talk to multiple towers at once with reconciliation being handled by a smarter backend, while TDMA required more explicit handoffs between towers. Way back when this was why on GSM your call would simply drop, but on CDMA you'd simply develop a severe case of robot voice.
3G almost all moved to WCDMA, which I believe had many of the same advantages in density as CDMA.
4G moved to somethind different (OFDMA?), which I haven't read enough about to really understand but which is technically quite different from WCDMA.
Most LTE uses yet another protocol. On T-Mobile, this was noticeable when T-Mo "refarmed" its network, converting a bunch of frequency bands from 4G to the newer higher-capacity but incompatible LTE. I know a few people who had 4G phones on T-Mobile and were offered free replacement phones because T-Mobile had basically yanked the actual 4G network out from under them.
Basically, there's a lot out there, it has changed a LOT over the last 10-15 years, and many of the changes were actually complete replacements rather than incremental upgrades.
In Germany, ordinary political demonstrations (from my experience, everything from 2k to 80k is enough) are sufficient for O2 service to totally break down, even in the center of major cities.
The Oktoberfest in Munich has similar problems despite operators putting up dozens of small BTS, but then again, it's up to 350.000 people concentrated on maybe 200.000 m² and it's really hard to get them all serviced good enough to live-stream their Brathendl and beer Mass to Instagram...
https://en.wikipedia.org/wiki/Mobile_cell_sites#Cell_on_whee...
What good are those towers if thousand people can connect but cannot use any internet because their backbone cable is so slow. I once was at a festival in Germany where they had such measures but lacked backbones (I had full signal strength in a remote array but the internet was unusable slow).
Not every town has fiber optic to support such a surge of people.
Those are good for a couple of gbps each and can go anywhere in a city that's got line of sight.
All registered attendees will soon receive an email with instructions on how to receive a full refund for the cost of their ticket. These instructions will be sent to the email addresses associated with your Pokémon GO account.
All registered attendees will receive $100 in PokéCoins in their Pokémon GO account.
Special Pokémon, Eggs, and check-in PokéStops appearing during Pokémon GO Fest have had their range increased to a two mile radius surrounding Grant Park through Monday morning, July 24. These Pokémon and Eggs will only be visible to Pokémon GO Fest attendees who validated the QR code they received when they entered Pokémon GO Fest. Attendees who were unable to validate their QR code during the event can do so through the special PokéStops through Monday morning.
All registered attendees will have the Legendary Pokémon, Lugia, added to their account.
Lugia? That’s some damage control right there.
WiFi is a thing and should he able to handle it just fine. Heck, they could even block anything but PGO and get away with low bandwidth too.
When voulenteers can manage to get WiFi working for thousands of nerds at indoor LANs... Let's just say that Niantic should be able to avoid this with a little bit of planning.
This is the company running one of the most unstable and buggy games I have ever played, so I would expect no less. At least the game is very fun for a lot of people... When it works.
If your comment is based on this event, it seems unfair: Who else has successfully done an event with similar bandwidth requirements before? Also, my understanding is that the cellular network is the problem, but you might correct me if I'm wrong.
Besides, Blizzard had spectacular problems in the past with WOW launch events, simply because no one had ever done something of that /scale/ before.
20k people in close proximity uploading and downloading constantly without breaks? What specific event is that comparable to? Music festivals don't have all attendees constantly using their data network.
Did they do anything at all to anticipate that load? It seems not. "We tried and failed" is understandable, but simply inviting 20k people and expecting them to play a mobile game is monumentally foolish.
In fact, after playing for a bit, I turned off the AR because of loading times.
It was a global sensation for a reason and despite it's extensive flaws, it was one of the funnest gaming experiences I've ever had. Not because of what happened in the game, which admittedly was beyond terrible at every level from core gameplay to polish and UI (I've seen and worked on better better at 24 hr gamejams. I have no idea why Nintendo didn't step in and take over development after the craze happened) The gyms closest to me were located in a popular shopping mall that had no shortage of players. It led to a few new meatspace friends for me.
There was something meta-magical about seeing someone walking down the street glued to their phone while you're doing the same, both of you realizing that you're trying to catch some rare pokemon. Often enough, it led to some friendly social encounter, providing just a glimpse of what AR gaming could be.
It also inspired the dev community to make all sorts of apps and mods that addressed many of the games shortcomings.I left after Niantic dismissed the dev community because the game could've been so much more with the power of the crowd. Instead, Niantic preferred to keep the game shitty so that they could extract profits from the parents of 8 year olds.
I was doubly disappointed when it was revealed that the Switch, a mobile game console, would have no GPS chip. Nintendo could have pioneered AR gameplay like they did with 3D gaming. Maybe that's expecting too much from a company that doesn't even get online gameplay.
It only lasted two weeks, but that's the standard shelf life of most games that aren't endless grindfests.
I personally never got into the game, while my cousin was totally into it, but where I live (which is a medium city for Europe), the fad has totally passed in less than 6 months after launch.