Comment by edent | Hacker News Reader

Comment by edent | Hacker News Reader

I work for a telco (I don't speak for them, usual disclaimers apply).

Everyone gets really ranty when operators delay firmware updates. Yeah, some of them delay to force you to buy a new phone, or to stuff the ROM full of bloatware. Most, however, do extensive testing to make sure that customers don't ring up call centres making angry complaints.

Carriers (by and large) test the phone to make sure that the network portion operates correctly. They make sure that it complies with local regulations and that it's not going to interfere with the network. They also test that basic cellular functionality works. Many go further and check all standard operations of the handset.

It's been several years since I was responsible for device testing, but here's a list of problems that I remember from big name manufacturers (Nokia, HTC, Sagem) on their "final" firmware.

- 999/112/911 features not working.

- Reboots when receiving a text starting with a 0.

- Factory reset when receiving an MMS with more than one attachment.

- Inability to switch from 3G to 2G on handover.

- Caller ID always picking the first name from the SIM.

It was, frankly, an unending parade of shoddy workmanship. We'd file reports, wait for an updated firmware, test it again, see that it had more bugs, refuse to ship it.

Frankly, a few "enthusiasts" whining that they don't have a 0.1 firmware upgrade is a sensible price to pay compared to millions of customers storming your shops and demanding refunds.

Google has an atrocious attitude to customer service - it's part of their business model https://shkspr.mobi/blog/2013/02/googles-customer-contempt-c...

Apologies to the sibling posts praising this for a different perspective, but this post goes way too far in the other direction. Yes testing updates is very important. But by "operators delay firmware updates" do you mean never release updates? By "some of them" "stuff the ROM full of bloatware" do you mean nearly all of them? Telcos also love to write and ship their own OS UIs which are themselves full of shoddy workmanship, then never release updates for many of the phones running them.

Listen, it is a shame the telcos get totally thrown under the bus when there's an entire supply chain full of issues, but I don't see what that has to do with this bug except in the sense that "there are bugs" not caused by the telcos.

It rather depends where you are in the world. UK operators are fairly good at releasing updates - but a lot depends on how many units the carrier sold.

In the EU there's a trend away from bloatware. But, again, depends on your country / market.

The real issue here is that GSM, MMS, etc are archaic and hard to code for. The industry's fight against VOIP seems to be the real issue here. Instead of all this complexity, why not just treat phones as data devices and voice is just another IP protocol that runs on them? I imagine the "shoddy workmanship" has a lot to do with carrier specific settings and vendor specific stuff that is difficult to code for properly. A modern telecom is a shitshow of legacy junk that should be thrown out.

Granted, this is kinda happening with some carriers and google voice, but the move to LTE-only with VOIP needs to happen sooner than later. The carriers seem to be fighting this because they, then, can't sell "minutes" and other old fashioned concepts. They would become a dumb internet pipe and they fear what that means from a profit perspective.

I'm not sure why you assume a switch to VoIP will help in that regard - in practical terms that means the SIP protocol for the voice (and SMS) signalling. While SIP is going to be the way forward, the specs are really full of ambiguity and not that well though out features. Most commercial vendors of SIP servers include full scripting languages in order for the admins to "normalize" the protocol in order to try to smooth over incompabilities - I can guarantee you that when phones are released in the wild with a SIP stack the telco hasn't tested, things will stop working until they've implemented workarounds for the SIP flavor of the day.

SIP is software. What runs on my cellular modem and its stack is firmware written by yet another party.

If SIP app #474 is screwing up, I switch to SIP app #475. This all can be done in userland as an application. Changing out my GSM stack? Yeah, I can't do that.

Well, with VoLTE currently being implemented in networks and phones, that SIP stack used to realize VoLTE on the phone can pretty much be considered firmware too.

However, you might fiddle around with various other SIP clients and find one that works for you. The other hundreds of millions of mobile phone users are not going to do that - so the problem will persist.

Instead of all this complexity, why not just treat phones as data devices and voice is just another IP protocol that runs on them?

They had a chance to do that with LTE. But the entrenched players decided instead to greatly increase the size of the specification (and the over-the-air protocol) to specially support voice (VoLTE).

Granted, handoffs from 2G to 3G to 4G are quite complicated, and then throw in roaming on top of that and it gets really complicated.

However, if I had a magic wand or time travel device, I'd rather have seen LTE be data only (with good quality of service support) and IPv6 only. That would have made the protocol vastly simpler, and then it would be possible to un-bundle voice altogether... eventually.

I don't know that such a design would be in the carrier's best interests though, so who's to say it could ever have happened, even with magic.

> compared to millions of customers storming your shops and demanding refunds

And yet somehow Apple is able to push updates without carrier approval and this doesn't happen.

Except for 8.0.1 which hosed cell service for everyone.

However, Apple was able to pull the update within hours and release a fix the next day.

Or 8.1.x that hosed most user's battery life (as in < 6 hours off a full charge) among a slurry of other issues.[1]

[1] http://www.forbes.com/sites/gordonkelly/2014/12/09/apple-rel...

And millions of customers stormed shops? Which is the rationale that is provided for carriers taking months to approve updates for everyone but Apple.

I am not defending carriers here. I just wanted to point out that Apple's approach isn't perfect either.

>Except for 8.0.1 which hosed cell service for everyone.

Only iPhone6 only if you updated a point release in the first hour, not that it is good.

What are you basing your statement on? Verizon and AT&T phones have been pegged at different minor versions of iOS many times.

People want to believe Apple is some magic company that can do no wrong, but that ignores the extensive amount of QA (both in house and at the telcos) that happens before any release like this (as a longish time OS X user, I should know.)

When an iPhone update is released, the update becomes available worldwide for everyone. There is no delay waiting for individual carriers to approve the update, like there is with Android and Windows Phone.

Not one single thing in your comment is factually correct. In fact, both points are demonstrably the opposite of the truth. Apple is not perfect. It is very good at marketing, very good at aesthetic design, decent to good at acquiring marginally above average hardware, and ranges poor to decent at software. The most important thing Apple is good at is marketing, and your comment demonstrates why.

Can you please provide the demonstrable truth that Apple gets carrier approval for their updates (which are released simultaneously worldwide, sometimes new updates go out mere days apart, which is definitely not enough time for every carrier in the world to have tested the update to approve it.

And also links the news articles where millions of Apple customers stormed shops after updates broke their phones.

> The most important thing Apple is good at is marketing, and your comment demonstrates why.

Apple has been able to become powerful in phone market. They enough power to push the carriers around, rather than the other way round. Therefore everyone but Apple has to have their updates approved by the carrier before they get pushed out. My comment was not pro-Apple. Merely pointing out the hypocrisy in how carriers claim they have to spend months testing updates for everyone except Apple.

That's because Apple owns the hardware. It's not really fair to compare the two shops in this way, considering the differences between an OS, and an OS + platform.

Nokia also owned the hardware but Symbian updates were carrier approved. Microsoft owns the hardware but Windows Phone updates are carrier approved. BlackBerry owns the hardware but Blackberry OS updates are carrier approved.

a bit more prudence from both apple and the carriers might have been smart to avoid antennagate, that being said, not the same as software updates.

I had accidentally moved $1,000 from the wrong bank account into Google Wallet, and called customer service to try and figure out what I should do, because there wasn't going to be $1,000 in the account I had selected, but Google Wallet was giving me access to the funds already.

The CS rep that responded was a nice guy, but had no clue what to do. We started spitballing together some possible ideas, and I suggested that we just let it go and Wallet will eventually figure out that the transfer didn't go through, and deduct the money back from my Wallet account. He thought that might be a good idea, but had no further ideas beyond that.

That was Google Wallet, and that was $1,000 floating around. I can't imagine the level of customer service available to anyone who hasn't actually put money directly into the equation.

Do they actually do that thing we all saw in The Internship, where they have random engineers field support calls? Because I felt like I was talking to a random engineer.

I used to work at Google in support. You were not talking to a random engineer - the support people are all dedicated support contractors. Most of the teams are all non-technical and the only thing they know about how the product works is from a two week training where most of the time is spent on learning how to avoid lawsuits.

support people at Google is there only to fill regulation minimums.

The teams I was a part of most definitely were not just for minimums (Chromebook/Glass). Google has incredible support for their hardware products.

that's honestly awesome to hear.

my experience in the past made me avoid newer Google products because support in the past was non existent. last thing i bought was a nexus one. i for the bad touch screen (registers ghost touches after screen is on for a while) and i could either complain on forums and be ignored or start a return with htc that from the forums histories ended with a device on worse condition and save touch screen bug sent back...

How do people think google puts software engineers on the phone randomly? That's just an insane idea. I mean, these guys all make what 150k starting with 250k+ salaries not unheard of and you really expect their managers to go, "Oh btw, you're on helpdesk this week. Put on your headset and talk to clueless people for 8 hours today."

It was in The Internship, that's what put the idea in my head, and as far as I know it's in there because they do/did do that, with the idea being you get a better understanding of who you're writing features for if you get your questions from the horse's mouth occasionally.

I once worked at a startup-bought-by-bigco that did almost exactly this. Granted it was level 4, not level 1 support; and it was on a quarterly, not weekly, basis.

I can promise that this is the one thing they don't do, because then sometimes you'd get a useful answer. Yes, talking to the Image Search guy about Google Wallet would be useless, but he might have some clever guess. At the very least he would display an understanding of why computers do the things they do.

As is, it has to be essentially non-technical people taking the calls. I can't think of why else they would respond to a complaint like "The age on my account won't update as I get older" with "Are you sure you're right about your age?"

Wow, thanks for that interesting insight into the telco side of firmware upgrade delays.

I humbly submit that the carriers could avoid a lot of ill-will if there was more transparency in that process. I hope there aren't contractual obligations in place preventing that because it really changes the way I personally see the problem.

They have always been fairly transparent in how this happens. The problem is they say they have to put these things through an extensive testing process before release...and then never release it. But hey, here's a new phone you can buy with the latest release!

Go to any carrier support forum / Twitter and they're usually pretty open about testing. But, yes, there are always contractual obligations. Not least, Manufacturer X doesn't want Carrier Y saying "Delays are due to..."

Motorola and Google goes (dumbly, imho) completely opposite on this.

they sell devices that are unlocked/developer edition, that are full of disclaimers that if you unlock boot loader some guaranties do no apply etc. and instead of providing h the new images first to this users to help then test, they usually roll out to the general population and weeks (Motorola, months) later make the images officially available.

i guess there's two reasons, one is that they can get some money from operators to release first on one network or another, the second is that they have no consumer support so they couldn't care less. alternatively it's just honest incompetence.

On the other hand, by delaying an update containing important security fixes, you leave your customers sitting ducks for the next drive by download attack.

We haven't seen any major botnets infecting mobile phones yet, but I think it is just a matter of time. Desktop browsers gets updated almost instantly world wide and they still fall prey for attacks, when mobile software lags for weeks or months at a time. The more people do their online banking on their phones instead of their desktops, it gets more valuable as a target.

This problem needs a solution, and sooner rather than later.

thank you for this post. i always felt that the other side of the fence was greatly underrepresented here.

I second that. We always get the same stream of negative comments towards telcos (not saying they're not deserved), but we don't get to see very often how this looks on the other side of the fence. It'll be wonderful to have more comments like GP's one.

Surely these are issues to do with phone manufacturers and not with the carriers?

Reply on news.ycombinator.com