Hacker News Reader

Comment by rkagerer | Hacker News Reader

Jeez - can't dial 911, randomly dials 911.

I don't understand why these devices can't even do the most core feature of a phone properly.

This will sound crass but the development teams (right up to CEO's) should be dragged out to the gallows and flogged.

If it was my product I would have made damn sure the 911 experience was perfect before shipping, and not rested for a minute until any bugs were solidly quashed - up to and including recalling all sold units and overhauling the flakey architecture if needed.

This is life safety we're talking about, not only for their users but also everyone else impacted by their blatent abuse of the emergency services system. Would we tolerate bridges that collapsed with equally ambivalent consequences for those who engineered them?

View on HN

In any other field, engineers would be held responsible and after so many "mistakes" they would lose their license.

Software engineers will fight tooth and nails to keep their privilege is being called engineers whilst having none of the responsibility when it comes to the harm they're causing.

Really? Who lost their engineering license from Firestone tires? Toyota gas pedals? Hasbro easy-bake oven? Graco high chairs?

Stop trotting out the same baseless comment over and over.

In my country, engineers are legally responsible for their work to the point where insurance is required to do engineering. And yes, engineers that are negligent are held responsible. Just because you're ignorant doesn't mean it doesn't happen.

Certainly liability insurance exists.

Let me know if find answers to my questions.

You know any software engineers with liability insurance? Also, you would have learned of many examples had you gone to engineering school, we have whole courses dedicated to laws and ethics.

Yes many software engineering firms hold liability insurance.

And many software engineering contracts contain liability clauses.

In most fields with “real engineers” do they also get told by management to ship things broken and/or with arbitrary deadlines, or do they have the ability to push back on things, with some legal recourse or means to avoid the threat of losing their jobs if they say no to something, that software engineers lack?

I don’t think things will change until corporate management changes (via being forced to, or otherwise).

If it was my product I would have made damn sure the 911 experience was perfect before shipping

For what it's worth, every telco switch upgrade I performed ages ago, early days of GSM the first number I tested was 911. I made sure the dispatcher could hear me. I don't know whats going on with the phone development side of things. That seems like a QA and customer feedback review problem. It probably also does not help that wireless vendors are slow/hesitant to update phones. There is a fear of bricking phones and customer support nightmares their words, not mine. I could flash update a phone over the air but this was in the 90's. No idea what that process looks like now. I assume they stage an update on a CDN after hopefully testing it extensively. Do all cell phones have two boot partitions in the event the upgrade process is sub-optimal(c)?

I can hallucinate couple different explanations for that:

  - Modern software is way too overcomplicated to take seriously.  

  - 911 is handled too specially, leading to oversights by implementers.   

  - Feature importance to you has nothing to do with implementation difficulties.

etc.

One more I'd throw on the list is a growing trend in our field of shipping buggy products with the expectation we'll fix them later via updates. It's a terrible drug the internet enabled.

The ironic thing is personally I only upgrade my phone every 5+ years and would be totally happy with longer development cycles.

Yes, I think the real problem is #2.

The only reliable way to test 911 features is a test lab, to which the average engineer doesn't have access to. On top of that, calling 911 isn't exactly as placing a normal call - so the only way to test is... to call 911.

Again, a test lab should help towards these things, but I doubt Google has one accessible to the average engineer working on the dialer. Plus, they most likely don't have a way to automatically test these changes - or they might happen as part of other "features" (remember the Microsoft Teams bug that caused similar issues?).

In the end, the smarter our smartphones become - the dumber they are at doing the one single thing they were initially meant to do - get help in case of an emergency.

> Again, a test lab should help towards these things, but I doubt Google has one accessible to the average engineer working on the dialer.

And why not? It seems like the real real problem is #4: Management doesn't take seriously people's need to reach emergency services because it's not a profit center.

A test lab is a room that is completely isolated from the outside (and in a way that the RF doesn't leak outside of the premises) where you can do these kind of experiments.

Considering the amount of teams working on the Google Dialer, and the fact that they might be distributed across multiple cities / countries - this sounds very expensive.

Building a Faraday cage is an afternoon woodworking project. You can build them in your garage or bedroom with hand tools if you wanted. You don’t even need a huge room, a portable phone booth sized space could be mass produced and delivered on site. A Faraday Booth might take a few dozen square meters of copper mesh and some basic lumber, drywall and finishing efforts. If it took more than $10k per office I’d be shocked. This is something simple and cheap enough it could be something a local director or manager could charge to their company card and assembled themselves if they really cared.

It shouldn't be too expensive, but before you put in equipment and pretend to be the phone network, you need to make sure that you're hitting regulatory limits on the amount of RF leaking out. Doing a good enough job to hit that above a gigahertz--- including things like conducted RF, and validating with measurements-- is gonna cost a bit more than you describe.

[1]. $1k, -95dB to 3GHz, in stock, manufacturer standard I/O plates sold separately. I think there's zero room for hypotheticals and challenges to talk about setting these up if you ask not webdevs but actual phone people.

1: https://jretest.com/product/jre-0912/

Who's the person who crawls into the tiny box and makes sure the dialer UI is doing the right thing, given that is what our primary concern is?

We've been talking about the cost of making a screened room. "Turnkey" ones cost about $20-50k, but you're going to pay contractors a fair bit beyond that. Not to mention whatever equipment you're putting inside.

https://www.ramayes.com/Refurbished_Radio_Frequency_Shielded...

You can get things like [1] - an RF blocking box with a window and attached RF-blocking gloves. This equipment is not at all exotic.

And if you don't like that option - you can also place test calls to 911 [2] by calling their non-emergency number and arranging a time. It would be easy to perform a once-a-month test call.

[1] https://jretest.com/product/jre-1812f-forensics-analysis-enc... [2] https://www.911.gov/calling-911/frequently-asked-questions/

What it seems like they really need is some general black-box testing time to try and better identify what is going on, along with looking at logs from devices of users who had the problem.

A screened room with a few devices and a simulator seems like a good resource to have, and it's a reasonable thing to procure at a center or two if they don't have it. It just costs several tens of thousands, not $10k.

It seems like at this point 911 usually works on Pixel, and there's probably even good unit test and automated integration test of a lot of the components-- prearranged test calls with a given carrier's 911 impl isn't likely to fix it. But "usually" isn't good enough: Google needs to actually figure out what's going on, no matter what.

$50k is chump change for Google. Regardless, there are costs and regulations associated with making a phone, and if you aren’t willing to adhere to that, you’d better keep making search engines and other websites.

> $50k is chump change for Google.

c.f. my comment:

> > > It shouldn't be too expensive, but

People just don't read and understand that you can agree with the broader scope of comment but not the details.

Our primary concern is dialer crashing! I don't understand why you're trying to shift the blame away from Google. That's just stupid.

> Our primary concern is dialer crashing!

Yes, so being able to use the phone UI is important for qualitative testing.

> I don't understand why you're trying to shift the blame away from Google. That's just stupid.

I feel like you didn't read my comment. I stated that a screened room isn't prohibitively expensive, but can't be delivered for the suggested $10k. In another comment, I said "Society spends a few billion dollars a year on E911 and related infrastructure. To have faith in it eroded because of a little bit of slipshod validation by phone vendors is a false economy."

Why isn't it possible to have nuanced discussions-- to condemn Google's failure here, but also to suggest that the capital expense to give people realistic test environments is probably higher than others are suggesting?

Frankly, you're being abrasive.

You must have the UI? Call your factory, ask them make an extra ICE and hook it up to the simulator through an attenuator and shielded cables. Easy. Cost couple cars worth, so what. Add noises, simulate phasing, handover, do anything you want to. premises is, you're a top mass market smart phone manufacturer, and Google, both of that at the same time.

No, you're trying to find a way to make it sound impossible. But what you're saying is more along, probably, "finding a parking lot to test brakes is impossible even for a car company". Something real stupid as that.

No. I'm just saying that it costs like $40-100k in practice to get a screened room into operation, not $10k, but that this is still a reasonable expense.

But you're too boiling over in vitriol to understand nuance. Somehow, in your mind, saying that you can't do it for $10k appears to be me defending Google.

Even if it costs $1M, Google either has to do it or stop selling phones and recall sold units.

I think maybe GP is getting you confused with the person who brought up the suggestion it's expensive in the first place, before we started looking into what the numbers really would be. While the initial statement of that sounds like defending Google, that wasn't you, and unpacking the costs is intellectually interesting on its own merits, say, for anyone who might want to do it themselves.

Fine, for permitting and regulations and inspections, let’s 100x the cost and say it’s an annual expense too. This is still well within Google’s budgets to provide to their Android teams.

This discussion is moot btw. Google already has faraday cage and bts equipment, at least in the London office (6ps) where a large part of the Android team is located. And obviously so - the costs involved are a rounding error for Google. The problem must be somewhere else.

If Google, of all companies, can't afford the necessary testing equipment for a critical function, who can? If they thought that emergency services would make them money, they could have a lab in every office. But it won't, and so they don't care.

Linus Tech Tips has one complete with an isolated private 5G network inside. I’m sure Google could manage.

Intrigued, looked up... hm. and am still wondering what they plan to do with that.

1: https://firecell.io/product/labkit/

They are building their "Labs" which aims to offer high quality testing. They also have purchased a professional power supply tester as well.

They noticed cell phone reviews over time have dropped signal or quality testing. Which makes sense since each reviewer is just testing in their local environment with no standards or baseline so they were pretty meaningless anyways.

Are they going to be able to test enough phones and wifi/radio equipment to make it worth while? Not too sure.

Arguing an expensive price tag for Google just doesn't resonate. They are building a phone - it should be able to do the one thing that it should, as others have said. Anything outside of contacting someone in an emergency should be secondary.

Society spends a few billion dollars a year on E911 and related infrastructure.

To have faith in it eroded because of a little bit of slipshod validation by phone vendors is a false economy.

Even the dimmest view of management should expect them to care about extremely bad PR.

So how do you explain Unity and Reddit and Twitter and Wizards of the Coast all belly flopping their PR this year? I'm sure they care, in some abstract sense, about PR (well, not Musk). I just think they're out of touch and incompetent at dealing with it. Some of them are even on the record saying it will blow over and the benefits of not caring are worth the cost, like the Reddit CEO. Clearly the costs of bad PR aren't high enough.

They care if bad PR results in less money. Wizards +Unity realized they pushed too hard and will ultimately cost them. Reddit thinks their decision still makes financial sense.

And Google is making exactly the same calculation. Maybe they'll do something to fix it after a hundred people die because they can't call 911 and it blows up nationally.

> the only way to test is... to call 911

You should be able to use 933 to test emergency services. https://support.bandwidth.com/hc/en-us/articles/210291778-Th...

Completely tangential to testing that the service connects: I have a young child and we've done practice "what do you do in an emergency" things, but are there any "fake 911" type services where a young child can call to practice a 911 call, so they can experience how the operator will talk to them, so in a real emergency it would be a bit less jarring?

Is that only a service that Bandwidth.com offers or would it work on a regular cell network?

I think it depends on the carrier you use, but a lot of other carriers support it.

I'm guessing pretty much all are carriers that are using Bandwidth.com for 911 service.

I've been wondering how to test this, thank you

Okay, but that's not going through the same paths of a 911 call, is it?

And by that I mean the emergency mode of the modems

You don't need a full concert hall to test 911, the simplest LTE test equipment can be just couple sandwiches big. They're also not expensive at Google scale. It does concern me that there were anecdotal posts that read to me like that, developing firmwares and not blatantly violating basic assumptions and principles and core premises of Google MDM cannot occur simultaneously. If I somehow had to, I would bet that that to be on the path to the root cause. I mean, a lot of software jockeys have to be explained that IP address isn't assigned to a CPU socket.

You need a special room as well - you can't accept that someone near your building might end up having his 911 call routed through your test infrastructure.

Alternatively, I think you can somehow connect the device antenna to your equipment, so that the signal doesn't even have to be transmitted over the air (or at the scale of a femtocell). Still, rather "expensive" to setup in multiple locations for just a team that develops a dialer.

Additionally, you'd have to test this across multiple OS versions, and devices. Still doable, but most likely not incentivized by the managers at Google

I don't see why it would be expensive, for no other than Google, to set up a microwave oven with a factory rooted phone and a femtocell inside on ~dozen locations worldwide. There's no special legal or financial complexities in doing that.

And selling phones that can't pass certification is just irresponsible. If you can't make a product work, you're free to be responsible and cancel a product. It's on Google to do businesses legally.

The faraday cage in a microwave is good for its purpose, but not good enough at blocking cell phone bands. They would need something slightly more expensive, anywhere from $1000 to $50k depending on how fancy they wanted to be. Of course Google should have no problem affording any of that, but these details do matter.

I suspect that the problem must be more complex than a bug in just the phone or the OS; for one thing, the problem is intermittent and doesn’t affect every phone. Still, the lack of action from the FCC is disturbing.

Can't you stand up your own test network for that? Other phones won't connect to that, so you don't need to fully rf isolate, as long as you have some trust in the test network not doing crazy things. It's well over a decade ago, in Europe, and I was only very peripherally involved, but it didn't seem that hard to get permits for such a test network indoors.

I think the problem is, 911/112 calls are routed through the best cell network available. This means that if somebody is close enough to your lab, their phone would still try to use your femtocell.

From very dim memory: I think there's some flag you can set that marks your test network as not suitable for that kind of thing. But even if not, IIRC you can set routing information for stuff like 911 on a subscriber basis in your test network - which you'd just do for the handset you're testing.

In that case, you'd just be changing yet another variable and not test the emergency mode of the modem - which might actually be broken due to a software update, right?

I believe some places you provide emergency services the number you will be calling from to test, so they can handle it appropriately.

Ultimately with a system like 911, you will always have to do final testing on the real deal. Because this is just too serious to get wrong.

How does every other phone manufacturer do it then?

> 911 is handled too specially AFAIK there is some special functionality to report location directly to emergency services when calling to speed up emergency handling

but if that would be my implementation I would track call in some very simple way and if that's second or third try to call 911 within hour then handle it as ordinary call without that extra functionality as a safeguard

Also possible that bug is due to poor handling of mobile network error so no functionality is lost, just a UI issue. Or there is a bug, but the frequency is much lower than mobile network failure rate so low priority.

Yeah, 911 calls should be implemented in an entirely different hardware subsystem, just like the flight control system of an airplane doesn't run on the same hardware as the entertainment systems.

My bet is on Google doesn't give a shit. The pixel 4a is a second tier device with constant ui crashes, glitches and design obviously not made for it.

> If it was my product I would have made damn sure the 911 experience was perfect before shipping

not absolving Google, but this is easier said than done

> Jeez - can't dial 911, randomly dials 911.

There's enough 911 calls for everyone, they're just not distributed equally.

Reply on news.ycombinator.com