Hacker News Reader

Comment by denysvitali | Hacker News Reader

Yes, I think the real problem is #2.

The only reliable way to test 911 features is a test lab, to which the average engineer doesn't have access to. On top of that, calling 911 isn't exactly as placing a normal call - so the only way to test is... to call 911.

Again, a test lab should help towards these things, but I doubt Google has one accessible to the average engineer working on the dialer. Plus, they most likely don't have a way to automatically test these changes - or they might happen as part of other "features" (remember the Microsoft Teams bug that caused similar issues?).

In the end, the smarter our smartphones become - the dumber they are at doing the one single thing they were initially meant to do - get help in case of an emergency.

View on HN

> Again, a test lab should help towards these things, but I doubt Google has one accessible to the average engineer working on the dialer.

And why not? It seems like the real real problem is #4: Management doesn't take seriously people's need to reach emergency services because it's not a profit center.

A test lab is a room that is completely isolated from the outside (and in a way that the RF doesn't leak outside of the premises) where you can do these kind of experiments.

Considering the amount of teams working on the Google Dialer, and the fact that they might be distributed across multiple cities / countries - this sounds very expensive.

Linus Tech Tips has one complete with an isolated private 5G network inside. I’m sure Google could manage.

Intrigued, looked up... hm. and am still wondering what they plan to do with that.

1: https://firecell.io/product/labkit/

They are building their "Labs" which aims to offer high quality testing. They also have purchased a professional power supply tester as well.

They noticed cell phone reviews over time have dropped signal or quality testing. Which makes sense since each reviewer is just testing in their local environment with no standards or baseline so they were pretty meaningless anyways.

Are they going to be able to test enough phones and wifi/radio equipment to make it worth while? Not too sure.

If Google, of all companies, can't afford the necessary testing equipment for a critical function, who can? If they thought that emergency services would make them money, they could have a lab in every office. But it won't, and so they don't care.

Building a Faraday cage is an afternoon woodworking project. You can build them in your garage or bedroom with hand tools if you wanted. You don’t even need a huge room, a portable phone booth sized space could be mass produced and delivered on site. A Faraday Booth might take a few dozen square meters of copper mesh and some basic lumber, drywall and finishing efforts. If it took more than $10k per office I’d be shocked. This is something simple and cheap enough it could be something a local director or manager could charge to their company card and assembled themselves if they really cared.

It shouldn't be too expensive, but before you put in equipment and pretend to be the phone network, you need to make sure that you're hitting regulatory limits on the amount of RF leaking out. Doing a good enough job to hit that above a gigahertz--- including things like conducted RF, and validating with measurements-- is gonna cost a bit more than you describe.

Fine, for permitting and regulations and inspections, let’s 100x the cost and say it’s an annual expense too. This is still well within Google’s budgets to provide to their Android teams.

This discussion is moot btw. Google already has faraday cage and bts equipment, at least in the London office (6ps) where a large part of the Android team is located. And obviously so - the costs involved are a rounding error for Google. The problem must be somewhere else.

[1]. $1k, -95dB to 3GHz, in stock, manufacturer standard I/O plates sold separately. I think there's zero room for hypotheticals and challenges to talk about setting these up if you ask not webdevs but actual phone people.

1: https://jretest.com/product/jre-0912/

Who's the person who crawls into the tiny box and makes sure the dialer UI is doing the right thing, given that is what our primary concern is?

We've been talking about the cost of making a screened room. "Turnkey" ones cost about $20-50k, but you're going to pay contractors a fair bit beyond that. Not to mention whatever equipment you're putting inside.

https://www.ramayes.com/Refurbished_Radio_Frequency_Shielded...

$50k is chump change for Google. Regardless, there are costs and regulations associated with making a phone, and if you aren’t willing to adhere to that, you’d better keep making search engines and other websites.

> $50k is chump change for Google.

c.f. my comment:

> > > It shouldn't be too expensive, but

People just don't read and understand that you can agree with the broader scope of comment but not the details.

Our primary concern is dialer crashing! I don't understand why you're trying to shift the blame away from Google. That's just stupid.

> Our primary concern is dialer crashing!

Yes, so being able to use the phone UI is important for qualitative testing.

> I don't understand why you're trying to shift the blame away from Google. That's just stupid.

I feel like you didn't read my comment. I stated that a screened room isn't prohibitively expensive, but can't be delivered for the suggested $10k. In another comment, I said "Society spends a few billion dollars a year on E911 and related infrastructure. To have faith in it eroded because of a little bit of slipshod validation by phone vendors is a false economy."

Why isn't it possible to have nuanced discussions-- to condemn Google's failure here, but also to suggest that the capital expense to give people realistic test environments is probably higher than others are suggesting?

Frankly, you're being abrasive.

You must have the UI? Call your factory, ask them make an extra ICE and hook it up to the simulator through an attenuator and shielded cables. Easy. Cost couple cars worth, so what. Add noises, simulate phasing, handover, do anything you want to. premises is, you're a top mass market smart phone manufacturer, and Google, both of that at the same time.

No, you're trying to find a way to make it sound impossible. But what you're saying is more along, probably, "finding a parking lot to test brakes is impossible even for a car company". Something real stupid as that.

No. I'm just saying that it costs like $40-100k in practice to get a screened room into operation, not $10k, but that this is still a reasonable expense.

But you're too boiling over in vitriol to understand nuance. Somehow, in your mind, saying that you can't do it for $10k appears to be me defending Google.

Even if it costs $1M, Google either has to do it or stop selling phones and recall sold units.

I think maybe GP is getting you confused with the person who brought up the suggestion it's expensive in the first place, before we started looking into what the numbers really would be. While the initial statement of that sounds like defending Google, that wasn't you, and unpacking the costs is intellectually interesting on its own merits, say, for anyone who might want to do it themselves.

You can get things like [1] - an RF blocking box with a window and attached RF-blocking gloves. This equipment is not at all exotic.

And if you don't like that option - you can also place test calls to 911 [2] by calling their non-emergency number and arranging a time. It would be easy to perform a once-a-month test call.

[1] https://jretest.com/product/jre-1812f-forensics-analysis-enc... [2] https://www.911.gov/calling-911/frequently-asked-questions/

What it seems like they really need is some general black-box testing time to try and better identify what is going on, along with looking at logs from devices of users who had the problem.

A screened room with a few devices and a simulator seems like a good resource to have, and it's a reasonable thing to procure at a center or two if they don't have it. It just costs several tens of thousands, not $10k.

It seems like at this point 911 usually works on Pixel, and there's probably even good unit test and automated integration test of a lot of the components-- prearranged test calls with a given carrier's 911 impl isn't likely to fix it. But "usually" isn't good enough: Google needs to actually figure out what's going on, no matter what.

Arguing an expensive price tag for Google just doesn't resonate. They are building a phone - it should be able to do the one thing that it should, as others have said. Anything outside of contacting someone in an emergency should be secondary.

Society spends a few billion dollars a year on E911 and related infrastructure.

To have faith in it eroded because of a little bit of slipshod validation by phone vendors is a false economy.

Even the dimmest view of management should expect them to care about extremely bad PR.

So how do you explain Unity and Reddit and Twitter and Wizards of the Coast all belly flopping their PR this year? I'm sure they care, in some abstract sense, about PR (well, not Musk). I just think they're out of touch and incompetent at dealing with it. Some of them are even on the record saying it will blow over and the benefits of not caring are worth the cost, like the Reddit CEO. Clearly the costs of bad PR aren't high enough.

They care if bad PR results in less money. Wizards +Unity realized they pushed too hard and will ultimately cost them. Reddit thinks their decision still makes financial sense.

And Google is making exactly the same calculation. Maybe they'll do something to fix it after a hundred people die because they can't call 911 and it blows up nationally.

> the only way to test is... to call 911

You should be able to use 933 to test emergency services. https://support.bandwidth.com/hc/en-us/articles/210291778-Th...

Is that only a service that Bandwidth.com offers or would it work on a regular cell network?

I think it depends on the carrier you use, but a lot of other carriers support it.

I'm guessing pretty much all are carriers that are using Bandwidth.com for 911 service.

I've been wondering how to test this, thank you

Completely tangential to testing that the service connects: I have a young child and we've done practice "what do you do in an emergency" things, but are there any "fake 911" type services where a young child can call to practice a 911 call, so they can experience how the operator will talk to them, so in a real emergency it would be a bit less jarring?

Okay, but that's not going through the same paths of a 911 call, is it?

And by that I mean the emergency mode of the modems

You don't need a full concert hall to test 911, the simplest LTE test equipment can be just couple sandwiches big. They're also not expensive at Google scale. It does concern me that there were anecdotal posts that read to me like that, developing firmwares and not blatantly violating basic assumptions and principles and core premises of Google MDM cannot occur simultaneously. If I somehow had to, I would bet that that to be on the path to the root cause. I mean, a lot of software jockeys have to be explained that IP address isn't assigned to a CPU socket.

You need a special room as well - you can't accept that someone near your building might end up having his 911 call routed through your test infrastructure.

Alternatively, I think you can somehow connect the device antenna to your equipment, so that the signal doesn't even have to be transmitted over the air (or at the scale of a femtocell). Still, rather "expensive" to setup in multiple locations for just a team that develops a dialer.

Additionally, you'd have to test this across multiple OS versions, and devices. Still doable, but most likely not incentivized by the managers at Google

I don't see why it would be expensive, for no other than Google, to set up a microwave oven with a factory rooted phone and a femtocell inside on ~dozen locations worldwide. There's no special legal or financial complexities in doing that.

And selling phones that can't pass certification is just irresponsible. If you can't make a product work, you're free to be responsible and cancel a product. It's on Google to do businesses legally.

The faraday cage in a microwave is good for its purpose, but not good enough at blocking cell phone bands. They would need something slightly more expensive, anywhere from $1000 to $50k depending on how fancy they wanted to be. Of course Google should have no problem affording any of that, but these details do matter.

I suspect that the problem must be more complex than a bug in just the phone or the OS; for one thing, the problem is intermittent and doesn’t affect every phone. Still, the lack of action from the FCC is disturbing.

Can't you stand up your own test network for that? Other phones won't connect to that, so you don't need to fully rf isolate, as long as you have some trust in the test network not doing crazy things. It's well over a decade ago, in Europe, and I was only very peripherally involved, but it didn't seem that hard to get permits for such a test network indoors.

I think the problem is, 911/112 calls are routed through the best cell network available. This means that if somebody is close enough to your lab, their phone would still try to use your femtocell.

From very dim memory: I think there's some flag you can set that marks your test network as not suitable for that kind of thing. But even if not, IIRC you can set routing information for stuff like 911 on a subscriber basis in your test network - which you'd just do for the handset you're testing.

In that case, you'd just be changing yet another variable and not test the emergency mode of the modem - which might actually be broken due to a software update, right?

I believe some places you provide emergency services the number you will be calling from to test, so they can handle it appropriately.

Ultimately with a system like 911, you will always have to do final testing on the real deal. Because this is just too serious to get wrong.

How does every other phone manufacturer do it then?

Reply on news.ycombinator.com