Hacker News Reader

Comment by jerlam | Hacker News Reader

There is currently some kind of geomagnetic storm which is causing a lot of GPS tracks to be offset over the past few days. It's all over the running/cycling forums. Could be related, here's an example:

https://old.reddit.com/r/Garmin/comments/vo17e2/the_magnetic...

View on HN

It couldn't possibly be GPS, right? Networks are unreliable, the cars must be able to reach a safe state if the network drops out. They appear to have stopped in the road. If these cars can't safely navigate their way to the side of the road using local sensors, they are wildly, hilariously unsuited for autonomous operation.

> at least a dozen autonomous Chevrolet Bolts from GM Cruise Automation were spotted blocking the intersection at Gough Street and Fulton Street for a couple of hours, according to reddit user seansinha.

There must be something weird about that intersection that is causing a bug.

> If these cars can't safely navigate their way to the side of the road using local sensors, they are wildly, hilariously unsuited for autonomous operation.

You are a lot more confident than me that this isn't the case.

Or I could be pessimistic about GPS -- these things are, albeit in limited numbers, on the road and actually being used by customers, right? I guess I'm surprised GPS is so reliable that this isn't a widespread, constant issue, if this is the normal "I lost GPS" behavior.

It could be a middle ground kind of thing. The cars have to trust something. For position you have GPS + some interpolation based on your movement sensors + some buildings/streets and maps (also unreliable since construction etc changes a lot, a lot of buildings and intersections look the same, etc).

Computers are very far away from understanding where they are from context alone. Hell most humans don't know where they are. So if you have a short GPS outage they will be able to continue navigating due to other sensors, but the longer the GPS outage/issues persist the more unlikely it is that they still know where they are.

There should definitely be a safety to drive to the side of the road and wait though.

Could it be GPS spoofing that is causing this then?

Malice, incompetence, all that jazz...

Still. I would expect some kind of sanity check so that they aren't stopping in the middle of the road.

If local sensors tell you the road is blocked, fine. But if remote data tells you one thing, and local sensors tell you something else, the fail safe should be pulling over to the side of the road.

Obviously there is no failsafe that pulls the car over, otherwise you would see that the cars had pulled over. This might be annoying on city streets but imagine if it were on the highway. An almost guaranteed 50 car pileup. If history is any guide, this will actually happen in the next few years and be followed by a lawsuit and regulation.

That's assuming that there's a space to pull over, which is often not the case on city streets. Especially, when every AV wants to pull over.

Stopping as close to a curb as possible is still better than stopping in the middle of the line. And much better than stopping on an intersection.

Also the assumption that the fail-safe _works_. Any of these fail-safes are much more complex than a classical fail-safe.

I mean, if we were driving together -- you have the steering wheel, I have the map -- and I say "woah I have no idea where we are," what would you do? I guess you would just drive around until I figured it out, or found a safe spot to pull over...

Dead reckoning works surprisingly well at least in Teslas, of which the precision can be seen when they move around in underground facilities and finally resurface: the car pops up pretty close to the entry point.

In any case, aren't all self-driving vehicles equipped with lidars and work in pre-mapped regions? I would 100% expect they are able to just use those maps with lidar to precisely geolocate themselves, probably more precisely than with GPS. And, indeed, GPS itself can at times be inaccurate or simply not available due to obstructions.

So what I'm saying, maybe it's wasn't the GPS.

> aren't all self-driving vehicles equipped with lidars

I believe Tesla has made a lot of noise about not using lidar.[0]

[0]: https://venturebeat.com/2021/07/03/tesla-ai-chief-explains-w...

In this context Tesla is not an example of a self-driving car I think, what was discussed was robo-taxis and I think all of them have lidar

Maybe at some point in the future, but Teslas aren't self-driving vehicles: you always need a driver in them, even (or maybe even: more so) in the case of FSD Beta.

I imagine these robotaxi vehicles would not have stayed on the road had they have had a safety driver in them :).

Yeah, I sort of expected a lizard brain/mammal brain split: features like object avoidance and the ability to pull over don't require the car to know where on Earth it is, and if the car can't identify to side of the road and drive to it safely with local sensors, then it just isn't fit to be on the road. That sort of stuff needs to be operate with an antilock-breaking/"fire the airbags?" level of reliability.

"Where on Earth am I" on the other hand is basically irrelevant to safety. Run it on the entertainment console or whatever. Interpolate position, come up with heuristics, whatever. I mean, it should be like two reasonably competent people on a road-trip -- if you are the driver and I'm the map guy, you aren't going to just stomp on the brakes when I inevitable lose track of where we are (I'm bad at navigating). You'd probably just go along the road while I worked it out, unless I was really confused, in which case you'd presumably pull over somewhere.

Yeah they have Lidars for 10s of 1000s of $ mounted and should be able to go for certain distances without GPS. If a GPS outage was designed as failure mode. But it would perhaps be one of the later items on any engineer‘s priority list since it is so reliable.

A network failure is, like, the world's most predictable problem though.

You've nailed the dirty little secret of the AV industry. Everyone's safety model is underpinned by the ridiculous assumption that a human will be (a) reachable with very high reliability over shitty cellular networks; (b) able to correctly interpret complex situations with extremely limited information; (c) able to resolve safety-critical situations by making planner adjustments, without direct control over steering and acceleration.

Wow. I had no idea that these things had such fundamental limitations. I'd feel a lot less safer using one now, to be honest. If you had to estimate, how long would it be before AVs would be able to do all their calculations locally? Is that something that's even under consideration at the time?

It is probably a chicken-and-egg problem. They need the data from these scenarios, so they can train the local model for later iterations accordingly.

It's a well-known thing in AV circles that some companies were actually faking their route by hardcoding them for demos etc. Also one shortcut is to use HD map data, and relying on 100k+ LIDAR sensor data that will never will be cost-effective. I know of only a handful of companies that rely on 'cheap' sensor data (cameras/radar) and use real-time calculations.

If a full time driver earns 25k-50k that's sort of equivalent to every regular taxi and uber having a $500k / $1M driving package onboard already (at 4%), right?

I don't know the industry but very naively it doesn't seem to me like $100k really breaks the unit economics? Particularly if volume / learning curve can reduce the price?

You are right in the way that the industry seems to be shifting from general-purpose self-driving to transportation/hauling.

> relying on 100k+ LIDAR sensor data that will never will be cost-effective

Eh, the $100k prices are mostly for high-end, low-production-volume research LIDARs.

There are LIDAR manufacturers claiming they can achieve prices of $500 [1] or even $200 [2] if they were fitted to every car made.

[1] https://guidehouseinsights.com/news-and-views/lower-cost-lid... [2] https://www.opsys-tech.com/post/are-we-nearing-a-shakeout-in...

Well, THAT has a Dropout-like HULU miniseries written all over it.

Tesla’s AV design philosophy differs from everyone else’s for just this reason. The idea of using centimeter-accurate models of the environment and having an always-on network connection is fundamentally at odds with handling surprises.

I don’t think a current Tesla would have a good time with no GPS either, but philosophically they stand to better handle intermittent unreliability and adverse conditions once they figure things out.

For some reason I thought these were supposed to be real products, that sounds more like they are still in the prototype/proof-of-concept phase.

This is the difference between Tesla and competitors.

Tesla cars „can” drive you „safely” without connection to the network.

Some say that discard of lidar was a huge mistake by Elon, but he knows very well that this is not the way to go forward.

What does lidar have to do with connection to the network? If anything, once GPS goes down, more sensing modalities can only be helpful.

Car should be able to drive without Lidar and connection to the network.

There is currently only a single car company that accomplished that.

Why without Lidar ? Why not without camera then ? Lidar has nothing to do with Internet connection.

NOAA doesn't list any meaningful geomagnetic storms right now:

https://www.swpc.noaa.gov/communities/space-weather-enthusia...

But that aside, I'd really hope that these cars are using professional WAAS-capable GPS receivers. Short of a major X-class flare which knocks out the WAAS geostationary satellites, the entire point of the system is to correct for these kinds of disturbances to the GPS signals.

Why does this make me wonder if something is going on with GPS due to the war in Ukraine?

EDIT: the following is incorrect per a reply below. Will leave the original comment below.

---

That was going to be my hypothesis. GPS resolution can be reduced in a conflict area.

I'll defer to someone with actual knowledge to add more info. :)

I believe the government promised not to ever reduce GPS resolution some 15 years ago, and new satellites supposedly do not even have the ability to do so.

TIL, apologies for the misinformation! Will correct my comment if it's still in the edit window

Oh my sweet summer child.

I am old enough to recall when they did so post-9/11.

The US government would never give up such a capability, that would be out of character for them. But they would sure as hell pretend like they did, which is very much in character.

I think they would. There's at least 3 fully operational GPS alternatives, there's no reason to have this capability anymore. Some receivers will combine all of these systems.

All these systems are rigged to jam and spoof at the push of a button I expect.

They are owned by: China, Russia, India, Europe and Japan.

There was a specific technical capability that they used to reserve the right to use, and I believe they no longer need to use that capability. Obviously this would only be the case if they have some alternative capability. But I forget the name of the feature so I am having trouble looking up sources. I obviously do not take the government at its word, but I can believe that a technical capability developed in the 1980's might no longer be the most useful counter measure. Especially since there are now 5 GNSS constellation systems, with GPS only being one of them, and low cost off the shelf receivers can read all of them.

Years ago a government promise and .25 cents could get you a phone call. Today you won't even get the phone call.

Reduction of GPS Resolution is limited to specific areas, it can be targeted. If they are doing it (probably are in Ukraine) then it only affects some areas. You can also bypass that restriction if you have the right military keys (read: Ukraine).

Yeah, that is what my (perhaps outdated?) understanding was.

What is said is the current capabilities. All other GNS systems have the same capabilities (though European GNS Galileo has some more modern features and the Soviet one is a bit of a shitshow)

This submission is worrying me a bit, my company's product depends on GPS+GLONASS with RTK assistance at high latitudes for high accuracy high frequency positioning.

The NTRIP RTK services we use just provides assistance for GPS+GLONASS, no Galileo or BeiDou. GLONASS is crucial because the satellite orbits are more visible at high latitudes providing better reception. Still, RTK should help cope with this problem but I might have to check last week's collections.

And your systems actually support decryption and entering the said keys.

You might be surprised what proportion of the hardware that operates in Ukraine (from both sides) is using civillian grade GPS receivers (i.e. almost anything with a price tag of less than 1M USD).

From the linked reddit thread:

>Does this happen when you're not using Glonass? I wonder if Russia has been 'tweeking' it.

>I've switched to Galileo and the problem is FIXED.

WAAS has been available on consumer GPS devices since the mid 2000's and is not "professional" nor does it address reception/interference issus; WAAS provides semi-real-time position refinement data based off drift observed by ground stations within a certain range to compensate for things like atmospheric disruption of the signal. WAAS is built-in to the vast majority of receivers these days.

Multi-band GNSS is what helps address issues around reception like multipath and interference. Receivers used to be pretty expensive, and the number of L5 satellites wasn't that high until the last few years, but now can be found in even a number of consumer electronic devices, and the number of L5 satellites keeps going up (and as more satellites have L5 signals, things will get better.) Several Garmin sport devices have it now, for example. A quick search shows Maxim sells a multi-band GNSS chipset for $10.

There is really no excuse for any autonomous vehicle company to not be at least evaluating multi-band GPS receivers, if only to give more reliable position data in urban environs.

I'd be surprised if at least some aren't using RTK receivers, but this requires a base station within a certain range of the cars, or commercial service (such as that offered to farmers by John Deere.) RTK just isn't that expensive compared to even a single LIDAR unit.

Edit: they can tolerate at least some degree of GPS degradation and use dead reckoning / lidar data in some cases, according to this person: https://news.ycombinator.com/item?id=31940167

I just read about a sunspot that recently doubled in size [0] and maybe launched a CME in our direction -- we don't really know due to a power outage at Stanford University that took the SDO offline [1], anyway food for thought, maybe it's coincidence.

[0] https://scitechdaily.com/auroras-incoming-giant-sunspot-has-...

[1] https://interestingengineering.com/a-huge-solar-eruption

I'm not really seeing anything weird with GNSS. I have a continuously-operating receiver that writes its location into a database every second; I did a query over the last 30 days and there is no abnormal deviation in position. (My receiver is in a terrible location that can only see a tiny sliver of the sky, and is single-band, so I expect about a +/- 50cm deviation without RTK correction and that's what I'm seeing.)

I actually haven't looked at this data for quite a while. I didn't consider the start of the Ukraine war a good reason to start looking at my data, and I delete it after 30 days to save $$$ on block storage. Dumb. (I added some proper downsampling today so I can keep it forever at reasonable cost.)

Trusting my own data is always suspicious, so I clicked around randomly on the NYS Spatial Reference Network website (https://cors.dot.ny.gov/; the title of the page is "Shop", but accounts and RTK access are free). They operate a bunch of Continuously Operating Reference Stations and publish statistics about the accuracy. I do see a lot of reported GLONASS cycle slips today. I have no idea if that is normal or not; this is the first time I've ever dug that deeply into station stats. But overall, the position accuracy at the stations I spot-checked are within .5cm, which is normal. (Most use 4 constellations; GPS, GLONASS, Beidou, Galileo; on 2 bands. So even if GLONASS is broken today, one constellation is enough for good accuracy.)

If the government has degraded GPS accuracy today as many commenters think have happened, many people would have noticed. The reference stations have known positions; if GPS starts saying they're somewhere else, that's impossible and would be flagged. GPS being significantly degraded would probably be front page news; not just noticed by a few exercise tracking enthusiasts on Reddit. Or maybe it's all a big conspiracy and the US and Russia banded together to hack my database and the NYS spatial reference network. It could happen, I guess, but my non expert opinion is that GPS is fine right now.

> my non expert opinion

Yes, clearly given the detail in your post, you are "not an expert". Thanks for being on HN and making this site great.

There is a lot of depth to GNSS! I could do what jrockway did on the grandparent comment and I would not define myself either as anything near an expert on the field.

I think an actual expert would know if those "warnings" are interesting or not. I can find the raw data, but I don't really know what it means ;)

I've been looking at GNSS (mostly GPS, really) interference around the world for a few months. I just generated some maps of the U.S. for the past few days: https://twitter.com/lemonodor/status/1542941996746039297

You can see that there isn't any significant interference visible in the Bay area, or the rest of the United states except for the typical activity in and around military ranges—That huge region in New Mexico experiencing moderate interference is not typical, but seems to correspond with planned testing around White Sands Missile Range, according to an FAA NOTAM[1].

It’s possible that my methodology wouldn’t pick up interference if it’s caused by a low-power jammer or some source of EFI that only has an effect within a few hundred feet.

More details on the methodology behind those maps:

The ADS-B signals that many aircraft broadcast include not just the aircraft's GPS positions, but also a measure of GPS accuracy (strictly speaking, ADS-B doesn't talk about GPS specifically and can handle any sort of navigation technology; I'm sure there are some planes out there reporting positions based on inertial navigation systems, with correspondingly low accuracy, or GLONASS-derived positions, or whatever, but my understanding is that right now something on the order of 99% of aircraft with ADS-B are using good old GPS so I'll just keep using the term GPS in this description). If you go to https://globe.adsbexchange.com and click on just about any aircraft, you'll see an info sidebar on the left of the screen. Scroll down until you see the ACCURACY section, and you'll see values labeled NACp, SIL, NACv, NICbaro, and Rc. Those are all self-reported measures of the accuracy of the data being sent by the aircraft[2]. NACp is "Navigation Accuracy Category for position", and is a good measure of whether the aircraft's GPS is working well. (A somewhat obscure feature of ADS-B Exchange lets you see a map of all aircraft that are currently reporting poor navigation accuracy for their GPS: https://globe.adsbexchange.com/?badgps)

To make the maps, I process a day's worth of data from ADS-B Exchange to find all the aircraft reporting poor navigation accuracy and then I color map hexes (using H3 hexes) according to the proportion of aircraft passing through that hex that reported bad GPS accuracy. Specifically, I'm counting an aircraft as experiencing "interference" if it at some point reported good navigation accuracy and then reported low accuracy. Doing this helps filter out aircraft that just have an ongoing issue with their GPS equipment, or don't even have GPS.

When I do that, areas where there is systematic interference—almost always jamming by military systems—become obvious. There are a few conflict zones (Syria, Cyprus, Israel) that have been experiencing jamming for years, and the U.S. often has smaller scale military testing, especially in the West and Southwest. You can also see the jammers that are apparently setup around Moscow to prevent drones from flying near Oligarch dachas[3].

I started making these maps in February before Russia invaded Ukraine because I thought it might provide an early warning of an invasion. I didn't see that, and in fact this technique doesn't do a very good job of mapping GPS jamming around the actual war zones because civil aviation stopped over Ukraine, so there are zero or few aircraft with ADS-B reporting their GPS accuracy[4]. Without that data, I can't make a map.

Sometimes I do see changes, like when Russia suddenly started jamming around Kaliningrad in March 2022, causing interference in many Baltic states and leading to Finland to cancel some flights[5]. Then a few days later, they just stopped.

[Edit:] I don't think too many people have realized yet what an amazing source of GPS interference data is available using ADS-B! It's like having thousands of sensors roaming the planet, broadcasting GPS accuracy data every few seconds. I sometimes wonder if I would disrupt someone's nascent business model if I started publishing my maps regularly.

(I have a Synology setup with 12 TB that I use to store various sensor data—I highly recommend it.)

1. https://www.faasafety.gov/files/notices/2022/Jun/WSMRNM_22-0... 2. https://twitter.com/lemonodor/status/1493789598077440000 3. https://www.gpsworld.com/jammers-at-dachas-add-to-russias-ab... 4. https://twitter.com/lemonodor/status/1497295859196649475 5. https://twitter.com/lemonodor/status/1500719113185816577

One thing I've found that I didn't expect: I can see hundreds, and even thousands of aircraft suddenly start experiencing severe GPS interference in a specific region, and sometimes there will be coverage in the media or bulletins by government organizations (though usually I see it a day or two before that happens), but sometimes there's just... nothing. E.g. the sudden, intensive jamming in Romania that started at the beginning of June https://twitter.com/lemonodor/status/1535321975639289857

The ADS-B data collection approach is brilliant. This thread is pretty old; you should consider writing a little summary and posting it as an article. Your visualizations on Twitter are also excellent.

It is worth noting however that the size, power consumption and antenna precision of a watch-sized or bicycle computer size GPS(+glonass, Galileo, beidou, qzss) receiver is considerably smaller than what should be in these vehicles.

I have not seen any mention of GPS service degradation in hobbyist drone enthusiast groups over the last week, some of whom use quite small GPS modules with serial connection to the flight controller for automated signal lost return to home features and such.

I would find it odd if they were not using precise positioning using local reference stations broadcasting corrections over IP or RF.

EDIT: Appears GPS/IMU/LIDAR is sufficient, my mistake. TIL.

They're almost certainly using dead reckoning with wheel rotation sensors to get the precision they need in an urban environment where radios are kinda trash.

I would think zero wheel rotation sensors and just LIDAR, which cruise uses extensively.

Tesla’s also use dead reckoning when GPS is lost. It’s way more accurate than I would have guessed. My GPS antenna was out for over a week and the map was only about 100ft off.

Highly doubt they'd be using GPS for anything more than putting them in the right neighborhood.

Edit: Looks like someone ex-Cruise confirmed this ITT

That's not what the comment said. They said the system can tolerate degradation / loss of GPS data for periods of time, in certain circumstances:

https://news.ycombinator.com/item?id=31940167

> Fine localization should be able to resolve without GNSS input (with caveats) and there are specific tested requirements to be able to tolerate a certain number of minutes of outage even in the absence of LIDAR before position accuracy degrades beyond acceptable limits.

I doubt it has anything to do with any geomagnetic storm, the issue seems to only affect Garmin devices.

That is most definitely not it, these vehicles have localization modules that rely on lidar point cloud matching, gps is generally only used for bootstrapping that process but is technically not even needed.

Are you telling me all these fleet of vehicles can be defeated with a GPS Jammer?

https://www.jammer-store.com/gj6-all-civil-gps-signal-jammer...

GPSs can and will be taken offline sometimes (in a region) due to military exercises, tests etc

This happens with a certain frequency in aviation

Relying purely on GPS (for self driving) won't work.

People in that thread are reporting a problem with gps + glonass while galileo on the same frequency as gps is fine.

Could it be using a different protocol or CRC system that's more resistant to that type of issue perhaps? There must be some difference, otherwise it would just be more GPS, right?

You're very optimistic

People were responding that switching from Glonass to Galileo fixed it. Could it be sabotage?

> Could it be sabotage?

Given that even Russian pilots in Ukraine have been found to be using GPS, it looks like that would not even be necessary.

https://www.globaldefensecorp.com/2022/06/21/exposed-capture...

Reply on news.ycombinator.com