Meltdown Update Kernel doesnt boot
bugs.launchpad.net
bugs.launchpad.net
It must be up there amongst the greats, probably with the "halt and catch fire" op code. Normally they just patch this stuff with microcode and never really tell anybody, this time that won't work.
I'm not entirely convinced it was a mistake at all (dons tin foil hat), Intel have been making suspicious design decisions to their chips for a well now (think this, Intel Me, hidden instructions, etc). It seems clear to me that this security by obscurity approach is quite frankly crap.
>And now I can't update my OS before making an installation USB because Canonical can't just follow Linus' releases. Thanks.
Linus' releases need some sort of buffering before they can be considered stable, often distributions will apply their own patches on top. Also consider the scenario where Linus releases a bad kernel and no testing has been performed before rolling out to all Linux users.
Keep in mind that whatever "evil agencies" would have asked for this would most likely find themselves vulnerable, and nobody would sign off on.
I do agree, however, the "security by obscurity approach is quite frankly crap". The fact that even large corporations (not the big 5) can't even get away from ME speaks volumes about why this is a bad idea. Facebook isn't the only company with important data.
Amen. It blows my mind that some people think clever techniques like speculative or out-of-order execution must've somehow how nefarious intentions behind them. Come on HN...
And in the case of SMTP, it's basically a pinata of bugs for the last 30 years regardless of platform
Alone the risk would not be worth to intel. Do you really think, nsa has enough money to compensate for this backslash and newscoverage?
Any nonuniform distribution in any quantity that is not part of the spec is exploitable!
Oh, you don't want to do that?
It just keeps popping up, someone finally thought to weaponize it.
Someone published its weaponization, you mean :)
But yet and still we found out. So yes, this security through obscurity approach is terrible (with a code embargo being the obvious exception).
They only update microcode when they have to. When doing otherwise risks... Well, this kind of mess.
You dont wanna know how many times I've rebuilt my gentoo system chasing after retpoline kernel & gcc builds that just... Break everything.
It should be interesting to see how it all develops
See https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SpectreAn...
Tbh it’s not the most meaningful of statements, but it’s food for thought.
Or, to put it another way, I have no clue to what you're referring--what do references have to do with "The Billion Dollar Mistake"[0]?
[0]: https://en.wikipedia.org/wiki/Tony_Hoare#Apologies_and_retra...
EDIT: my apologies, that joke was actually pretty good.
However, I completely missed the pun. Cheers :)
> On system reset, the vector table is fixed at address 0x00000000.
Also, I'm not an expert on the C standard, but in my understanding, it doesn't "break" it. That is:
* Address 0 and the null pointer are distinct
* A 0 literal treated as a pointer is guaranteed to be the null pointer
* The null pointer is not guaranteed to be represented by all zero bits
* If you get a pointer to address zero via pointer math or by other means than a 0 literal, you can still access address zero.
So the question of what happens when you actually do that is purely up to your compiler and architecture. In most cases, if you manage to get the NULL pointer value through pointer arithmetic, it will still compare equal to the 'actual' NULL pointer and treated as if it was a literal 0, so that doesn't allow you to get around NULL pointer checks. The only situation where it really matters if the NULL is only known at runtime, since that may have implications on optimizations. Since dereferencing the NULL pointer being undefined behavior, the compiler can remove such dereferences, but it can't remove the dereference completely if it can't prove the pointer is always NULL. There is nothing preventing the compiler from adding extra NULL checks in that aren't in your code however, which would foil the plan of generating a NULL pointer at runtime to dereference it. So unless your compiler explicitly allows otherwise, you cannot reliably access the memory located at the value of the NULL pointer - as far as the standard is concerned, there is no such thing.
Talking specifically about the ARM vector table, that largely works ok because only the CPU ever has to actually read that structure, normally you C code won't have to touch it (If you even define it in your C code. The example ARM programs define the vector table in assembly instead). If you did ever have a reason to read the first entry of that table from C though, you could potentially run into issues (Though I would consider it unlikely, since the location of the vector table isn't decided until link-time, at which point your code is already compiled).
On that note, it's worth adding that POSIX requires NULL to be represented by all zero bits, which is useful. Lots/most programs actually rely on this behavior, since it is pretty ubiquitous to use `memset` to clear structures, and that only writes zero bits.
(Sorry for the long comment, I've just always found this particular part of the standard to be very interesting)
I am not sure your point. The reason modern systems don't map memory to 0x0 is because NULL pointers exist. It is a reflection of a leaky abstraction equating pointers to references. That leaky abstraction has (or so the argument goes) caused >$1B in software bugs.
The other mindset would be "malloc always has to allocate memory or otherwise indicate failure; you cannot cast from a integer to a pointer; you cannot perform arithmetic on a pointer to get a new one; you must demonstrate there are no hanging references when freeing". This is essentially what rust did for safe code.
The reason why I indicate so much skepticism is that rust is the first time I've seen the problem solved well in the same problem space as C. Ada has problems of its own. It's more about how small assumptions can have massive economic (and health, and safety, and ethical) consequences. Certainly comparable to a speculative execution bug leaking memory in an unprotected fashion--in both cases the bugs find their way through human error in evaluating enormously complex systems for incorrect assumptions :)
Llvm won't use it for anything. (I think it starts putting things at 8). Trying to access it explicitly in C will generate `unreachable` instructions.
https://www.lucidchart.com/techblog/2015/08/31/the-worst-mis...
"biggest" by net financial loss to a single entity? I dunno. How much did that failed NSA launch cost the state again?
I didn't hear about that one
I suppose if there was an exploit targeted at a specific program, it would be possible to work out what location the secrets are stored in?
Additionally some hardening methods like stack protector make stack allocated objects stand out a lot from register values.
Given that whoever writes is would also have access to the other program, they would have a lot more information on where to look in memory.
(Obviously if you have nation states or serious criminal organizations trying to breach you regularly, this is more serious)
Heartbleed was touted as being bad by those that didn't read too far into it. You could scrape memory, sure. But it was always random fragments. This lets you make targeted address attacks. Force a process to use that memory space through a NOOP and now you can start scraping at will. Or you can just do an entire memory dump and pull things out in plaintext (like scraping Firefox passwords, which we've seen done already).
The only reason this isn't worse is it requires the ability to execute code on the machine. It has high (near absolute) impact, but low-to-moderate on the ease of execution.
This title is held by autorun.inf which has caused over 20 years of broken, vulnerable behavior and, AFAIK, is still going strong.
It wasn't that bad, because people took it seriously. But there were still tons of practical systems affected and billions of corporate dollars associated with fixing it.
So when you say "biggest mess up" you gotta define specific qualifiers. Because Meltdown/Specter is going to be solved by simply... buying a new CPU. (And retrofitting the old ones). So it consist of mostly a patch.
A BIG important patch, granted. But it's still just a patch. But some ATM's aren't going to start spewing money like they did on Y2K.
If kernel 4.4 doesn't work, I recommend compiling the 4.15 kernel. (Note, however, that you may need to apply a patch to NVIDIA drivers).
I've done this on Ubuntu 16.04 LTS, 17.10, and Debian 8 so far this week. To compile, set CONFIG_PAGE_TABLE_ISOLATION=y. That is:
sudo apt-get build-dep linux
sudo apt-get install gcc-6-plugin-dev libelf-dev libncurses5-dev
cd /usr/src
wget https://git.kernel.org/torvalds/t/linux-4.15-rc7.tar.gz
tar -xvf linux-4.15-rc7.tar.gz
cd linux-4.15-rc7
cp /boot/config-`uname -r` .config
make CONFIG_PAGE_TABLE_ISOLATION=y deb-pkgBut for those who would be willing to risk breaking a few things to try something out, building a kernel is a worthwhile effort, and the meltdown / spectre bugs provide a perfect excuse to do it.
Of course, I also go into any updates with a rollback plan. ITIL sucks, but one thing it taught me was the value of well documented plans any time you make changes to production systems.
According to the top comment in one of the posts in HN even 4.9 and 4.4 use a less ideal patch: https://news.ycombinator.com/item?id=16085672
I can’t really judge how much RH engineers are capable of fixing that kind of stuff in a kernel that’s officially out of support upstream.
Based on the general quality of RHEL/RHV I trust them to do the right thing, but I have no insight whatsoever in how kernel development actually works.
I'm kind of shocked Amazon doesn't have something like Linode's Fennix, but you can always do an EBS snapshot of your /boot volume and revert it if a kernel upgrade breaks stuff.
Only don't patch if your server is isolated and not connected to the internet.
Always have a backout plan when doing upgrades, I'm just glad EL and derived distributions have an easy way to do it with yum's transaction history.
All the notes say is that 109 fixes it.
linux (4.4.0-109.132) xenial; urgency=low
* linux: 4.4.0-109.132 -proposed tracker (LP: #1742252)
* Kernel trace with xenial 4.4 (4.4.0-108.131, Candidate kernels for PTI fix)
(LP: #1741934)
- SAUCE: kaiser: fix perf crashes - fix to original commit
> diff -u linux-4.4.0/arch/x86/events/intel/ds.c linux-4.4.0/arch/x86/events/intel/ds.c
> --- linux-4.4.0/arch/x86/events/intel/ds.c
> +++ linux-4.4.0/arch/x86/events/intel/ds.c
> @@ -415,7 +415,6 @@
> return;
>
> per_cpu(cpu_hw_events, cpu).ds = NULL;
> - kfree(ds);
> }
>
> void release_ds_buffers(void)
Though it seems more likely to me that this bug was filed as a placeholder for the already-written patch and verification thereof, and the person filing it simply didn't bother with the urgency field since it wasn't really a bug-report-as-such.
So it sounds like it was probably specific chipsets and not CPUs, but who knows.
[1] - https://www.engadget.com/2018/01/09/microsoft-halts-meltdown...
[2] - https://support.microsoft.com/en-us/help/4056892/windows-10-...
The Microsoft link you provide says "Microsoft is working with AMD to resolve this issue", so they're not just brushing it off and telling customers to contact AMD.
One has to learn that lesson at some point.
It's not like Windows that bricks your laptop. It's a handful of hardware config, and you can easily boot with an older kernel.
This is certainly a pain, but it's hardly the first time a broken kernel has shipped. Reasonable recovery mechanisms are in place.
If you write anything criticizing editing or proofreading, there will be a fault of some kind in what you have written.
Stop staring at my finger. Please ;)
Or did you mean something else?
I wouldn't expect them to replace any CPU, unless it was manufactured recently and still being manufactured.
But a refund in some capacity? That's reasonable, I think. In the meantime, we would have to settle for the software fixes.
Is there a market of 99%+ seemingly authentic fake Intel chips out there?
In the EU virtually every product comes with a 2 year warranty. So every CPU sold in the EU in the last two years should be replaced for free by Intel, even through OEMs.
I wonder what potential class action lawsuits Intel might be facing.
https://www.intel.com/content/dam/www/public/us/en/documents...
Note that the cost of overly onerous regulation (e.g. requiring that every computer manufacturer replace these chips even though the problems can largely be worked around in software) is of course passed onto consumers.
The warranty and any other legalese from intel is irrelevant here, this is about consumer protection laws of various countries that supersede an intel warranty. A serious post sale drop in performance would be enough for a refund on any computer purchased in many countries. In Australia if I bought a computer 6 months ago I'd be entitled to take it back to the store for a refund, then it's up to them to argue with dell and dell to argue with intel.
> Note that the cost of overly onerous regulation (e.g. requiring that every computer manufacturer replace these chips even though the problems can largely be worked around in software) is of course passed onto consumers.
Demanding that a product works and in lieu of that offering a replacement or refund is not overly onerous regulation, it's a very basic standard protection.
For something like Meltdown/Spectre, the patches/workarounds reportedly barely affect some workloads, but cause drastic slowdowns for others. So already not everyone's affected to the same extent. Then you have computers with easily replaceable CPUs vs. stuff like phones and laptops which probably were only designed to work with a single CPU, and the manufacturer's already working on their next model and doesn't want to waste money building replacement parts for the previous one. At that point, maybe you have a complaint with e.g. Apple for selling you an iPhone that doesn't work as performed because they had to work around a security problem, and Apple might themselves go after Intel. The whole situation is a lot more complicated than "it should totally be covered under the warranty."
(because it's a design flaw)
> Intel and other technology companies have been made aware of new security research describing software analysis methods that, when used for malicious purposes, have the potential to improperly gather sensitive data from computing devices that are operating as designed.
> […]
> Recent reports that these exploits are caused by a “bug” or a “flaw” and are unique to Intel products are incorrect.
(https://newsroom.intel.com/news/intel-responds-to-security-r... ; emphasis mine.)
A recall would be great, but there's no way they'd be able to do it. Vehicle recalls are a bit different because they impact physical safety. Digital safety doesn't get the same priority.
Then imagine all chips from 1995 to 2015, having to make them again, they don't have the machines anymore.
(besides you should not be taking your car to the dealer if you value your wallet)
Spectre/Meltdown is a wakeup call for many things, one of them probably being for computer manufacturers to not solder the CPU to the Motherboard and for the x86 world to stick with a standard socket, to facilitate replacing parts.
Good luck with that. A large portion of affected CPUs/SoCs are in mobile devices and ultrabooks. Socketed chips simply won't fly in those kinds of devices.
Or just the money actually. If you can’t replace my broken item, a refund is always appreciated.
The later is pretty analogous to issuing firmware and OS patches to mitigate the flaw.
Ford Pinto, anyone...
Another one that many people don't know about is a problem with old 2-door Tahoes; a bracket on the driver's side seat likes to fail upon quick acceleration - such as when getting on the freeway, for instance.
One minute you're upright, pushing the pedal to get up to speed, the next - whoops! - flat on your back! If you're lucky, you live to tell the tale...
AFAIK, GM never issued a recall about that one (it caused me to pass on a really nice lifted 4wd Tahoe a few years back)...
An AMD CPU?
I didn't realize that when I made the comment, and I agree my suggestion falls flat now that I know.
> You have to replace at least the whole motherboard on a desktop or probably all of your laptop except the discs and maybe the RAM.
I'm ok with putting that responsibility on Intel to remedy the situation, even if it deeply hurt them financially or put them out of business. If you sell a faulty product that doesn't live up to its description, yes you risk actually going out of business. But with the fact that AMD has Spectre this idea of replacement no longer makes much sense and your original idea of a partial refund makes the most sense.
The problem is this nearly every single processor Intel shipped for a decade so Intel doesn't have the cash flow to RMA that many replacements. They're going to fight tooth and nail to avoid this.
In reality they could probably argue standard
depreciation on a product and offer the remaining
value as a discount towards a working product...
So since ‘95 or ‘96.
This whole thing is the equivalent of discovering that if someone throws enough nails on the road your car will blow a tire, spin out of control and kill you. With the kneejerk reaction of trying to fill everyone's tires with concrete to avoid the tires blowing out rather than trying to figure out how to keep people from throwing nails on the road (with the idea that spiteful users are more likely on toll roads) in the first place.
Spectre/Meltdown ONLY is an issue if you run some untrusted code on your system. If you avoid this, there is not problem. Yes, we like to be able to run untrusted code (such as in a web browser / javascript), but that is not the fault of a car manufacturer that you like to pick up hitchhikers.
Yup, this will hit the EC2/etc users hard, but those people have already IMHO given up on absolute performance by putting themselves in shared environments where bad neighbor syndrome can already hit their perf pretty badly.
But for whatever reason (probably because its easier) the current plan just seems to be to use the big hammer.
Your idea of black/white listing processes might bubble up as a solution in some scenarios. Perhaps it could be pledge-like; if you're savvy enough, try implementing it, or fleshing out the details.
Realistically though, they might be able to do it for cpu's that aren't soldered on (think just about every laptop) made in the last year or two, but would they really fab new versions of 10 year old cores? Its not like many of those lines are even running anymore, so they would basically have to redesign/layout and reverify everything.
Probably easier/cheaper just to send everyone a new machine.
This incident demonstrates why you really don't want catastrophic bugs in the CPU. The fact that the hardware vendors missed this one makes you wonder what else is out there.
feels like this would happen:
>intel agrees to give consumers a $30 price break in response to meltdown/spectre
>in other news, intel raises the prices of next generation CPUs by $30
I haven't even seen a proof of concept exploit that has the same conditions as in the wild. All the POC exploits seems to have been given some assistance in various ways (such as being given root perms or a preknown memory address).
Does anybody have an example of this JavaScript exploit or any exploit that would work in the wild?