Hard drive hack provides root access, even after reinstall | Hacker News Reader

Hard drive hack provides root access, even after reinstall

spritesmods.com

472 pointspd0wm90 comments

This was a great read. One of the things we've done in the past is to modify the firmware of the drive to be able to give errors on command. The purpose was for testing RAID systems in real life scenarios. One can include a 'unit test' drive in a RAID array which will run through a series of known bad disk behaviours. From the simple like returning read failure, to the more complex like returning the wrong block or returning a block that has been silently corrupted (both things NetApp observed in the wild on 'real' drives), and my personal favourite acknowledging a write but not actually writing the data (nearly killed the Cisco relationship they had at the time)

I especially like the idea of cannibalizing old HDDs (with bad spindles but good controllers) to become microcontrollers in new projects.

Not a bad idea, but also nearly everything we interact with, technology-wise, has microcontrollers of some form or another. The AVRs so adored by the arduino community actually exist in large volumes in automobiles, and even crappy USB keyboards and mice which we might throw out have microcontrollers in them.

So I'm all for scavenging compute bits for future projects, but it is by no means unique to HDDs.

The key is tools, and OpenOCD is one of them, which let you "talk" to these systems. I picked up a Black Magic probe [1] and am building cables for it to talk to one of my ARM boards. That kind of stuff makes the spelunking possible.

[1] http://www.blacksphere.co.nz/main/blackmagic

also the things cost pennies! For example the ATMEGA328 will run you a grand total of $2.88 in a quantity of 1.

The difference is that the enterprise storage space gets such firmware from the vendor rather than hacking it on its own. At least where I've been so far (NetApp is not included so far :-)

Before I got my hand on vendor provided error injection I would have thought this to be of great use but hacking in ARM assembly to get this would be quite a task.

If you liked this, then you might like Travis Goodspeed's really cool talk about "Writing a Thumbdrive from Scratch" (for antiforensics) [1] at the 29th Chaos Communication Congress [29c3].

[1] http://www.youtube.com/watch?v=D8Im0_KUEf8

At some point, computer systems will be more like biologic systems - they will just carry a more or less permanent "flora" of parasites, that will have to be tolerated unless they become explicitly hostile.

Any sufficiently complex system will exhibit this trait sooner or later.

That's an interesting way of looking at "parasitic" programs. Maybe we could even get those parasites to work for us, like trapping them as the house pen-tester. You could employ the constant barrage of attempted ssh logins as remote connectivity check or something.

Travis Goodspeed is one of my favorite bloggers. Definitely worth checking out if you like this kind of stuff: http://travisgoodspeed.blogspot.com/

The thing that interests me, though, is the idea of modifying your hard drive firmware for better performance.

My understanding is that the effective width of the write head is 10x the width of the read head... E.g. with the right firmware, it should be possible, if you are okay with a write-once medium, to write the outermost track, move the write head in 1/10th what you'd normally move it, then write the next track, etc... and get 10x the space out of the drive you normally would. In theory, the read head wouldn't have trouble. (of course, this would be write once storage, as the effective width of your write head is still pretty huge; but for a bunch of things? I can totally work with that... if more than X% of a drive was garbage data, I copy the good data to a new drive and reformat the old one. Done.)

I hear rumors that both the major drive manufacturers are actually shipping drives with this technology, but are only selling those drives to really big players, for some reason.

Here's a reasonable reference to the 'shingle' technology, and he roadmap for the rest of us:

http://www.theregister.co.uk/2013/06/25/wd_shingles_hamr_roa...

but that's the thing, with the datasheets (and, well, a lot more skill than I personally have) we should be able to setup something like shingling on the cheap disks we have today.

Of course, from reading the article, I'm not sure I'm any closer to that particular dream.

Shingled writes require a special asymmetrical write head, you can't do it with current drives. Actual shingled write drives are not yet shipping AFAIK.

I'm just using shingled writes as one example. Your kernel could, for example, more efficiently reorder reads and writes with more information about the physical drive layout. Hell, just removing the bad-sector remapping (and moving it up to the kernel or the like) would help solve the performance degradation that remapped sectors cause during apparently sequential reads/writes.

Ignoring the doubters, there are plenty of cases where custom drive firmware might be useful.

I read online (probably on HN or similar) that Amazon Glacier is using drives with custom firmware that keeps them spun down to 300rpm so that more drives can fit in a rack without power and cooling concerns.

That's certainly an interesting case that certainly wasn't possible without drive manufacturers stepping up to Amazon's wishes. Being able to do custom mods like this to your own disks is pretty excellent as well.

I'm sure the people who make the drives are trying to get as much performance as possible from the firmware. They're also working with information you won't have.

While this is true they are also optimizing for a wide range of operating environments. If you could reduce the size of the optimization space you could get a better result. There are many places where I could think of better behavior out of the disks, their current interface is somewhat limited and very much a black box. If you could open it up a little and maybe provide for some extra operations you could gain quite a bit.

For example, most RAID systems don't really care so much about the first error on the disk, if the disk fails to read we can save a lot of time by not retrying too much and just go to build from the RAID. If by any chance this is the second (RAID5) or third (RAID6) error than you want a much stronger retry logic. Current disk firmwares do not allow for such logic.

Everything worth doing isn't already done yet. I'd rather encourage others to try novel ideas and approaches. Most won't work, but a working model in this space could become a great company. (Because the potential market for smarter disk interaction is huge.)

>I'm sure the people who make the drives are trying to get as much performance as possible from the firmware.

Huh. I think it's fairly common that companies engage in price discrimination by producing a lot of the same hardware, then crippling the hardware sold to the lower-end. Note, my example of hard drive manufactures doing this has to do with the next bit of your quote:

>They're also working with information you won't have.

So the 'crippling' I whine the most about is the difference between 'consumer' and 'enterprise' hard drives.

If you aren't running a hard drive in a raid, if it's just one drive in a desktop, generally speaking, if there's a problem? you want the thing to keep retrying, if there is any chance at all that it might be able to resolve the problem.

If it's just one drive in a desktop, it's almost always best to do something that will make the drive go slower than to cause the drive to fail.

My situation? where drives are sitting in a RAID? almost the exact opposite.

So yeah; me? I spend twice as much money to get "enterprise" drives that are almost identical, mechanically, but come with slightly better firmware. Firmware that just fails, rather than waking me up in the middle of the night.

(A friend of mine has been telling me: "Luke, a hung drive is just a special case of a slow drive; You need to monitor read/write latency and proactively fail slowish drives. check out blocktrace" - and he's probably right.)

Note, WD has TLER, which they say you can change with WDTLER.exe. In my experience? works on about half the drives you try, and even then those drives are far more likely to get slow (but not completely hang) than an 'enterprise' drive.

Now... let's talk about bad sectors. Filesystems have been handling bad sectors, well, for most of my life now. they can do it fairly well.

The problem with letting the firmware handle bad sectors is that the OS doing read/write reordering assumes that if you write sector 559 560 561, those are physically sequential. Once the hardware firmware remaps sector 560 off into the fucking boonies, my nice sequential read is now completely fucking random... and way slower. My point is that something like ZFS can handle bad sectors way better than the drive firmware, because it's got a lot more information. A lot more information in the case of read errors... all the firmware can do is hang you up retrying; the RAID layer could actively grab that block from another drive.

So yeah, they have information I don't have... and my computer would go dramatically faster if I could have that information. My pager

From what I know when a drive "reallocates a sector" it actually reallocates a track or something very close to that. So that at least for the rotational sequencing the performance will not change that much. Ofcourse, the track that used to be just one track seek away now became further off.

There are also several places along the way where reallocations go to and the drive tries to find the closest one to the reallocated tracks to avoid too large seeks.

My knee-jerk reaction was, why didn't WD sign the code and use on-chip fuses and a secure boot path to verify the code before transferring control to anything outside their boot ROM? (Many ARM-based systems-on-a-chip are capable of doing this).

Adds cost, for one thing. But you can arrange for the unit to never run a byte of code (even one loaded from the platter) that didn't come from WD.

Your typical motherboard's BIOS code is not signed. Your video card's BIOS is not signed. Your network's card firmware is not signed. Your optical disc drive's firmware is not signed. Etc. This threat vector exists with each of these devices.

As always security is a trade-off. The threat vector of flashing a backdoored BIOS/firmware is irrelevant for 99% of the market: most people will never be targets of such highly-technical attacks.

PS: I tip my hat off to Sprite_TM; fascinating research! I love to disassemble firmware myself :) I liked how you were able to reverse-engineer the data structures in RAM.

There is another thing to consider:

Adding security to a system imposes costs on the use, maintenance, and support of those systems. Can you imagine the scale issues associated with maintaining PKI over the millions of devices deployed? How about hundreds of millions?

TPM is present in many, many laptops yet most IT departments leave it un-configured. Why? Because when you replace the hard drive and it changes the boot vector parameters, the machine will no longer boot and you have to work inside the auth mechanisms of the TPM infrastructure do to simple hard drive replacements. (nb: I'm not an IT guy so I might be a bit off here, but you get the gist)

So the reasons for NOT including security are pretty damn big compared to the risk. As security guys are fond of saying: "If I can get physical access to your machine, all bets are off." Keyloggers, evil maid BIOS, HID attacks, firmware attacks on peripherals, etc are all possible ways to compromise a device. If you're wondering how STUXNET got into an air-gapped security facility in Iran, I'd bet that a method like this was the prime candidate. That's how I'd do it.

There's not much maintenance involved in the security of an embedded system. Once you do it, assuming you do it right, you're done.

Now, security is a process. You can expect breaches of high value hardware, and need to react to them.

The security model for consoles STARTS with the assumption that the attacker has physical possession of the hardware. This makes things interesting; it's certainly a big differentiator for PCs versus tablets, and one of the reasons why the Windows group at Microsoft has had a lot of trouble making their stuff secure on non-PCs.

But for a hard drive, things should be pretty contained. I estimated it would have taken a couple of weeks to secure one major embedded system I worked on; assuming the interfaces are limited, it's not a huge deal.

Come to think of it, isn't this how Xbox (360?) piracy protections were breached for a long while? People were flashing hacked DVD-ROM firmwares to make burned discs appear like legit ones.

You had to duplicate some key material, IIRC. Also, not as trivial as you might think.

This lets you play "backups" of games, enabling piracy. It doesn't let you run unsigned code (those required other cracks) or sign anything (such as save files).

In Secure Boot environments, this is all untrue - all option ROMs are signed, and the firmware should only accept signed updates. That pushes the problem out to ancillary devices that we've traditionally thought of as safe, but this demonstrates that they really need to reconsider.

I said "typical". Most desktop and server computers do not have or do not fully enable Secure Boot (or else you would not be able to plug in a random 5-year old NIC whose firmware is not signed).

Pretty much all desktop systems shipped in the past 8 months have fully enabled Secure Boot by default. Yes, this means that you can't plug in a random 5-year old NIC and PXE, just like it means you can't plug in an older graphics card and get any output before the OS starts.

The latest generation of SAS enterprise drives do exactly this. All firmware is signed and there is extra hardware to ensure unsigned code is never run. They also disable the JTAG port before the drives leave the factory so there's no opportunity for shenanigans.

These features are required by enterprise customers to prevent just this sort of tampering.

The knee jerk reaction to secure boot-anything from the technical community has been generally "No!", "It's a trap" etc.

The knee jerk reaction is not to secure boot but to who has the ability to set the keys. The technical community likes to be in control of that.

That is closely tied to "who gets to audit the source".

E.g. the FOSS community wasn't a fan of only-trusted-secure-boot when it was microsoft holding the keys and the source and releasing neither.

For PC's and smartphones, which can have higher level security structures, the community is violently against secure boot. For firmware based embedded components, most people aren't so strongly opposed to it.

Not that their opinion is the community's, but the FSF is not against secure boot even for PCs and smartphones, only against "restricted boot" - that is, secure boot without giving the keys to the user.

Here you said "knee jerk" where you meant to say "logically principled and wise". Also "common sense" would work in certain circles.

Secure boot isn't a technical solution to a soft/firmware update problem. It's a control mechanism to solve a management problem. Crazy idea, don't put the interface that has access to firmware on the standard interface. Use its own interface and allow motherboard manufacturers to support it for enterprise/datacenter geared systems.

Good question; how well's that working out on the iPhone?

The iPhone is a /lot/ more complex than a disk drive, with a bunch more ways to talk to the world. (Also, I don't know if the iPhone has a secure boot ROM. It might, whereupon Apple blew it somewhere else).

This stuff /is/ hard. I sat next to a bunch of folks doing this on a console platform and the techniques and exploits were, in a word, breathtaking. But if you know what you're doing and your scope is limited -- to a smallish device, for instance -- you can make it very hard for someone to crack.

something I hadn't really considered about hard disk encryption, before reading this, is how it could protect against compromised disk controllers. if the OS encrypts the data stored on the disk, it would be a lot harder (perhaps, with the right composition, impossible) for a malicious disk controller to insert/change/modify important data (like code, or password files) stored on the computer.

we think of the system as a holistic entity, but turned on its head, you can see how the inside of a computer is just a network...

Deleted comment

yes there is, you encrypt the OS that is stored on the hard drive

edit: sorry, chain looks like this: BIOS reads boot block from hard drive, checks digital signature, signature invalid, boot fails. BIOS check for signature works in concert with TPM to do key storage, so you're fine. albeit, sleeping with 'evil' technology like UEFI and TPM.

You're still going to have the "who can you trust" problem. If you're worried about a compromised disk controller then you ought to be worried about a compromised BIOS or TPM too.

Ought you? Those are all different bits of hardware. One being subverted doesn't mean the others are - if nothing else, the attacker isn't likely to have a vulnerability for every combination of firmware and chip.

This sort of attack is usually going to be more trouble than it's worth to execute, but that doesn't mean it's out of reach for a motivated, educated individual.

You don't need a combination, just compromising the BIOS gets you root. You don't need to separately compromise the disk firmware, it's either or.

The point is that it's the same kind of attack. Relying on the BIOS may save you from an attack on the disk firmware but that doesn't much help if the same class of attack is still effective against the BIOS.

Sure. My point is there's no guarantee the attacker will be able to compromise the TPM or BIOS just because they can compromise the disk controller.

I'd bet most systems see different disk controllers more often than they see different BIOS chips. I'd bet (though not at so high odds) that reasonably secure TPM chips are relatively easier to find outside of the high-end server niche. I'd bet that most non-state actors executing this sort of attack wouldn't have equivalent exploits ready for many different types of hardware.

All of those factors shift risk around (again, what little risk there is from this sort of vulnerability). Forgetting about patching a hole here because of an equal-sized hole over there is silly.

except the TPM is explicitly designed to resist this kind of attack and be tamper-evident / tamper-proof, it's security hardware. so if someone can successfully attack the TPM, you are having a very big problem and will not go to security today.

>so if someone can successfully attack the TPM, you are having a very big problem and will not go to security today.

"That isn't supposed to happen" doesn't mean it won't happen. The Titanic wasn't supposed to sink.

The point is, you want to be able to recover from attacks. It isn't about security today. The premise here is that you've already been compromised to the point that the attacker may have been able to screw with the firmware on your hardware. What you need then is not an assurance that the thing that already happened not be very easy, what you need is a way to hard reset the hardware to a known-good (i.e. factory) state given the assumption that every piece of EEPROM in the machine has been replaced with malicious code. Having something like a jumper on the logic board that will do that in hardware would be a welcome security feature.

Maybe I misunderstood, but didn't the harddrive have direct memory access (DMA)?

The DMA mentioned in the article was internal to the drive, between the hardware interfaces and it's internal cache. The drive did not have DMA access to system memory.

Even though it wasn't discussed in the article, I think firewire and thunderbolt external drives DO have direct DMA access to system memory. Google for SBP-2 and DMA, and a bunch of articles about protecting against firewire attacks against full-disk-encryption (among other things) appear.

I know you're right on FW. I believe DMA was designed in because they recognized that the CPUs of the time weren't powerful enough to move uncompressed full-resolution video from around. I don't know about Thunderbolt, but I'd expect you're right.

Thunderbolt is just pci-express in new clothes, so yes, it does.

Good point.

IIRC DMA was designed to allow peripherals (floppy drives, parallell ports, etc.) to move data from their input lines to regular memory without having to bother the CPU. Uncompressed, full resolution video wasn't too much of a real concern back in the days of the 80286 :-)

Hence http://www.breaknenter.org/projects/inception/ and similar attacks.

It had DMA to its own memory, not to the host computer's memory.

Could this attack compromisse dedicated/rent servers? If so, the attacker could rent, install the exploit on the hardware and terminate the contract. What about cloud servers? Sure there are virtualization layers, but can't those be breached? If so that would pose imense danger given the distributed nature the hardware exploit could render the entire farm vulnerable

The attack could compromise other servers yes. I think the scenario you describe is a possibility, although there are some technical feats that would make wide-scale exploitation difficult - you need to know what you want to modify ahead of time which would be difficult.

Virtualised environments that don't pass the vendor specific commands should be immune to the attack though. As others have said, encryption would probably allow tampered pages to be detected. I'd be interested to see if the modified firmware could ignore new firmware...

> encryption would probably allow tampered pages to be detected

Careful!

It can, but doesn't always. For example, eCryptfs currently doesn't protect against tampering; it uses Cipher Block Chaining (CBC) mode without a HMAC or other signature.

(I'm working with some colleagues to add Galois/Counter Mode (GCM) support to eCryptfs, which does provide some form of tamper-detection.)

Installing linux on a hard drive never sounded impressive before.

Well, to be fair, it's a bit of a pain with UEFI.

But this is really amazing. I'd love to see how it could be extended to other OSes, if possible?

I'm not sure about the other controllers, but if this one has a Cortex M3, then anything that runs on an M3 could hypothetically be ported.

One of the SE sites assembled a list. Shockingly, the question isn't closed yet!

http://electronics.stackexchange.com/questions/27594/what-op...

The Feroceon CPUs are pretty hefty too. They're powering the Marvell Kirkwood platform which is used in things like the Sheevaplug and some of the QNAP TS-* NAS devices. Debian runs great on those. (2.0ghz CPU, 512mb ram). Probably the biggest trouble here is the lack of an MMU (?) .

uclinux can run on mmu-less systems, so I do suspect you can run Linux on this hard drive.

The first hack read on hacker new I have seen for a long time.

After reading this, i know I am unworthy to comment.

However, I will be forwarding this to my wife who gives me a hard time when I, before getting rid of an old computer, remove the HD and give five or so well placed hits with a hammer on the whole HD assembly.

Well, HN is not just about hacks, but I did expect more contente derived from Black Hat.

EDIT: right now on page 1: https://news.ycombinator.com/item?id=6146279

What? You don't consider "growth hacking" real hacking?

That depends if you're just buying the pills from the spams or going to the hardware store and building your own pumps etc.

I really hope you are being ironic.

A fascinating read, and an excellent piece of work.

It reminds me of a similar proof-of-concept hack on a common network card firmware: http://esec-lab.sogeti.com/post/2010/11/21/Presentation-at-H... (the slides linked from that page have a good more technical overview that the blog post).

I think some hard drives like some Seagates has a serial console in the firmware that provides low level access that data recovery companies for example use.

I'd love to read more info about this!

http://www.msfn.org/board/topic/128807-the-solution-for-seag...

http://elabz.com/forums/electronics-repairs/list-of-seagate-...

Thanks for the links :)

While I haven't written it up anywhere, it's also fun to point out that I've had success talking to a Seagate drive by wiring the debug port directly to the TTL serial pins on the debug header of a Linksys WRT54G router.

Does a jellybean part just mean its very common?

Yes, it means that you can buy them like jellybeans (and they're about the same size, and black, which is either the best or worst flavor.)

> Because Linux caches the shadow file (like all files recently accessed), I have to generate a lot of disk activity for the file to be 'pushed out' of the cache

http://linux-mm.org/Drop_Caches

$ echo 3 > /proc/sys/vm/drop_caches

or as non-root

$ echo 3 | sudo tee /proc/sys/vm/drop_caches

I think the idea is you force a disk read or write operation by interacting with the system remotely (for example uploading a file or sending a particular HTTP GET request that ends up in the log), without having shell access or write access to /proc :)

I do not believe that using sudo exactly counts as "non-root".

They meant "when not root".

In the attack scenario here, the author doesn't (yet) have root access.

Great article. But what I came away from it thinking was about how much money is spent by state security institutions to prevent this sort of thing, and yet secrecy breeches at scale are the Walkers, Mannings, and Snowdens using USB sticks and DVD's and copiers.

This is some hard core hacking! Love it! First, as others mentioned, this is why you should always encrypt your os drives. Second, it also got me thinking, how many other devices are open to this kind of attack. Like a network switch, perhaps? Say you buy 100 network switches, alter the firmware to call home and maybe even load a Linux instance, and then resell them on amazon, eBay, or even better, give a "good" cash deal to some local IT company. Then you just seat back and wait for your 100 bots to call home for their new business class Internet homes.

This is incredibly scary. Will HD vendors start implementing firmware code signing anytime soon? Or will some enterprising hackers start working on an open source firmware implementation?

That's a whole world of spying opportunities. A government could make secret deals with hard drive manufacturers. Perhaps not US government, but Taiwan government, if it makes you happier... (I'm from neither country)

Could? How do you know they didn't do it already?

This is very cool. I have a pile of dead and old hard drives. I should see if my local hackerspace has something that can connect to JTAG, and if so, see what secrets the old drives contain.

I remember Dejan Kalijevik from them nokia s/w. Is he talking of the same Dejan?

what is that cortex-M3 chip doing? Did the NSA put it there?

Reply on news.ycombinator.com