Hacker News Reader

Comment by Galanwe | Hacker News Reader

I guess I've been living under a rock, because I had the souvenir that disks with volatile write caches did have a small power bank, able to sustain the disk for the few seconds it needed to flush its caches (what it called a BBU?)

Which leads me to another question, should it really be the responsibility of each component to gracefully handle a random, instantaneous power loss?

Couldn't we have a small power bank on the motherboard keeping, say, 2 seconds of power and signaling emergency shutdown to components?

In a sense, that is very similar to sending a SIGTERM to a process to allow it to gracefully stop.

View on HN

Consumer drives are generally junk.

Personally I think it should be the responsibility of the operating system. All filesystem operations on modern computers should be atomic, and either entirely succeed or entirely fail. And the OS should work with hard disk manufacturers to make sure their devices meet whatever expectation the OS makes of them.

Hilariously, the OS already uses transactions for its own filesystem metadata operations. But they don't share that capability with end user applications, which end up either being quietly broken in the event of power loss (most apps) or horribly complex to make up for the OS's shortfalls (most databases).

There are dozens of solutions in userland, from git's content-addressing to on-disk B+trees. There should just be a simple, effective, correct transactional filesystem API provided by the OS that applications can simply use. The best part of this proposal: a solution implemented by the OS directly would probably be faster than what most people would implement in userland anyway. Its ridiculous this doesn't already exist.

There's simply no structural reason for power failure (or application crashes) to be able to put hard disk data into an inconsistent, corrupt state.

McKusick is an OS writer (4.2BSD and 4.3BSD), he is writing about the challenges systems programmers face with deceptive hardware.

Hello Llama, go back to your training data and try to get it right this time.

I think this kind of design would have crap performance (or at least enforce it). The reason is that the disk controller has its own set of RAM that it employs on the other end of the PCIe bus which could be battery backed and be designed to keep the controller alive long enough to flush everything. That’s not possible if the OS is in charge (the power requirements for the full OS are orders of magnitude higher). Additionally, you can’t have hybrid NVMe spinning rust disks with that kind of design.

Finally, it’s unlikely that implementing it in the OS would be faster because there’s a bunch of flash translation logic and garbage collection that has to happen - much faster if all of that happens on the other side of the PCIe bus with a dedicated coprocessor with a dedicated SPI directly to raw flash.

I’m imagining a software solution using some form of journaling / checkpoints. Instead of overwriting data in-place, send all writes to fresh, unused storage and then “commit” by updating a pointer to avoid write amplification. You’d need a bit of extra machinery to handle reads on the old data. As I understand it, NVMe drives already send all writes to unused space and then update pointers to reclaim old capacity.

We could start with a nice implementation in the OS. Probably with a way for applications to opt in and out. And then optimize over time. Optimizations could include better integration with the way NVMe devices actually work, rather than the way they pretend to work to the rest of your computer.

file systems already do journaling and a bunch of other fancy techniques. Drives themselves then have their own versions in addition to that

Additionally, apps can do whatever they want in custom partitions if they need better storage perf (databases for example do this). Getting lower level than that is probably doable, but you run the risk of becoming firmware that is tied to particular drive controller design

Remember that computer storage technology is over 70 years old, many many things have been tried/implemented in that time.

File systems only do journaling for their own metadata.

Has this been tried? It seems like an obvious idea to me, but operating systems have evolved remarkably little in the last few decades. Especially the filesystem.

https://en.wikipedia.org/wiki/USN_Journal

> All filesystem operations on modern computers should be atomic

Well that seems a bit over the top. You would pay the price of full consistency for every write, while just one write every 3 years will fail. It would be insanely slow and wasteful.

Not to mention, that would basically disallow any kind of OS or hardware caching, which is immensely important for 99% of use cases.

> the OS should work with hard disk manufacturers to make sure their devices meet whatever expectation the OS makes of them

This is a bit of wishful thinking. We've seen already on numerous cases (Nvidia?) that OS developpers have very little leverage on hardware manufacturers. The best they could do is issue some kind of "Linux certified" label, which honestly nobody would care about.

Also, I guess there is some form of optimum that users and providers reach in terms of stability versus performance. If one OS provides x10 performance for 99.99% of uses while an other one provides 100% safety for /10 performance, I'm not sure users would choose the latter.

> Not to mention, that would basically disallow any kind of OS or hardware caching, which is immensely important for 99% of use cases.

Why would it disallow caches? Just issue writes to a new location on disk and switch reads across to it after the write transaction has committed.

In any case, most writes aren’t performance sensitive. I’m thinking things like saving a file when code editing. My editor shouldn’t need to do some obscure, fragile renaming dance just to overwrite a file safely. It wouldn’t write 10x slower. Just have the OS do what it needs to do so my editor doesn’t have to.

And big transactions (like installing software or OS updates) could just do the whole update to a new location and then atomically swap to it. It shouldn’t cost any performance at all beyond a few pointers being moved around at the end of the transaction.

Maybe doing it for all writes is too much. Let databases opt out if they want to do their own thing. But I think it would be a much better default. Right now all good software does some version of this themselves. An OS level solution would allow all those hacks to be removed. If anything, I’d expect performance to improve.

I think we have a quiproquo about the level of atomicity here.

You seem to be talking about _application level_ atomicity, like when you issue 4 write() to a file and want an "all or nothing" flush of these 4 writes.

I'm talking about OS level atomicity, as in "I want a write() (or fsync) to block until it is flushed to disk".

Am I understanding correctly?

Atomicity means something either happens entirely or it doesn't happen at all.

> I'm talking about OS level atomicity, as in "I want a write() (or fsync) to block until it is flushed to disk".

Thats not atomicity. Thats a blocking API. And we already have an fsync that does that.

Blocking and atomicity are unrelated properties. You could have a blocking write thats not atomic (so if your computer turns off halfway through the write, it might have only half completed). And you can have an atomic write thats non-blocking (async). Or both or neither.

> Thats not atomicity. Thats a blocking API. > Blocking and atomicity are unrelated

It's the same in this context. fsync uses a blocking API to implement atomicity. That is, after an fsync() all previous writes should have been written.

> we already have an fsync that does that.

Did you actually read the article...? The whole thing is about the fact that disk controllers can eagerly notify that a write was flushed to disk while in fact it is still in the disk write cache, and could have been either partially, fully or partly written, thus tricking fsync().

> It's the same in this context. fsync uses a blocking API to implement atomicity

No, it doesn’t.

In that context fsync is a write barrier. It doesn’t make writes atomic. The difference is this: If you issue a handful of writes then your computer crashes, some, all or none of the writes may have completed. And some writes might be “skewed” - where part of the write succeeded and part did not. Atomicity means a set of actions happens as one indivisible unit, and it either all happens or all doesn’t happen. Think database transactions. Filesystem writes are not like that. They are not atomic operations[1].

Fsync does not change the behaviour of the write system call. Fsync only stalls your program until all writes have been written to disk, at least assuming your drive isn’t lying to you. Just like a memory barrier. If your computer crashes before fsync returns, all bets are off.

[1] With the exception of aligned writes under the device block size. Allegedly. I’d still like to see this actually documented somewhere.

Windows Vista introduced in 2006 a transactional file system API for user applications.

It was recently deprecated because almost nobody uses it.

https://learn.microsoft.com/en-us/windows/win32/fileio/trans...

I wonder why it wasn't used. Any insights? IMO they should have just made the regular write calls atomic across the OS and have applications opt out if they want.

My guess would be that nobody seriously considers Windows servers for enterprise storage. I've worked for several storage companies and Windows support in that world is something that people usually want to overlook, or to provide a bare minimum of. I.e. think about companies like Dell EMC (their storage division) or Infinidat etc... it's mostly Linux with rarely some BSD in the mix.

Non-atomic stuff is simpler on the implementation. Even if the behavior is worse for the user, it's easier to write, and you have fewer requirements to satisfy. So, not having a real pressure from the customers, it's just easier not to do this.

They mention in the docs it was somewhat complicated to use.

Very few apps need more than the ability to atomically swap a file.

And for the complicated multi write cases, a lot of software just use sqlite which deals with it.

> There's simply no structural reason for power failure (or application crashes) to be able to put hard disk data into an inconsistent, corrupt state.

No structural reason no, but a lot of other reasons. You can install your favourite atomic file operations library in your PL of choice and run some benchmarks to identify reason #1.

This is like saying all network traffic should be tcp. For many applications there is simply no good reason and nobody would be willing to pay the additional costs.

> BBU

It's usually a RAID card cache Battery Backup Unit, which keeps the cache contents till the power would be resumed.

HDDs doesn't have anything like this and enterprise grade SSDs have a super capacitor to flush the cache.

Add to this SMR shenanigans.

> should it really be the responsibility of each component to gracefully handle a random, instantaneous power loss?

IMO, no. Sure, theoretically it sounds nice, but nowadays you can have literally gigabytes of data in transit and it's hard to make sure what even if you manage to write it what it would be logically consistent.

> Couldn't we have a small power bank on the motherboard keeping, say, 2 seconds of power and signaling emergency shutdown to components?

You are asking for:

a) a bank of supercaps - pricey, require proper isolation, have a lot of stored power => fails with sparks at best; or

b) LiPo rechargeable battery - pricey, require proper isolation, can pillow => fails with toxic smoke and fires.

And still this doesn't guarantee the data would be consistent.

BTW, there was a thing back in the day, UPS which could be installed in the case (usually 2x5.25"). While it sounds neat, it's the worst of all other variants:

small capacity - no place for a big battery

needs servicing - every 3 years replacement for battery

still can fail with sparks and fire

This battery per device just means more devices would be broken, too.

Just like the numerous ups and raid batteries I've had that reported they were fine until the moment they were actually needed.

ATX PSU must supply power for 16 ms after power off. It also supplies power good signal which can indicate that power is not available. So operating system has a chance to flush caches. Of course that's not few seconds, but should be enough for practical uses. I don't know whether popular operating systems implement this behaviour.

Some server disks might have a backup power source, but that's certainly not a consumer hardware.

My understanding is that disks use the inertia of their spinning platters to power themselves long enough to flush their cache in the event of sudden power loss.

Yeah, I just had a look and it seems BBUs don't actually power the disk. They just keep the volatile cache alive until the disk is powered to flush them.

They use this to finish writing the sector they were writing so they can guarantee they always completely write the sector or not at all.

As far as I know this mechanism is not used for write caches.

Reply on news.ycombinator.com