Hacker News Reader

Comment by elevenbits | Hacker News Reader

Absolutely. SSDs have gone from 100+MB/s random (already a huge improvement over the spinning rust of the time) to maxing out PCIe x4. Over the same timeline, what have filesystems done?

- Apple made APFS, which seems to show roughly the same performance as HFS+ on SSD hardware. Some benchmarks show faster, some show slower, but it's certainly no 5X improvement across the board.

- btrfs and ZFS have some amazing features, but remain niche and don't seem to yet have wide deployment. Linux distros seem locked on EXT4.

- Windows has NTFS (and ReFS for the server)

At a consumer level, I care about: a) reliability b) performance c) power. SSD manufacturers so far have delivered hugely on all three. I'd love to see how much more they could deliver by moving a lot of filesystem functionality onto the device. Sure the first few iterations might suck like early SSDs did, but in a few years?

View on HN

>Absolutely. SSDs have gone from 100+MB/s random (already a huge improvement over the spinning rust of the time) to maxing out PCIe x4. Over the same timeline, what have filesystems done?

This is the most "apples to oranges" comparison I've ever seen!

Filesystem do something totally different than hard disks or SSDs, and you can't replace one with another. One stores data in hardware form, the others defines the format, reliability (e.g. journaling), querying, metadata, attributes, security, etc of the data.

The point is there, though: hardware has more room to improve than software does, when it comes to IO-heavy tasks.

Err... Firmware has a history of being closed, buggy and quickly non-supported by the manufacturer. Block level abstraction exposed to the client is as brain-dead as it gets - and it is a good thing. Just imagine that your SSD had an S3-type interface for five years. And you only could use the tools that Amazon released five years ago. No updated versions. No bug fixes. And no "backend changes" to workaround bugs/etc and that was your only copy of the data.

We'd have hacks to support all of our OS level concepts like writable, readable, listable, executable, hidden, system, archive. How would someone implement tail, what about keeping something open for writing? I'm not sure any of this stuff is thought through except by OS writers. If we suddenly put this task to device manufacturers, would tail suddenly never work again? And if that doesn't work, what kind of monstrosity will happen if we simply attempt to start windows or linux on it?

And even though our OSes have bugs, at least we can reasonably debug and patch them. The same cannot be said of completely proprietary and invisible firmware.

Uh, alot of this is in firmware on the controllers of the SSDs. And often it's buggy as fuck to the point that Linux has blacklists hardcoded to deal with crappy firmware in SSDs.

I confess I'm not sure why people are talking about bugs now. I was talking about how hardware can access higher levels of performance and efficiently guarantee more things (eg. early persistence guarantees) than software can.

If the firmware has severe bugs (which is incredibly common with all firmware) and is hard to debug and patch, the performance is irrelevant. Not to mention that there are many examples of existing firmware bugs where the performance of "simple" hard drives tanked -- adding more code to already-buggy firmware that has a history of performance problems is unlikely to improve performance (let alone reliability).

This is a rather different claim to saying there wouldn't be a benefit. I'm not saying filesystem accelerator SSDs would necessarily be in sum a good thing, just that in principle it seems like the kind of thing that should perform better than today's abstractions.

Because the hardware is implemented with software.

You are rather missing the argument. If KV SSDs are much faster than doing the same with a software DB on top of a block abstraction, is that not proof that these are inequivalent? GPUs run software too, but that doesn't mean it's in anyone's best interests to run that on the CPU—not all software is the same, precisely because the hardware is different.

The more I think about it, the more I'm thinking that allowing SSDs to present a higher level API than block storage would be an incredible win. Consider:

- Currently, SSD controller firmware goes to incredible lengths to present something that is not a contiguous block device (due to the way flash works) as a contiguous block device to the OS. Likely this includes a huge amount of logic for storing and updating the mapping between logical blocks and actual hardware storage location. Since the control APIs currently are very low level, the controller basically has to buffer everything, transform that into what the flash needs to do, and then execute, not unlike how a CPU decodes instructions and converts to uops or reorders. Presumably this is quite difficult as early controllers had all sorts of problems such as performance tanking when the disk got close to full.

- The filesystem then pretends that the very non-uniform storage provided by the SSD is actually a uniform block and introduces a layer of metadata in a bespoke format to determine the actual disk locations of files, directories, and their associated metadata. Along with that, it can include lots of very nice extra features such as encryption, journaling, hashing (for integrity and de-duplication), CoW, symlinks, hardlinks, directory hardlinks, etc.

- It seems likely then that were the storage API to be raised to a higher level (files and metadata, rather than block storage), the SSD controller can implement every one of those features with superior reliability, performance, and power usage. We've already seen where software encryption went to hardware encryption -- the result is that the encryption is mostly transparent to performance. What would happen if a file's extents mapping was simply the physical location mapping the SSD controller is already doing? If if could transparently perform hashing on read/write or keep a proper journal of file operations? Handle CoW transparently?

And all of that implemented in a way that those just become standard storage features rather than requiring a certain OS to run.

I'd really like to go other direction, and have SSDs expose non-uniform storage.

Right now, if my filesystem dies, I have a ton of tools and manuals for recovery. This is the format is well known, and diagnostics tools are easily available.

If my SSD dies, it's just gone. In two cases I had, the drive just was not allowing any file reads at all -- I had zero chance to recover the data. The drive internals are totally opaque.

So no, I would prefer my SSDs to be super dumb -- I do not trust the manufacturers to get recovery tools right.

I'd love to see something like Open Channel SSD's ( http://lightnvm.io/ , https://openchannelssd.readthedocs.io/en/latest/ ) becoming mainstream.

Spec at http://lightnvm.io/docs/OCSSD-2_0-20180129.pdf

> This specification defines a class of SSDs named Open-Channel SSDs. They are different from traditional SSDs in that they expose the internal parallelism of an SSD and allow the host to manage them. Maintaining the parallelism on the host enables benefits such as I/O isolation, predictable latencies, and software-defined non-volatile memory management, that scales to hundreds or thousands of workers per terabyte.

>We've already seen where software encryption went to hardware encryption -- the result is that the encryption is mostly transparent to performance

Yeah but are you really sure that hardware encryption is safer? Are you sure that it's not secretly saving the key somewhere where it can be extracted? Are you sure that the hardware encryption works at all and that it's not just writing the data unencrypted?

In fact there are so many examples of hardware encryption being objectively broken[1]. I wouldn't trust it at all, and treat it as plain-text storage that probably has more firmware bugs that will lead to data loss.

[1]: https://news.ycombinator.com/item?id=18382975

You do realize its all software at one point right ?

It just that instead of running on main cpu, it would run on SSD controller's microchip. And will probably be binary only, that people wont be able to inspect or fix.

Samsung has good hardware, but not that great software.

I'd rather have the opposite. SSD hardware exposing their internals, to os developers.

The big thing is that you can choose which file system you want. On linux and want a btrfs filesystem? No problem. Oh you want to read a NTFS volume? Just download the ntfs packages and you're good to go.

BTRFS, NTFS, ZFS, EXT4, and APFS/HSF+ don't really matter for general users sure, but each has certain advantages and limitations that make the flexibility should you need it very useful.

Key word being "on linux". On the other 99.9% of devices, I have no such choice and the filesystem required by the OS simply presents a layer of incompatibility. Modern OSs all run on essentially the same hardware (same CPU architecture even!) but store the same data in different, incompatible ways. I want the filesystem to join the list of things that are no longer OS-specific.

I want my storage hardware to have a higher level API than "block device". I don't want to format a drive in some OS-specific format, I want a built-in format that I can access from anything. I want this format to be durable, well specced so that I can easily access historical data. I want embedded devices to support richer storage functionality than afforded by FAT32+.

Consider that before S3, we stored files on servers in bespoke ways where we had to worry about permissions and filesystem differences (path too long!) and encryption and durability and a hundred things that we now... don't. Instead we have a simple API with dozens of conforming backend providers, including the ability to do it yourself. Imagine if computer storage experienced a similar revolution?

But S3 is so unified because has heavy limitations -- only basic file ops, no consistency guarantees. It actually has less capability than FAT32 in some ways (For example, no attributes, no way to say "create file but do not overwrite it")

Sure, someone can make super-limited drive which only does 3 basic ops (read, "upsert", list). Would it be useful for general purpose computing? I doubt it.

> It actually has less capability than FAT32 in some ways

I'm not sure I'd agree with that:

- "no attribute": S3 does actually have quite sophisticated ACLs and permissions that can be set at the object level

- "no way to say "create file but do not overwrite it": there are a few ways you can do that. Versioning, bucket policies, etc.

In fact I disagree with both yourself and the GP with your assumption that S3 is simple. If you want to do anything "enterprisey" with S3 then you absolutely do need to be aware of the maximum size of allowed objects (these days it's 5TB but it wasn't so long ago when it was only around 5GB). Then there's bandwidth costs (which means you need to also take into account VPC traffic and thus S3 endpoints) and ACLs (soooo many different places you can set permissions on objects - it's easy for someone untrained to get totally lost). There's different storage options in S3 too - depending on the frequency you want to request the data.

S3 is only superficially simple - but then the same can be said for any file system. If you just want somewhere to dump files then any fs will work. But when you start needing to performance tune and do all the other considerations that the GP highlighted with local server storage, then S3 quickly becomes as complicated as every other option out there.

ZFS, exFAT, and UFS are all supported on at least NT, Darwin, Linux, and I believe the major BSDs. I will happily grant that only exFAT is likely to work for embedded devices, but you do have real options.

Nowadays most SSD failures happen in the SSD controller, not in the physical storage. I wouldn't trust SSD manufacturers to write reliable software. At least not without waiting 3 years to weed out the obviously bad brands.

The people you have building incremental changes with great momentum aren't the people you want working on a revolutionary change with a hugely different problem space, at least from a business perspective.

Reply on news.ycombinator.com