Hacker News Reader

Comment by elevenbits | Hacker News Reader

Would be interesting if this evolves into a full filesystem implementation in hardware (they talk about Object Drive but aren't focused on that yet). Some interesting future possibilities:

- A cross-platform filesystem that you could read/write from Windows, macOS, Linux, iOS, Android etc. Imagine having a single disk that could boot any computer operating system without having to manage partitions and boot records!

- Significantly improved filesystem performance as it's implemented in hardware.

- Better guarantees of write flushing (as SSD can include RAM + tiny battery) that translate into higher level filesystem objects. You could say, writeFile(key, data, flush_full, completion) and receive a callback when the file is on disk. All independent of the OS or kernel version you're running on.

- Native async support is a huge win

Already the performance is looking insane. Would love to get away from the OS dictating filesystem choice and performance.

View on HN

> Would love to get away from the OS dictating filesystem choice and performance.

Not wishing to be a wet blanket, but do you really believe having disk manufacturers dictating filesystem choice and performance will actually be an improvement?

They already have a huge impact on performance and in my experience most people don't care which filesystem they use as long as the performance is good.

Absolutely. SSDs have gone from 100+MB/s random (already a huge improvement over the spinning rust of the time) to maxing out PCIe x4. Over the same timeline, what have filesystems done?

- Apple made APFS, which seems to show roughly the same performance as HFS+ on SSD hardware. Some benchmarks show faster, some show slower, but it's certainly no 5X improvement across the board.

- btrfs and ZFS have some amazing features, but remain niche and don't seem to yet have wide deployment. Linux distros seem locked on EXT4.

- Windows has NTFS (and ReFS for the server)

At a consumer level, I care about: a) reliability b) performance c) power. SSD manufacturers so far have delivered hugely on all three. I'd love to see how much more they could deliver by moving a lot of filesystem functionality onto the device. Sure the first few iterations might suck like early SSDs did, but in a few years?

The big thing is that you can choose which file system you want. On linux and want a btrfs filesystem? No problem. Oh you want to read a NTFS volume? Just download the ntfs packages and you're good to go.

BTRFS, NTFS, ZFS, EXT4, and APFS/HSF+ don't really matter for general users sure, but each has certain advantages and limitations that make the flexibility should you need it very useful.

Key word being "on linux". On the other 99.9% of devices, I have no such choice and the filesystem required by the OS simply presents a layer of incompatibility. Modern OSs all run on essentially the same hardware (same CPU architecture even!) but store the same data in different, incompatible ways. I want the filesystem to join the list of things that are no longer OS-specific.

I want my storage hardware to have a higher level API than "block device". I don't want to format a drive in some OS-specific format, I want a built-in format that I can access from anything. I want this format to be durable, well specced so that I can easily access historical data. I want embedded devices to support richer storage functionality than afforded by FAT32+.

Consider that before S3, we stored files on servers in bespoke ways where we had to worry about permissions and filesystem differences (path too long!) and encryption and durability and a hundred things that we now... don't. Instead we have a simple API with dozens of conforming backend providers, including the ability to do it yourself. Imagine if computer storage experienced a similar revolution?

ZFS, exFAT, and UFS are all supported on at least NT, Darwin, Linux, and I believe the major BSDs. I will happily grant that only exFAT is likely to work for embedded devices, but you do have real options.

But S3 is so unified because has heavy limitations -- only basic file ops, no consistency guarantees. It actually has less capability than FAT32 in some ways (For example, no attributes, no way to say "create file but do not overwrite it")

Sure, someone can make super-limited drive which only does 3 basic ops (read, "upsert", list). Would it be useful for general purpose computing? I doubt it.

> It actually has less capability than FAT32 in some ways

I'm not sure I'd agree with that:

- "no attribute": S3 does actually have quite sophisticated ACLs and permissions that can be set at the object level

- "no way to say "create file but do not overwrite it": there are a few ways you can do that. Versioning, bucket policies, etc.

In fact I disagree with both yourself and the GP with your assumption that S3 is simple. If you want to do anything "enterprisey" with S3 then you absolutely do need to be aware of the maximum size of allowed objects (these days it's 5TB but it wasn't so long ago when it was only around 5GB). Then there's bandwidth costs (which means you need to also take into account VPC traffic and thus S3 endpoints) and ACLs (soooo many different places you can set permissions on objects - it's easy for someone untrained to get totally lost). There's different storage options in S3 too - depending on the frequency you want to request the data.

S3 is only superficially simple - but then the same can be said for any file system. If you just want somewhere to dump files then any fs will work. But when you start needing to performance tune and do all the other considerations that the GP highlighted with local server storage, then S3 quickly becomes as complicated as every other option out there.

>Absolutely. SSDs have gone from 100+MB/s random (already a huge improvement over the spinning rust of the time) to maxing out PCIe x4. Over the same timeline, what have filesystems done?

This is the most "apples to oranges" comparison I've ever seen!

Filesystem do something totally different than hard disks or SSDs, and you can't replace one with another. One stores data in hardware form, the others defines the format, reliability (e.g. journaling), querying, metadata, attributes, security, etc of the data.

The point is there, though: hardware has more room to improve than software does, when it comes to IO-heavy tasks.

Err... Firmware has a history of being closed, buggy and quickly non-supported by the manufacturer. Block level abstraction exposed to the client is as brain-dead as it gets - and it is a good thing. Just imagine that your SSD had an S3-type interface for five years. And you only could use the tools that Amazon released five years ago. No updated versions. No bug fixes. And no "backend changes" to workaround bugs/etc and that was your only copy of the data.

We'd have hacks to support all of our OS level concepts like writable, readable, listable, executable, hidden, system, archive. How would someone implement tail, what about keeping something open for writing? I'm not sure any of this stuff is thought through except by OS writers. If we suddenly put this task to device manufacturers, would tail suddenly never work again? And if that doesn't work, what kind of monstrosity will happen if we simply attempt to start windows or linux on it?

And even though our OSes have bugs, at least we can reasonably debug and patch them. The same cannot be said of completely proprietary and invisible firmware.

Uh, alot of this is in firmware on the controllers of the SSDs. And often it's buggy as fuck to the point that Linux has blacklists hardcoded to deal with crappy firmware in SSDs.

I confess I'm not sure why people are talking about bugs now. I was talking about how hardware can access higher levels of performance and efficiently guarantee more things (eg. early persistence guarantees) than software can.

Because the hardware is implemented with software.

You are rather missing the argument. If KV SSDs are much faster than doing the same with a software DB on top of a block abstraction, is that not proof that these are inequivalent? GPUs run software too, but that doesn't mean it's in anyone's best interests to run that on the CPU—not all software is the same, precisely because the hardware is different.

If the firmware has severe bugs (which is incredibly common with all firmware) and is hard to debug and patch, the performance is irrelevant. Not to mention that there are many examples of existing firmware bugs where the performance of "simple" hard drives tanked -- adding more code to already-buggy firmware that has a history of performance problems is unlikely to improve performance (let alone reliability).

This is a rather different claim to saying there wouldn't be a benefit. I'm not saying filesystem accelerator SSDs would necessarily be in sum a good thing, just that in principle it seems like the kind of thing that should perform better than today's abstractions.

The more I think about it, the more I'm thinking that allowing SSDs to present a higher level API than block storage would be an incredible win. Consider:

- Currently, SSD controller firmware goes to incredible lengths to present something that is not a contiguous block device (due to the way flash works) as a contiguous block device to the OS. Likely this includes a huge amount of logic for storing and updating the mapping between logical blocks and actual hardware storage location. Since the control APIs currently are very low level, the controller basically has to buffer everything, transform that into what the flash needs to do, and then execute, not unlike how a CPU decodes instructions and converts to uops or reorders. Presumably this is quite difficult as early controllers had all sorts of problems such as performance tanking when the disk got close to full.

- The filesystem then pretends that the very non-uniform storage provided by the SSD is actually a uniform block and introduces a layer of metadata in a bespoke format to determine the actual disk locations of files, directories, and their associated metadata. Along with that, it can include lots of very nice extra features such as encryption, journaling, hashing (for integrity and de-duplication), CoW, symlinks, hardlinks, directory hardlinks, etc.

- It seems likely then that were the storage API to be raised to a higher level (files and metadata, rather than block storage), the SSD controller can implement every one of those features with superior reliability, performance, and power usage. We've already seen where software encryption went to hardware encryption -- the result is that the encryption is mostly transparent to performance. What would happen if a file's extents mapping was simply the physical location mapping the SSD controller is already doing? If if could transparently perform hashing on read/write or keep a proper journal of file operations? Handle CoW transparently?

And all of that implemented in a way that those just become standard storage features rather than requiring a certain OS to run.

I'd really like to go other direction, and have SSDs expose non-uniform storage.

Right now, if my filesystem dies, I have a ton of tools and manuals for recovery. This is the format is well known, and diagnostics tools are easily available.

If my SSD dies, it's just gone. In two cases I had, the drive just was not allowing any file reads at all -- I had zero chance to recover the data. The drive internals are totally opaque.

So no, I would prefer my SSDs to be super dumb -- I do not trust the manufacturers to get recovery tools right.

I'd love to see something like Open Channel SSD's ( http://lightnvm.io/ , https://openchannelssd.readthedocs.io/en/latest/ ) becoming mainstream.

Spec at http://lightnvm.io/docs/OCSSD-2_0-20180129.pdf

> This specification defines a class of SSDs named Open-Channel SSDs. They are different from traditional SSDs in that they expose the internal parallelism of an SSD and allow the host to manage them. Maintaining the parallelism on the host enables benefits such as I/O isolation, predictable latencies, and software-defined non-volatile memory management, that scales to hundreds or thousands of workers per terabyte.

>We've already seen where software encryption went to hardware encryption -- the result is that the encryption is mostly transparent to performance

Yeah but are you really sure that hardware encryption is safer? Are you sure that it's not secretly saving the key somewhere where it can be extracted? Are you sure that the hardware encryption works at all and that it's not just writing the data unencrypted?

In fact there are so many examples of hardware encryption being objectively broken[1]. I wouldn't trust it at all, and treat it as plain-text storage that probably has more firmware bugs that will lead to data loss.

[1]: https://news.ycombinator.com/item?id=18382975

You do realize its all software at one point right ?

It just that instead of running on main cpu, it would run on SSD controller's microchip. And will probably be binary only, that people wont be able to inspect or fix.

Samsung has good hardware, but not that great software.

I'd rather have the opposite. SSD hardware exposing their internals, to os developers.

The people you have building incremental changes with great momentum aren't the people you want working on a revolutionary change with a hugely different problem space, at least from a business perspective.

Nowadays most SSD failures happen in the SSD controller, not in the physical storage. I wouldn't trust SSD manufacturers to write reliable software. At least not without waiting 3 years to weed out the obviously bad brands.

Yea agreed. Also if hardware manufacturers did a bigger share of the file and storage system, they could in theory add some hardware optimisations for the most important filesystem software requirements. The kind of things you can't do in software alone.

What most people care about is not what developers care about. I sure hope drives will not dictate what kind of file system we have to use.

Current filesystems are software-defined on top of block-based storage, I don't see a reason why we can't still have software-defined filesystem layers built on top of key-value-based storage. The key-value storage just lets you potentially keep a more meaningful lookup id versus block storage so could offload a little bit of work from the filesystem layer, while keeping the exact same filesystem API in place.

> They already have a huge impact on performance and in my experience most people don't care which filesystem they use as long as the performance is good.

Some of us care about not losing metadata randomly too... or suddenly having hardlinks get duplicated...

Yes! -- because they at least have an incentive to make it work with more than one operating system. Even if it's not great, at least it'll be compatible.

It's insane that there still isn't a standard file system. They all do 99% the same thing now. Making them incompatible is just a pissing contest, and the users lose.

Making it work isn't the same as not making it utter shite. Samsung has been pretty problematic with resolving firmware issues.

So basically, we need another exFAT, but for SSDs?

They most certainly don’t do 99% the same thing, especially now. We’re seeing an advent of new CoW filesystems (ReFS, APFS) alongside the ones that have been around for a while (ZFS, BTRFS). These all differ on multiple axes: handling RAID/volume management at the FS layer (which all do significantly differently), inherent/conventional filename behavior (e.g. 2 byte characters vs single byte, Unicode normalization), direct support of various POSIX filesystem features, bitrot prevention. Plus all the other common filesystems still in use from the last few decades (ext4, xfs, ntfs).

That’s not even including facts of how operating systems use filesystems that aren’t technically enforced in the filesystems itself (e.g. case on Windows). There is a lot more going on here than a pissing contest at user’s expense. In fact, if e.g. Windows threw in the towel and started using UTF-8 file names with case sensitivity tomorrow, the users would be the first to yell that half their software no longer works!

> A cross-platform filesystem that you could read/write from Windows, macOS, Linux, iOS, Android etc.

I don't see how this is related to have hardware key/value. The only thing affecting filesystem support is os developers supporting filesystems.

> having a single disk that could boot any computer operating system without having to manage partitions and boot records!

Again, not related to object stores at all. You still need your UEFI to be able to find your boot images on the disk.

> receive a callback when the file is on disk. All independent of the OS or kernel version you're running on.

Hardware interupts are always going to have to be handled by the OS.

> Native async support is a huge win.

All I/O is already asynch on the hardware level. And again, I don't see the relation to the technology actually discussed in the article.

I'm really not sure about your first point.

> A cross-platform filesystem that you could read/write from Windows, macOS, Linux, iOS, Android etc.

Fat32? There's plenty of newer file systems, for the most part if they aren't cross-platform by default then either the vendor has their own solution (Apple) or your system is fairly modular by design (linux).

> Imagine having a single disk that could boot any computer operating system without having to manage partitions and boot records!

The BIOS/UEFI Still needs to know what to boot, so the drive must be partitioned somehow. You'll want to store that data somewhere, and now you have a boot record.

---

I'm not these drives solve either of those issues. Now the other ones you mentioned, those I agree with.

One of the first things that happened with the advent of k/v databases is that devs immediately re-implemented relational joins in software.

I suppose it’s only fitting that the top-rated HN comment is someone suggesting that a k/v SSD should be used to reimplement...filesystems.

When you say "in hardware", it's not really what you think. These drives generally have low-power general-purpose CPUs in them, like perhaps a little ARM Cortex-M. Any filesystem implemented in the drive will just be regular code running on a different CPU.

I really don't see huge wins here. This will just increase the surface area for bugs, with less ability to fix them, and less ability to recover data when things go wrong.

I definitely do not trust drive manufacturers to write high quality software. It's just not one of their core competencies.

The CPUs may not be as little as you think. Samsung's latest consumer NVMe SSDs - 970 line - can have up to 6 watt TDP at load (and have metal heatspreaders even), and if I remember correctly, the controller is like an 8-core ARM chip that runs at 3 GHz or something. That's more powerful than an Android phone.

> A cross-platform filesystem that you could read/write from Windows, macOS, Linux, iOS, Android etc. Imagine having a single disk that could boot any computer operating system without having to manage partitions and boot records!

You would still have to manage namespaces, so that would be partitions in disguise, and you would still have to manage some special object name for "the thing you load and run to bootstrap the operating system", so that would be boot records in disguise, and you would still have OS specific ways to store their OS specific meta data that represents OS specific semantics, so that would be more or less filesystems in disguise. The only remaining thing would be basic interoperability ... or in other words: What FAT provides now.

> - Significantly improved filesystem performance as it's implemented in hardware.

Obviously, no such thing woud be implemented in hardware, but rather just in software on a processor on the disk device. So, it's still software, just with the guarantee that you can not possibly fix it, improve it, or adapt it to your needs.

> - Better guarantees of write flushing (as SSD can include RAM + tiny battery) that translate into higher level filesystem objects. You could say, writeFile(key, data, flush_full, completion) and receive a callback when the file is on disk. All independent of the OS or kernel version you're running on.

Which has exactly nothing to do with the topic at hand. Any "normal" SSD could do all of that. The fact that storage devices more often than not don't care about correctness has nothing to do with whether that is possible, and everything with whether you can build a cheaper product that gets better benchmark results if you don't care. Having an even more complex interface, if anything, is likely to make manufacturers cheat even more on this.

> - Native async support is a huge win

Wut?

> Already the performance is looking insane. Would love to get away from the OS dictating filesystem choice and performance.

So, you would like to get away from being able to implement any filesystem you like, or choose one of a dozen or so that fits your needs best, because that free choice in your mind somehow translates to "dictating your choice", to a situation where the manufacturer of the hardware dictates the filsystem you use (that is, the filesystem that is implemented by the filesystem driver running on the processor on the disk hardware), because ... having no choice of filesystem gives you the freedom to choose, I suppose?

Furthermore, I highly doubt this would be the reality. I imagine SeagateFS, Western Digital HyperFS, paying more for a better FS.

I wouldn't be surprised if we get Filesystem DRM, maybe Filesystem subscriptions, you can only backup Seagate drives to genuine Seagate backup systems, and that costs extra.

But maybe I'm just cynical....

You'd also want a way to identify related groups of files and handle permissions, I suggest a prefix convention so every file that starts with "/home/flukus/" belongs to me and isn't modifiable by other users. /s

From a high level users perspective file systems are already key value stores so this isn't going to change anything. There may be some gains from moving to hardware, but given hardware manufacturers track record with buggy, broken, closed source software I wouldn't be too keen.

> A cross-platform filesystem that you could read/write from Windows, macOS, Linux, iOS, Android etc.

"Future possibilities" ... in 2019.

It frustrates me to no end that there are some areas where programmers are willing to go to extreme lengths to be perfectly compatible (C/C++, Unicode, IEEE FP, ethernet, TCP/IP/HTTP, HTML/CSS/JS, PNG/JPEG, ...), and in other areas it's just accepted that everybody is completely incompatible and users have to deal with the mess (newline characters, SQL dialects, filesystems, syscalls, opcodes, graphics APIs, driver models, some multimedia codecs still, ...).

I can put a "Pile of Poo" character in a text file and have it work correctly everywhere from my telephone to a server in Finland -- but I can't put that text file on a disk and expect to see the file on two PCs that happen to be running different operating systems.

I wish we were far enough along that I could complain we can't (efficiently) run one compiled program on any operating system -- there's really no technical reason we can't do that -- but we're not even close. We can't even look at a list of data files on a disk.

What is everybody working on?! We don't need any more web 2.0. We need to fix 50 years of historical accidents and incompatibilities. In 1969, Richard Hamming said "Today we stand on each other's feet", and AFAICT nothing since then has changed. Nobody cooperates with anybody else. As Alan Kay once said, "The real computer revolution hasn't happened yet". Adding more JavaScript trackers is not going to help us get there.

OK, rant over. Sorry.

Those JavaScript trackers pays for WebAssembly. Right not it's about the closest to the holy grail for cross-platform applications. The (for-profit) companies pushing the state of the art in browser technologies all happen to run on ads.

*now

>filesystem implementation in hardware

there is no hardware, its just someone else's embedded firmware

A cross-platform filesystem that you could read/write from Windows, macOS, Linux, iOS, Android

I wonder if that is possible to implement right now... It’s simple to adapt something like a raspberry pi to become a usb device, pluggable into another computer. It could emulate a USB drive, but depending upon the host computer’s OS, it could present the raw drive data as a different file system, fat/ntfs/ext4/whatever, and translate the host’s reads and writes back into the appropriate reads and writes of an internal ‘universal’ filesystem.

You’d have to make compromises to cope with FS-specific features (like how to translate users/groups onto a more basic FAT representation of the storage, etc) but in principle it could work.

...except I’m not sure if a usb device can detect the OS of the system it is plugged into?

I think that's partly what MTP was supposed to be, without the os detection - a way to present the files to a host OS without needing to expose the underlying FS

> ...except I’m not sure if a usb device can detect the OS of the system it is plugged into?

Great idea, and if done inside an FPGA or just using some microcontroller without involving a full OS into the stack, it would be absolutely perfect. But hey, isn't that essentially what they are doing but they are changing the "API".

When it comes to fingerprinting the host OS from a USB device: https://media.blackhat.com/us-13/US-13-Davis-Deriving-Intell...

I'd rather not have yet another poorly cooled, poorly programmed chip that's prone to catastrophic failure (most common failure of SSDs is the controller, and USB 3.0 controllers are known to fail a lot) in the middle.

If you are willing to have your transport layer be IP, then the disks could expose NFS. USB has the ethernet device class, so a usb device could like an ethernet adapter. There would need to be a discovery phase when a device is inserted, probably using something like zeroconf.

It seems unlikely that the performance would be significantly better from implementing a filesystem in hardware, especially since current filesystems can store data in system memory as a cache, which is always going to be lower latency and higher bandwidth than an external device. It is also nice to be able to decide at runtime how much memory to allocate to the filesystem cache vs other uses.

I think you'd still need bit-level data adressing to make sure you can do stuff like RAID to make sure you can handle drive failures with parity, or you need to always 1:1 the data.

Reply on news.ycombinator.com