Why SATA is now obsolete

nowadays all of the things (ha) use flash behind higher level of abstraction, be it emmc, UFS, or SD controller. There are no phone soc firmware tweaks.

To avoid an inherent bottleneck?. Perhaps we'd be better served by a larger number of key:value stores with greater parallelism?

"... the device still has its freedom to control block-allocation decisions, enabling it to execute critical tasks such as garbage collection and wear leveling ... Next, we present Nameless Writes, a new device interface that removes the need for indirection in flash-based SSDs. Nameless writes allow the device to choose the location of a write; only then is the client informed of the name (i.e., address) where the block now resides."

So it sounds like this approach is actually removing responsibility from the filesystem, not the firmware.

[1] http://research.cs.wisc.edu/adsl/Publications/yiying-thesis1...

to use your example: imagine if you could use the fusion-io FS on the intel flash.

also remember that several SSD companies that even had their own flash fab are now gone because their firmware was crap.

The obvious way to implement this is to expose the raw flash to the OS and put an abstraction layer into the OS that does what the drive firmware currently does so it can be used with existing filesystems. Just that would have its own benefits because it would remove the black box from around that code and let people improve it. It would also make the drives less complicated/expensive and remove the burden from smaller manufacturers to maintain solid firmware (which they've often failed to do), and make it a lot easier and more reliable to secure erase drives.

Once you have that you can start looking at improving particular filesystems by taking advantage of the additional information now available to the OS.

The actual memory contents behind the controller are dependent on physical characteristics. Using the FTL firmware, the flash+controller (such as eMMC) vendor is free to do all kinds of tricks, for instance, depending on the quality of a particular batch of NAND. Bad batch? Use more of the spare for error correcting code. Particular memory pattern that generates interference? Tweak the scrambler. Slow? Interleave between a couple of NAND chips. (These examples are hypothetical)

Tying filesystems to the physical layer makes no sense. It would mean I wouldn't be able to use 'dd' to copy a partition to some other device, since it would have different physical characteristics which the filesystem would need to take into account. It would mean that I wouldn't be able to take an iSCSI volume and write it out to disk to 'de-virtualize' virtualized storage.

"Flash-based solid-state drives (SSDs) have revolutionized storage with their high performance. Modern flash-based SSDs virtualize their physical resources with indirection to provide the traditional block interface and hide their internal operations and structures. When using a file system on top of a flash-based SSD, the device indirection layer becomes redundant. Moreover, such indirection comes with a cost both in memory space and in performance. Given that flash-based devices are likely to continue to grow in their sizes and in their markets, we are faced with a terrific challenge: How can we remove the excess indirection and its cost in flash-based SSDs?

We propose the technique of de-indirection to remove the indirection in flashbased SSDs. With de-indirection, the need for device address mappings is removed and physical addresses are stored directly in file system metadata. By doing so the need for large and costly indirect tables is removed, while the device still has its freedom to control block-allocation decisions, enabling it to execute critical tasks such as garbage collection and wear leveling.

In this dissertation, we first discuss our efforts to build an accurate SSD emulator. The emulator works as a Linux pseudo block device and can be used to run real system workloads. The major challenge we found in building the SSD emulator is to accurately model SSDs with parallel planes. We leveraged several techniques to reduce the computational overhead of the emulator. Our evaluation results show that the emulator can accurately model important metrics for common types of SSDs, which is sufficient for the evaluation of various designs in this dissertation and in SSD-related research.

Next, we present Nameless Writes, a new device interface that removes the need for indirection in flash-based SSDs. Nameless writes allow the device to choose the location of a write; only then is the client informed of the name (i.e., address) where the block now resides. We demonstrate the effectiveness of nameless writes by porting the Linux ext3 file system to use an emulated nameless-writing device and show that doing so both reduces space and time overheads, thus making for simpler, less costly, and higher-performance SSD-based storage.

We then describe our efforts to implement nameless writes on real hardware. Most research on flash-based SSDs including our initial evaluation of nameless writes rely on simulation or emulation. However, nameless writes require fundamental changes in the internal workings of the device, its interface to the host operating system, and the host OS. Without implementation in real devices, it can be difficult to judge the true benefit of the nameless writes design. Using the OpenSSD Jasmine board, we develop a prototype of the Nameless Write SSD. While the flash-translation layer changes were straightforward, we discovered unexpected complexities in implementing extensions to the storage interface.

Finally, we discuss a new solution to perform de-indirection, the File System De-Virtualizer (FSDV), which can dynamically remove the cost of indirection in flash-based SSDs. FSDV is a light-weight tool that de-virtualizes data by changing file system pointers to use device physical addresses. Our evaluation results show that FSDV can dynamically reduce indirection mapping table space with only small performance overhead. We also demonstrate that with our design of FSDV, the changes needed in file system, flash devices, and device interface are small."

It can essentially do three operations: read block, write block, erase bunch of blocks at once. For the purposes of filesystem, this is actually an workable storage interface that does not need any additional abstraction layers.

Storage presents block devices to operating system. Spinning Rust and all manner of SSD use lots of clever engineering to appear as a block device.

Block device is the most basic abstraction of all storage. You really want MySQL to manage NAND? ;)

Moving it from firmware to file system does not improve performance by itself, but it enables optimizations. And FTLs in firmware are far from optimal.

In fact in the thesis' abstract, the proposed solution is still a layer of indirection. Instead of the FTL giving an abstracted address space, the "device" is given the responsibility of controlling wear, gc, and parallelism by dynamically reallocating blocks itself, and the filesystem picking up some more of the management stuff.

I am certainly not qualified enough to read this dissertation critically enough to know if it is fundamentally sound theoretically, but as a scientist I know that it is close to meaningless until it has been experimentally tested and reviewed..

"All problems in computer science can be solved by another level of indirection, except for the problem of too many layers of indirection."

Hence this level of indirection, the discrete "drive", is what he thinks should go poof.

We move ever closer to "Computronium".

I'm sure someone will figure something out but pretending it's an HD does kind of solve that issue for now.

A pity we didn't see this until now, but we've adopted your suggestion.

All: you can get this kind of thing taken care of sooner by notifying us at hn@ycombinator.com.