Hacker News Reader

Comment by ww520 | Hacker News Reader

Appending log based file system has a lot of appeals. A lot of hard problems, like atomic write, become trivial. A log based FS shares similar properties as the transaction log in RDBMS, making consistency and recovery easy. Write performance is fantastic. With enough cache memory, read performance should be good, too. The only down-size is the performance hit when doing garbage collection. More research in this area would be helpful.

View on HN

Totally agree. Log based file system makes things much easier, such as atomic write, data consistency and failure recovery. Along with COW semantics, transaction isolation also becomes easier. By its append-only writing nature, file versioning is trivial as well. Garbage collection is performance downside indeed, but it can be mitigated by COW switching and delayed bulk collection IMHO. Overall, the cost is worthwhile and that is what I am currently doing in ZboxFS: https://github.com/zboxfs/zbox.

That looks promising. Worth taking a look.

https://en.wikipedia.org/wiki/Journaling_file_system

https://en.wikipedia.org/wiki/Page_cache

Journaling FS only puts metadata and FS commands in log, not structuring the FS itself as an appending log.

Haven't log-based file systems been a thing for a while now? https://en.wikipedia.org/wiki/Log-structured_file_system

They have, but they had fallen way out of favor for a couple of reasons - write amplification, cost/complexity of segment cleaning, poor performance for some workloads, etc. They've experienced a bit of a resurgence thanks to flash and SMR, though. Personally, I've always thought log-structured filesystems were elegant and wish they'd been more actively developed during the long "winter" during which their COW/write-anywhere cousins stole all the limelight.

Was hoping there are more research attempt at the Log Structured FS. For example,

- Borrow some ideas from generational garbage collection. Young generation in SSD (or mirrored in RAM) with copying GC to get rid of old versions of fast changing data blocks.

- Utilize some deduplication techniques with content based signature.

I think elements of that generationality are the foundations of the Log Structured Merge Trees used by KV Stores like LevelDB and RocksDB. Atleast I think that's the same concept, I'm not well read on filesystems.

Yes, LSMT is a good example of pushing the idea of a hybrid append log and in memory data structure further.

However, LSMT is for relatively smaller data set, i.e. ordered key-value. It has worse write amplification than a simple append log. The level-0 memtable flushed to the write-ahead-log counts as one write. Writing to the level-1 sorted files counts as 2nd write. Merging the sorted files counts as 3rd write. There're 2~3 writes per change.

Also it doesn't offer help to address the frequent update block problem. All versions of a data change are written to disk. A merge is needed to get rid of the old versions.

But it has a number of good implementation ideas that can be borrowed.

Reply on news.ycombinator.com