Level 2 Advanced Replacement Cache for ZFS

Basically, L2ARC is "Level 2 Advanced Replacement Cache" for ZFS, where level 1 is in RAM, and level 2 is on an SSD (usually a cheap/large MLC SSD, as opposed to an expensive/small SLC SSD for ZIL). Basically, it's using a SSD to act as a huge cache for reads, so they don't have to be serviced by a slower spinning rust array.

Prior to this change, after every system reboot, the L2ARC would be cleared and not used/filled until reads from disk happened. On a system that is rebooted frequently (or even infrequently), this can result in slower performance until the cache has been primed.

My understanding is that that with this change reads can happen from the L2ARC devices after a reboot (the "persistence"), which removes the ramp up of usefulness of the L2ARC.

For some background:

ZFS, a modern WAFL clone [1], has a replacement algorithm called ARC [2] which can concisely be described as a hybridized MRU/MFU (Most Recently Used/Most Frequently Used) replacement algorithm to decide which pages make the most sense for keeping in memory. There is considerable literature surrounding replacement algorithm design, I have little to say about ARC other than it is patented [3] and can be outperformed by newer algorithms.

Note that this is quite different from the traditional approach of FS/buffer-cache design. One usually expects the OS kernel to manage the buffer-cache for you (OS X has it's Unified Buffer Cache (UBC), NT has it's Cache Manager (Cc), etc.). However ZFS includes it's own incredibly complex caching subsystem around. I do not know why they didn't want to improve or modify the Solaris kernel's segmap subsystem but there are consequences to this design. Notably, ZFS's memory usage is quite a bit higher because of ARC.

The idea of performing read-caching in memory with ARC seemed like such a good idea to the ZFS designers that they allow for a second level of ARC to take place: L2ARC. L2ARC essentially runs the ARC algorithm between SSDs and HDDs to, hopefully, speed up the performance of random reads in a ZFS storage pool.

Now to steer back towards what this code dump seems to be about. If you recall from before, ZFS's ARC is a replacement algorithm based on usage and it needs to know which things to put where. This so-called persistent L2ARC remembers where things were on a L2ARC device so that the storage pool can take advantage of the fact that data is on the SSD on, say, a reboot.

Huh? Why did this require extra code? Remember, ARC was about caching: it didn't need to remember anything. When coming back online, complicated things happen: transactions get replayed, metadata integrity needs to be rechecked, etc. Implementing a persistent cache that is crash safe is incredibly difficult but not uncommon: auto-tiering [4] solutions like Fusion Drive [5] have to provide this kind of safety.

[1] http://en.wikipedia.org/wiki/Write_Anywhere_File_Layout

[2] http://en.wikipedia.org/wiki/Adaptive_replacement_cache

[3] http://patft1.uspto.gov/netacgi/nph-Parser?patentnumber=6996...

[4] http://en.wikipedia.org/wiki/Automated_Tiered_Storage

[5] http://en.wikipedia.org/wiki/Fusion_Drive