Cache warming at Netflix: Leveraging EBS for moving petabytes of data
netflixtechblog.medium.com
netflixtechblog.medium.com
Even so (or because) I started my Internet life with a 14.4 kbps modem, went through all the upgrades and now have a 70 Mbits DSL line at home, manage servers all over the world, I was surprised by the incredible amount of data streaming is moving.
But I have been mentioning this for quite some time, we are fundamentally limited by our time spent to watch video. Which means our appetite for Data wont grow forever.
How much Netflix were you watching?
I wonder how that reduced price might influence the technical advice from Netflix.
I wouldn’t be racing to emulate the way they do things.
Netflix or not, emulating how someone else conducts their business without a firm understanding of the underlying reasons is often a bad idea.
An additional issue is that the same interface is used for serving the data out to the cache customers as is used for fetching from S3, (serving hundreds of gigs to hundreds/thousands of clients from ram) which is something your laptop also probably isn't doing.
S3 is big and reliable. It isn't fast or cheap.
What's the underlying technology here? If EBS is implemented over iSCSI AFAIK this is not supported currently available to do in most distros.
> On the destination instance, the EBS volume is mounted with RO (Read-Only) permissions. But the caveat to this is — we do not use a clustered file system on top of EBS. So the destination side can’t see the changes made to EBS volume instantaneously. Instead, the Cache Populator instance will unmount and mount the EBS volume to see the latest changes that happened to the file system. This allows both writers and readers to work concurrently, thereby speeding up the entire warming process.
In the before times, I had read about attaching a disk to two different scsi controllers (in different hosts) and you can also do that with fibre channel or extra fancy double ended sas drives. But that was almost always as a way to access the drive from one system at a time.
> Shared QFS adds a multi-writer global filesystem, allowing multiple machines to read from & write to the same disks concurrently through the use of multi-ported disks or a storage area network. (QFS also has a single-writer/multi-reader mode which can be used to share disks between hosts without the need for a network connection.)
* https://en.wikipedia.org/wiki/QFS
A few jobs ago I inherited a setup where files/artefacts were uploaded to a file server but had to be shared with clients, but for 'legal' reasons the internal host could not be exposed to the outside world—but (some of?) the data could be. So an external disk pack was purchased and one SAS port went to the internal machine and the other SAS port went to the external machine.
"On the destination instance, the EBS volume is mounted with RO (Read-Only) permissions. But the caveat to this is — we do not use a clustered file system on top of EBS. So the destination side can’t see the changes made to EBS volume instantaneously. "
> Standard file systems, such as XFS and EXT4, are not designed to be accessed simultaneously by multiple servers, such as EC2 instances. Using Multi-Attach with a standard file system can result in data corruption or loss, so this is not safe for production workloads. You can use a clustered file system to ensure data resiliency and reliability for production workloads.
> Multi-Attach enabled volumes do not support I/O fencing. I/O fencing protocols control write access in a shared storage environment to maintain data consistency. Your applications must provide write ordering for the attached instances to maintain data consistency.
Or from the Netflix document:
> It ignores files with partial file system metadata. Cache populators can see incomplete file system metadata due to the way it mounts the filesystem, with “No Recovery” option. It proceeds with complete files, leaving incomplete files to be processed by a subsequent iteration. This is explained further in step 7.
AKA you get random read errors, etc, and need to cope.
Behavior of something like this can be expected to be very kernel version and filesystem dependent, even with the most defensive application access strategy.
Pretty much every modern file system is tolerant to computer crashes via journaling. Mounting an in-use FS as read only is effectively the same as mounting a file system after a computer crash. Not self consistent, but close enough it can be repaired in memory (i.e. without touching the disk) via replaying the journal.
Filesystems generally don't replay the journal to memory, but instead replay it to disk as part of going read-write. Conventional replay of the journal would make these issues even worse.
But it's worse than this and the above doesn't solve the problems. Say you've mounted already, the filesystem is changing under you, and assume no cache for simplicity (caching makes this even worse):
For instance, say update A truncates a file and frees up some storage. update B allocates this free space to another file. Update C writes to it.
A different system starts a read, and finds the metadata for the first file. Then it hesitates, and later reads the data from update C in the wrong file.
Most filesystems are not written with this use case in mind-- mount readonly while someone else updates. It's something that I've tried doing with varying degrees of success in the past (sometimes perfect, sometimes with strangeness at the application level, sometimes with kernel panics).
probably cheaper and more reliable too than provisioning a full size volume and relying on implementation quirks in existing single host filesystems and ebs.