Visualizing Ext4

I did a true graphical visualisation of ext4 at FOSDEM a few years ago. The video is here, the visualisation starts at about 20 minutes:

https://archive.fosdem.org/2019/schedule/event/nbdkit/

Edit: If you're confused about the bit where I talk about the filesystem trims in "blue", well that's because apparently the projector at FOSDEM could not render the light blue colour I was using. I didn't know about this while giving the talk, it looked fine on the laptop screen. There's an accompanying video on my blog which is rendered correctly: https://rwmj.wordpress.com/2018/11/04/nbd-graphical-viewer/

I think in many people's goal to "simplify using a computer" it ends up making things that could easily be educational without actively trying to teach you anything -- basically sparks curiosity and informs a bit. (like this great example the author shows here). One example of this (that previously existed in actual computers) is the old trusty red hard drive light telling you that the hard disk is active... if you were like me, you knew the game was going to actually load this time when it showed a particular pattern and you heard the hard drive make a satisfying sequence of fast disk reads. Seems like a nice compromise is to hide the "advanced view" but keep it there for the curious people who likely will be the next generation of computer nerds making the world go 'round.

There's a command-line utility called pixd [1] that generates similar data visualizations on the command line. That said, it only shows static representations of binary data and is not nearly as cool as buredoranna's animated gifs showing filesystem changes over time.

It can be helpful to plot these sorts of pixel arrangements on a Hilbert curve, rather than plotting pixels line by line. I learned this trick from a Ghidra plugin called cantordust [2]. 3blue1brown offers some mathematical intuition for the effectiveness of a Hilbert curve pixel arrangement [3].

[1] https://github.com/FireyFly/pixd

[2] https://inside.battelle.org/blog-details/battelle-publishes-...

[3] https://www.youtube.com/watch?v=3s7h2MHQtxc&t=311s

I found this nbdkit demo for visualizing filesystem IO interesting - https://rwmj.wordpress.com/2018/11/04/nbd-graphical-viewer/

This inspired me to do this experiment:

dd if=/dev/zero bs=1K count=$(( 256 * 3 )) of=a.ext4

mfks.ext4 a.ext4

mkdir a

sudo mount a.ext4 a

cd a

sudo chown 1000:1000 .

python3 -c 'open("a", "wb").write(b"\xff\x00\x00" * 2000)'

python3 -c 'open("b", "wb").write(b"\xff\xff\x00" * 2000)'

python3 -c 'open("c", "wb").write(b"\xff\x00\xff" * 2000)'

cd ..

sudo umount a

(echo -n 'P6\n512 512\n255\n' ; cat a.ext4 ) > a.ppm

convert a.ppm a.png

The resulting a.png is reversible - you can convert it back to .ppm file, skip first 15 bytes and you should get a valid .ext4 back.

Very cool. This kind of data visualisation can really help understand some of the intricacies of how the disk format actually puts things on disk. e.g. the metadata carefully prealloced for at least some usage. I was interested to see what would happen when it ran out of space but unfortunately the animation stopped before that time was reached

This reminded me of innodb_ruby [0]. Super useful set of tools to visualize and learn about the InnoDB structure. Example usage [1].

[0]: https://github.com/jeremycole/innodb_ruby

[1]: https://blog.jcole.us/2014/10/02/visualizing-the-impact-of-o...

If the author is looking at these comments: you could save some bytes transferred and give the user video controls (pause, scrub, adjust speed etc) by converting the gif to a video with something like

  ffmpeg -i ext4.gif -pix_fmt yuv420p -c:v libx264 ext4.mp4

and serving it with

  <video controls>
    <source src="ext4.mp4" type="video/mp4">
  </video>

You can use the Kaitai IDE to visualize various binary formats, down to each byte (or bit). If I remember correctly it has definition files for ext4.

Looking at this diagrams I wonder if there are any file systems that allow for metadata to be stored on a separate device. For example store data on HDD and metadata on an associated SSD drive. I guess the benefits would not be extraordinary to outweigh the added complexity since metadata is much easier to cache in memory.