CursedFS – Disk image that is simultaneously ext2 and FAT | Hacker News Reader

CursedFS – Disk image that is simultaneously ext2 and FAT

github.com

188 pointscodezero49 comments

IIRC Apple used to do something like this on their installer DVDs (back when they shipped the OS on DVDs). The Mac OS X installer was on an HFS+ volume pointed to by an Apple Partition Map (which would only be recognized by Mac OS X and the Mac's boot firmware), but the DVD also had an ISO 9660 header (that'd be recognized by Windows) pointing to Windows-compatible drivers for the Mac hardware. Completely separate volumes, with completely separate content.

Microsoft's FAT to NTFS conversion tool has been doing this sort of thing for years, too. It just puts the NTFS master file table and other stuff into the FAT free space, or a designated file. Of course, being a conversion tool the intent was that both sets of data structures described the same files.

* https://docs.microsoft.com/en-gb/windows-server/administrati...

Equally interesting are the tools that convert from the "MBR" style partition table to EFI partitioning. Again, this requires that the MBR-style partition table have free spaces into which the EFI partition table can be put. They have to be in fixed positions, though, unlike the NTFS MFT.

btrfs has a similar convertor with the other quirk that the source ext partition remains as a file in btrfs and deleting it "commit"s the conversion.

https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-conver...

Small quibble, the original file system, named ext2_saved, is a subvolume/snapshot. As long as it's kept around, you can revert the conversion while losing any changes made to the Btrfs file system.

The comments explain the strategy in some detail. ReiserFS conversion is supported in addition to ext[234]. Because Btrfs requires only a little fixed space, just its superblocks, in theory any file system could be converted.

https://github.com/kdave/btrfs-progs/blob/master/convert/mai...

The difference here, though, is that NTFS does not have BTRFS's reserved area at the start of the volume. The BTRFS converter has to relocate foreign data structures out of that reserved area, meaning that the volume is not simultaneously both the old and new formats. Ironically, given what I mentioned, this conflicts most heavily with FAT.

Whereas the only fixed-position data structure in NTFS is the BPB in the Volume Boot Record, which it can even mostly share with FAT. In the FAT to NTFS conversion there is a point where (BPB aside) the two sets of filesystem structures are simultaneously valid.

> btrfs

Is this usually pronounced as individual letters (bee tee arr eff ess), or like "butter FS"?

EDIT: Wikipedia [0] says it's pronounced many ways, such as "butter fuss", "better F S", "butter F S", "b-tree F S", or simply by spelling it out.

[0] https://en.wikipedia.org/wiki/Btrfs

Distro installation images (at least the Fedora ones) are kinda like this. There is a bunch of partitions and weird metadata in the right places, so that you can dd the image on a USB stick (or burn it to a DVD) and it will boot on any BIOS or UEFI PC and on Mac. Yes, there really is a HFS partition on the media just for this. :)

Hybrid discs were common back in the CD-ROM era for games and multimedia discs. There'd be an ISO partition for PC executable installer and a Mac partition for the Mac version, and both partitions would have have pointers into a single copy of the large game data/video files that there wouldn't have been space to have 2 copies of on the disc.

https://en.wikipedia.org/wiki/Hybrid_disc

The concept happened way before that too. Even though the Atari ST and Commodore Amiga had different floppy disk formats (and were bitter rivals), because they shared their main processor (MC68000) a great deal of games and software were available for both.

Occasionally, that happened on the same floppy disk, via a horrific sector format interleaving trick invented by Rob Northen (who did a lot of copy protection stuff at the time). Notably, the Future Publishing magazines ST Format and Amiga Format started out as the combined ST/Amiga Format and the "coverdisk" was exactly that - readable by both, with different files on each machine.

This reminds me that one of the filesystems that could probably be added into the polyglot is Amiga's Fast File System, as the superblock is in the middle. Rumor has it that this minimised seek times from anywhere else on the floppy, but I would think that one doesn't need to read the superblock that often.

When Apple created HFS+ they did a similar hack that would show the disk with a TeachText read me file named “where did all my files go?” if you tried to mount it as HFS.

That means, Macs with older versions of the OS would deal gracefully with a format they were never intended to run.

I remember thinking it was the coolest thing ever.

For extra fun, fill them up with polyglot files!

https://medium.com/swlh/polyglot-files-a-hackers-best-friend...

So the suspense is killing me :) ...if you insert a memory stick formatted this way into a Linux box, which one will show up?

(Forgive my ignorance if the answer is obvious -- I've only ever used Linux as servers, never as a desktop.)

And are there any other OS's that can read ext2? Neither Windows nor macOS seem to be able to, out of the box.

I actually didn't try to let it autodetect... until now.

    ~/cursedfs% sudo mount cursed.img mountpoint/
    mount: /home/kuba/cursedfs/mountpoint: more filesystems detected on /dev/loop0; use -t <type> or wipefs(8).

It's too smart :/

Props to whoever thought to check for the case of fricking polyglot filesystem images, though. Probably intended for something boring like corruption, but still, nice.

Apparently one could create something like this by accident in the early days of Linux: https://twitter.com/makomk/status/1217542964995678209?s=20

I'm pretty sure I managed to create one of these by accident when first using Linux, though with much messier results. Same root cause though: the ext2 and FAT superblocks didn't overlap, and back in the day some of the filesystem creation tools weren't so good at making sure to overwrite conflicting superblocks.

Author here, happy to answer any questions.

What makes this particular FS "cursed" besides the obvious intuitive answer you can glean from the reference twitter thread?

I feel like this name nicely captures the fact that this almost surely has no real-world use and is likely to break stuff (file system autodetection, for one...). I also like to draw parallels to the notion of a cursed idea.

From GitHub:

> It turns out this is surprisingly simple to do: just create a FAT volume with a lot of reserved sectors and put the ext2 into the reserved sectors. This works because the filesystems choose different places to put their superblock: FAT uses the very first sector, while ext2 leaves the first kilobyte unused.

So, is it really two independent file systems? At least when it is read only?

It seems writing to one file system is likely to destroy the other, since they would be unaware of each other's allocated blocks.

> So, is it really two independent file systems?

Yes, they just don't interact with each other.

> It seems writing to one file system is likely to destroy the other, since they would be unaware of each other's allocated blocks.

The entirety of ext2 stays within FAT32's reserved blocks (that is, between the bootsector and the File Allocation Table), so FAT32 won't write over ext2. The inverse is also true, as ext2 knows its formatted size, so it won't write outside of the reserved sectors.

So basically the perfect combination if you want to boot a Raspberry Pi from a single partition for some reason. I like it!

Since the two filesystems are independent, it seems they could have independent content. A filesystem that could boot DOS or Linux.

Can you add ISO 9660 and/or UDF to the mix? Both do not use the first several kilobytes of the volume (hybrid CDs created by genisoimage use that trick to combine HFS with ISO 9660).

From a quick look at the specs, it would seem like it is possible to add either of those, but probably not both. I'll probably add this in "v2.0" - I'm planning to redo this in a way that actually interleaves the filesystems, which would also allow the ext* part to be larger than 32 MB.

And can you add fuse support, to keep at least ext2 and vfat in sync?

The best usecase I can see: /boot, used for both windows and Linux with a special entry to start a live linux for recovery.

Having this partition on a thumbdrive and being able to cat it to /dev/sda1 would be ideal

Since ZIP files have their header at the end of the file, perhaps the disk image could also be a valid ZIP file?

ZIP files also have inline headers with duplicate metadata that may or may not be the same (e.g. filenames can and do differ).

That's why I've always felt conflicted over ZIP. On the one hand, having a contiguous index is nice from a performance optimization standpoint, and that's why ZIP is more commonly adopted to build upon, such as JAR files. On the other hand, the potential for (and actuality of) conflicting metadata is a nightmare from a security perspective, and it violates several rules of thumb in software engineering, such as that there should only ever be a single source of truth (major), a single parse (major), and to avoid duplicating data or context (minor, but can be sufficient to ensure there's only ever a single source of truth and a single parse), which generally ease implementation and help ensure consistent behavior. Likewise, while combining the container format and compression can improve convenience and user experience (especially if there's a single dominate implementation), it's also nice to keep these layers separate, which not only provides simplicity and consistency benefits, but also permits them to evolve independently.

This is also possible with NTFS+ext[2,3,4]. I did work on modifying mkfs.ntfs do create a polyglot filesystem like this, but ran into issues that the windows filesystem check was not happy with the result (though it was mountable as either ext4 or ntfs in Linux). https://pastebin.com/MujcN5MZ

If the media is read-only (or we pretend that it is), is it possible to have the filesystems share data at least for some files, by reusing the blocks?

Yeah, this is one of the things I'd like to explore when I make the formatter more full-fledged and not a 10 line bash script.

I'm hoping to try marking the datastructures of each filesystem as a bad block or something similar in the others, so that as long as the shared data isn't removed or modified, writing is OK.

This is cool as a computer forensics exercise: the FS could look empty on one side but hide a lot of structured informing the other.

I have yet to meet any filesystem forensic software that wouldn't immediately identify this. Pretty much everything I've ever used would just tell you that it found multiple filesystems and show you both. Recovering files is often possible even without any filesystem metadata at all.

To make that point with an example, my go to for any new disk image or real drive is disktype¹ which produces:

    --- cursed.img
    Regular file, size 4 MiB (4194304 bytes)
    FAT12 file system (hints score 4 of 5)
      Volume size 3.916 MiB (4106240 bytes, 2005 clusters of 2 KiB)
    Ext2 file system
      UUID 2D74B033-E8A6-4738-98B2-02BCC3F0D98E (DCE, v4)
      Last mounted at "/home/kuba/cursedfs/mountpoint"
      Volume size 64 KiB (65536 bytes, 64 blocks of 1 KiB)

1. http://disktype.sf.net

The filesystems aren’t synchronized, though. Quite misleading! I’m not sure there’s a technical reason why they couldn’t theoretically be synchronized (either in real-time via a custom fs layer or as cron task, etc).

I don't know if the title has changed, but "disk image that is simultaneously..." made it very clear to me that this would imply a static state carefully crafted file, not some synchronized dynamic file system driver.

Should be called pornfs, you can use it to hide porn.

Seems pretty cool. Not sure where I personally would use it, but when interacting with legacy this could make things much easier.

I didn't envision a usecase, I just found the question of whether this is possible interesting. If I had to guess, this will break more things than it will help with - the filesystems are independent, not "synchronized".

Add encryption support for the ext2 part and it's a decent upper "security through obscurity" layer. If the wrong person finds your USB stick, they will likely only ever notice the FAT filesystem. Like for a spy dead drop use case.

Looks like you, my good sir, have found a use case for this.

Yes, plausible deniability was the immediate use case I was thinking about as well. Curious if/how it would mount on Android (with USB-OTG cable)?

Or more pratical: a partition on the SD card of your phone that can boot to a live linux, then store the live data locally in the ext2 part

I did try to do that with my android phone, but apparently it requires some specific kernel modules. I've to figure out which, and how to cross compile them for the phone kernel.

This seems about as likely to work as hiding emails in the draft folder and not sending them. Computer forensics have looked at all the bytes on the disk, not just the active files reachable from the root directory, for decades.

I mentioned it as an additional layer. And it's not always a sophisticated adversary that might find it.

Snowden fooled everyone with an sd card inside a Rubick's cube.

Snowden did suggest that scene in the movie, but didn't say he'd actually smuggled anything like that.

https://www.vulture.com/2016/09/edward-snowden-snowden-rubik...

> “First of all, I just wanted to say that none of us know [how it happened],” Stone said at the Q&A, when asked about how they came up with it. “[Snowden]’s the only one who knows, and one day he may reveal it. And number two, it was his idea — it was a suggestion that we responded to and ran with.”

I'd imagine they would have done forensics if he was caught with an sd card at a checkpoint, but I'm not sure one would set off a metal detector in the first place. It's more likely a security org would epoxy the sd and usb slots on computers with confidential data, or have audits for any usb devices the hosts encounter.

Some more context...

"He [Snowden] said that the Rubik’s cube was put in for the film and he wouldn’t be divulging how he smuggled out information. However, he added, everyone in the office did have a Rubik’s cube. “So they were floating around and coming in and out all the time,” he said."

https://heavy.com/entertainment/2016/09/snowden-rubiks-cube-...

To me he's at least suggesting a similar glaring shortfall in CIA processes that let him smuggle it out.

JFYI, in order to try and solve a specific problem some 13 years go we envisioned (and also started as POC) a "OFS" or "Overlapped File System", where a filesystem contained also a (partial) image of itself (mappable as image):

http://reboot.pro/topic/2887-boot-bartpe-as-img-from-hdd-wit...

Basically a same file (same extents) was accessible both from the "outer" and from the "inner" filesystems.

Then the idea was abandoned as another valid (and also simple/existing) solution to the actual problem was found through the use of "included in the standard" CDFS graft points:

http://reboot.pro/topic/2887-boot-bartpe-as-img-from-hdd-wit...

Reply on news.ycombinator.com