The File System Paradox (2007) | Hacker News Reader

The File System Paradox (2007)

technet.microsoft.com

73 pointsluu34 comments

Back in the days of MS-DOS, I used to wonder why MSDOS.SYS had both the H (hidden) and S (system) attributes set, whereas COMMAND.COM, without which a disk wouldn't boot, had none of these.

Only much later did I realise that the S attribute actually meant something like "the bootloader references this file by disk sector - DEFRAG.EXE, leave well alone". So MSDOS.SYS/IO.SYS would contain the filesystem driver, and then load COMMAND.COM "by name".

Well for one thing you could replace COMMAND.COM.

https://en.wikipedia.org/wiki/4DOS

I don't think that was ever true. DOS 7.x proved this as MSDOS.SYS became a text file. The DOS boot process always required IO.SYS to load MSDOS.SYS [0].

[0] http://www.dewassoc.com/support/msdos/dos_boot_process.htm

You don't think what was ever true? What you said after doesn't contradict anything they said.

If you read the link, the file attributes (Hidden and/or System) of place no bearing on the MBR/Bootloader. Even the OP Article also acknowledged that. You can also read about this on wikipedia [0]. Bootloaders are essentially JeOSes [1] (in modern parlance) that can do root directory searches for key boot/OS files, independent of their file attributes. I've seen custom bootloaders depend on certain files being in certain locations or having certain names. Dave Dunfield's [2] "DOS-less" EMBEDPC [3] comes to mind as a bootloader that supported this.

[0] http://en.wikipedia.org/wiki/MSDOS.SYS

[1] http://en.wikipedia.org/wiki/Just_enough_operating_system

[2] http://www.classiccmp.org/dunfield/dos/index.htm

[3] http://www.classiccmp.org/dunfield/dos/embedpc.zip

OP was saying (roughly) that the file attributes were used by, for example, DEFRAG.EXE to avoid having files like MSDOS.sys re-arranged ("defragmented") on disk because they need to be in certain locations the bootloader expects.

That's fully compatible with what you're saying. Yes, you need those flags to keep boot-up working properly and, yes, the bootloader ignores the flags.

The files only need to be in certain logical locations (like the root of the drive), not physical locations. Fragmentation is not an issue either. See what FreeDOS say about this [0].

[0] http://help.fdos.org/en/hhstndrd/base/attrib.htm

About TempleOS, Terry Davis wrote:

TerryADavis 30 minutes ago [dead]

My bootloader uses a fixed LBA block address. When I install Kernel.BIN onto the disk, I patch the bootloader with the LBA of Kernel.BIN. It is stored contiguously. My bootloader knows nothing about filesystems.

God said it was the right way.

They appear to be banned. All of their comments are dead even the non-controversial ones like this. Too bad.

Why don't they let up/down votes sort out the quality of comments and stop banning non-spammers. Seems heavy handed.

Just let people upvote banned users comments out of oblivion if they think it's worth seeing.

Raymond is almost always 100% right, but in this case he (or Adrian) is a bit off. The mini filesystems in NTLDR can absolutely traverse subdirectories. After all, things like the kernel are found in places like C:\WINDOWS\SYSTEM32\NTOSKRNL.EXE and NTLDR has to load that. There is a "nanofilesystem" in the boot sector(s) which only knows how to load C:\NTLDR, but NTLDR could easily restore a hibernation file that was in a subdirectory.

The real reason these are in the root is because There Can Be Only One. You can have multiple versions of windows in different directories on the same drive, but only one NTLDR or BOOT.INI or HIBERFIL.SYS. If you put them in subdirectories, then you can now have more than one and NTLDR doesn't know which one to use.

And as somebody else pointed out, this all got rewritten after Windows XP.

Of course, now that there is EFI, the "BIOS" contains its own rather fully-featured FAT filesystem driver, so it could theoretically boot and resume from hibernate without requiring the files be in the root.

I don't think that's a better way of doing it. The "dumb and simple" BIOS way, where all it did was load the first sector of the boot device into memory and jump to it, was far easier to understand and debug.

It's still better than pulling down the entire weight and redundancy of GRUB. I guess you could make the argument that the bootloader is sort of a vestigial necessity from historical shortcomings in PC firmwares that have now been, if not necessarily improved per se, at least juiced up.

But yes, we're heading into the direction of booting kernels as EFI executables from stub loaders, at least in the contemporary GNU/Linux world. Further, you can just hardcode a cpio archive of an initramfs inside a kernel configured for building with EFI stub support.

Downsides are your CONFIG_CMDLINE (boot params) becomes hardcoded into the kernel configuration, unless you use something like efibootmgr to keep regenerating kernels with different boot parameters. But given the trend of lowering and hiding customizability, I expect it to be par for the course. systemd merging in Gummiboot into their source tree should give you an idea of what's to come in the future.

(U)EFI has a very open architecture. MS could well write an NTFS driver and register it with the firmware to be loaded prior to handing off to the NT boot manager.

These sort of things tend to go against the grain of platform portability, though, even though pretty much the only platforms Windows appears to support these days are (or have been) consolidating around UEFI and ACPI.

Cool explanation but I wonder why other operating systems don't have the same restriction. On OSX, for example, it used to be here: /private/var/vm/sleepimage

I suppose they have a clever boot loader with advanced drivers?

I think bootloaders for Linux have more stages: the initial problem is the same, but then GRUB loads its configuration from [/boot]/grub2/grub.cfg. If it can load a config file, it can also load FS drivers or hibernation files, and prepare the actual kernel boot more subtlety.

Keep in mind this article was written in 2007 about the majority of NT installations at that time. Vista introduced a new unified boot architecture that replaced NTLDR with something that could work on BIOS and UEFI as well (and removed the monolithic EFI loader).

I don't know whether they do, but a clever alternative is to have a stupid boot loader. You don't need to know how to traverse directories, you just need to know what blocks on disk to read. So, just write the number of the first block of the hibernation file to a known location (say in the first disk block) and you are all set.

Well, all: the file may be fragmented. Just store information about where to find all the blocks in the first block of the hibernation file (alternative view of this is that you have a very simple alternative file system on the disk that stores only one file (or a few, if you also want the ability to find the kernel for a cold boot)

Risk with such approaches is that they may not survive defragmentation of the drive. That's a minor risk for a hibernation file, I would say.

It's not minor. It's a freaking pain in the rear to have to fix this every time you defragment or move/resize a partition.

before the os goes into hibernation it can write the location of the hibernation image file to a special place for the bootloader to find. OSX also reserves space for the hibernation image, so it shouldn't get fragmented. And the defrag program probably doesn't increase the fragmentation for large files anyway.

If your defragmenter knows about the hibernation system, it will update the 'pointer' to the hibernation file.

Also, you can only hibernate an OS that knows about hibernation. Such an OS would be foolish if it wouldn't write the current location of the hibernation file whenever it went into hibernation.

So, this only is a problem if you hibernate your system, defragment its disk on another system using a defragmenter that doesn't know about hibernation files, and then try to wake up from hibernation. That might happen to people using an old version of a defragmenter, but I doubt it will happen often.

It's because the bootloader contains support for given filesystem. GRUB supports several filesystems, and so does UEFI (FAT read-write) and EFI on Macs (FAT read-write, HFS+ read only).

Back in the good old days, when LILO was the preferred boot loader for Linux, it did not have any idea how to read filesystems, and the block locations were simply recorded into the MBR.

I remember when GRUB first came out, thinking it was somewhat heretical to have filesystem code in a boot loader.

Slackware still uses LILO. Although most people will look at you like you're a fossil for using it these days, I've found it quite simple to configure and it has generally worked without a hitch.

Grub1 was not so bad to configure either, but Grub2 looks like a real stinker.

The biggest gotcha with LILO is that thing about writing the config to the boot sector. But then i have not used it in some years...

Personally I prefer syslinux, where the config resides in a easy-to-understand textfile, /boot/syslinux/syslinux.cfg, which can be on FAT or ext2/3/4 or btrfs. The only limitation is that all files that syslinux reads must be on the same partition.

Grub2 OTOH is definitely an abomination. Whoever thought it a good idea to require compiling the bootloader config into a binary should be given a thorough slapping with a medium-sized trout. That thing is an overengineered solution looking for a problem.

Are you sure that GRUB2 saves the config as a binary? I think on my Arch update-grub generates /boot/grub/grub.cfg which looks somewhat similar to good old menu.lst.

The binary part may be a bit harsh. But if i recall correctly grub2 compiles one config into another, and the result is topped with a "do not edit!" warning.

If you poke into the resulting config, you'll notice that it's just convenience†, and you could write the file by hand.

† in the same way that autotools is "convenience" saving you from writing makefiles and configure scripts.

Yes you can write the file by hand, but what you write is going to get overwritten on the next system update to grub2.

However, it is not due to fault of grub2, but due to distro-specific scripts that "helpfully" generate grub config.

HFS+ stores the extent records for certain essential files (the bootloader, the catalog file, the extents overflow file, the journal, etc) in the file system header. So in order to find their physical locations you just need to read in the files system header and parse it.

Its always fun to understand how such basic things are done when you don't have an OS loaded.

A very good article on how does your computer boot up.

http://www.tldp.org/HOWTO/Unix-and-Internet-Fundamentals-HOW...

Reply on news.ycombinator.com