Linux from Scratch
linuxfromscratch.org
linuxfromscratch.org
It took me a solid weekend to get everything built. I was able to get a basic LFS system built on a Saturday, and on Sunday I did the "Beyond Linux From Scratch" edition. At one point I got stuck trying to debug a weird interaction between systemd and PAM that took me a while to unravel. That was humbling, I thought I knew just about everything about Linux, but turns out there are large areas where I just don't have a clue.
The docs are well written and maintained, so there wasn't a lot of frustration there. Even if you're not an old hand at Linux you can likely get pretty far by just diligently following the instructions.
I struggled a lot more trying to make a decent desktop environment than I did getting the OS setup. I spent so much time trying to get a nice-looking toolbar (polybar) and basic stuff (like how patched fonts work) took me an embarrasingly long time to sort out. I also didn't know what a compositor was, or why you might want one. I enjoyed figuring out the basics of compton, which allowed me to get cool transparent backgrounds on windows[0], although I never did quite figure out how to get rounded corners.
Visit here for inspiration: https://www.reddit.com/r/unixporn/ (safe for work, despite the name).
Pick a window manager, learn how to use it, and go from there. Popular options are i3 (i3-gaps if you want some space between windows), sway (i3, but for Wayland), bspwm, awesome wm. Usually they're used in combination with polybar (highly customizable status bar) and rofi as an application launcher.
Then add a, say, 10Gb volume to your guest instance and party like you're on the road to mad skillz.
How founded are my fears? :-P
Has anyone followed this from start to finish and can now comfortably compile their own linux kernel from scratch?
I also did the whole book (manually!) as a slacker student. I guess if you do it/let it run automatically as you read the book, you'll know your installed OS inside and out...
I still feel pretty comfortable in compiling a linux from source. Userland specialization is also a neat topic.
Bottom line: It's not worth it.
I use debian now.
It wasn’t my call, and it’s a defense move rather than an opinion on the state of Gentoos security. I know they take things very seriously, but my assertions aren’t enough to make my employer feel comfortable.
I started using Linux when it was still necessary to compile your own Kernel to support literally anything going beyond the absolute base setup.
The most difficult parts were bootstrapping the system and getting the glibc stuff right. The rest is indeed rather robotic, although I think there is some kind of package manager now included.
Once you get X and google chrome it becomes pretty self sustainable. Updating the kernel is easy but upgrading the compiler / C library is a nightmare.
A lot of programs you might want to use provide binaries that makes things a bit easier, but there were some annoying days of dependency chasing if you have to install from source. I ended up writing some custom scripts to ./configure, make, and make install into a deb from .tar.gz.
About the initial LFS process, I had messed up a few steps pretty bad, and had to restart / go back one or two times. The worst mistake was embarking on the journey first by installing everything into a USB flash drive. Eventually data corruption / loss ruined the yacc binary (or similar? hard to remember) and I spent days trying to figure out what went wrong. After moving to an SSD it worked better :)
Definitely one of my favorite and most rewarding projects I've done.
I could see value in building a basic Linux from scratch. Beyond that you are just copy pasting commands from READMEs for hundreds of projects.
I don't think it's worth pursuing to a GUI, unless you have a burning desire to customize or fix brokenness in that experience you've had on other Linuxes.
The point is definitely not to continuously manage your own distro you built from scratch, unless that's your job or hobby.
I use the knowledge I gained from the experience to debug many random instances of brokenness on Linux, both personally and professionally. It also made compiling from source a fairly comfy option when necessary (e.g. you need to fork, or backport a piece of software).
If curious see also
2019 https://news.ycombinator.com/item?id=20149111
2016 https://news.ycombinator.com/item?id=11829373
2012 https://news.ycombinator.com/item?id=4488162
2012 https://news.ycombinator.com/item?id=3677350
It's interesting that in 2016 the top comment was about how out-of-date the documentation was, which made it very challenging for the novice to approach. When I did it in 2019, my experience was very different -- I marveled at the quality of the instructions and documentation, as well as how up-to-date it was.
In fact, just this week I was googling for some information about building GPG without having to deal with generating PDF documents, and I stumbled upon an LFS GPG page[0] that had been updated that day. There seems to be a fairly active community who is maintaining the guides.
[0]: http://www.linuxfromscratch.org/blfs/view/svn/postlfs/gnupg....
I did this when I was in early high school. I just got into Linux, and this was super fun to do, it really showed me how an operating system worked and all the moving pieces and how they all fit. I did it all on a 1ghz machine with 256mb of ram, and some slow DSL. It took me a 4-5 days back then, but it was a blast for me.
I haven't done it since. But today I am a senior devops engineer, and that knowledge was so valuable to me today. It is so much easier for me to reason about systems and linux in general.
Honestly, I am really tempted to go through this again as a weekend project in a VM or something
We use Debian in an IoT product similar in footprint to a pi. I know Linux OK, but some areas are still pretty opaque to me (or evolved, once upon a time I knew unit pretty well, systemd, I can manipulate it, but I'm no guru).
So would LFS be a good fit for me? Or better some other educational adventure?
Think of it like the complete LFS, with integrated basic X11 (optional), automated. Either selfhosted from installation, OR crosscompiled even from Microsoft Windows to whatever else is currently supported by it. One single way to operate on diverse systems. Even more advanced than the build system of FreeBSD, by which Gentoos Portage has been inspired. Similar thing for its PKGSRC, which is the package manager for anything not contained in that base system.
( http://pkgsrc.org / https://pkgsrc.se )
If you ever experienced that, you realize how big the mindfuck of any Linux is, and wish for being in a parallel universe where something like this would be the basic standard to build upon.
Skim the http://netbsd.org/docs/guide/en/
edit: To be clear, this is not the normal mode of operation, you don't have to do this to use it, but it is integrated, so that you can generate install images from which to bootstrap other systems.
OK, impractical when you have only one system and are new to this, but otherwise?
> Symlink Style Package Management
For this I highly recommend https://zolk3ri.name/cgit/zpkg/ which I have been using for years now. It works wonderfully.
Environment variables and their defaults:
ZPKG_SRC = ~/.local/pkg
ZPKG_DST = ~/.local
ZPKG_DB = ~/.db
By the way, I have noticed that someone created a package manager with the same name, but its initial commit was in 2019, while this one's was in 2017.
I actually had a class not too different in spirit from that once, but it wasn't even close to graduate level. It was a class I took at Brunswick Community College while working on an A.S. degree in Computer Programming.
Here's how it went: I had been at BCC before, when I got an Associate in General Education degree. I had taken a few computer related classes, and had gotten to know the main instructor who taught most of the computer programming stuff. Maybe it helped that we were both named "Phil" but we struck up a mild acquaintanceship if not an outright friendship.
Fast forward a few years, after I had transfered to UNC-W, then dropped out, then returned to BCC to get another associate degree. I signed up for a class titled "Operating Systems", which turned out to be more of a "Survey of Operating Systems" not any kind of "OS internals" (keep in mind, we were at a community college here). Anyway, I walk into class the first day and my old friend Phil is the instructor. He looks at me, smiles, and says "I have a deal for you."
I don't know what the rest of the class did all semester, but he pointed me to a room off to the side of the main room we were in, that held:
1. An ancient IBM AS/400 minicomputer
2. The biggest stack of those big-ass reel tapes I've ever seen
3. A ginormous stack of manuals
The deal he offers me? Get an operating system installed on this machine, get it booting (except they don't call it "boot" in AS/400 land, they call it "IPL" for Initial Program Load or something to that effect), put it on the Internet, and be able to log into it remotely. If I do that, no matter how long it takes, I get an "A" for the class, and don't have to do any of the other work the others are doing.
Apparently this monstrosity had been donated to the college by a local company that decommissioned it. And I guess the college had nobody on staff (besides instructor Phil) who knew anything about the things. Not sure how long it had been sitting there before the job fell into my lap, but I digress...
As it were, it took a little while (at the time I'd never touched an AS/400 before), but I managed to get it going. Then I spent the rest of the semester coming in and just playing around on that thing. I got my "A" and even more, Phil helped me land my first real IT job as an AS/400 operator. Which all led, indirectly, through fits and starts, to where I am today.
I don't work specifically with AS/400 (iSeries now, I think) machines any more, but I have a sort of sentimental attachment to them I guess. Not so much for RPG/IV and SEU though.
Gentoo has its problems, but I can't imagine life without it. The day I can't use Gentoo is the day I quit tech.
The people who I know that use Gentoo are addicted to constant updates (they apply them more than once a day).
There is something about compiling non-trivial software, like the kernel, with all that text flashing by the screen, that is purely satisfying to watch.
Quite the opposite for me. I struggle to do it once a month. I would often go months without it, but that turned out to be a really bad idea.
No - it's the USE flags that make it awesome. And the ability to apply custom patches, etc as needed.
Knowing how all the pieces fit together from doing LFS and how to bootstrap them was really helpful even though we never compiled anything. It also helped when things get wedged (won't boot or a GUI freezes) and you can fix it without having to power cycle the machine, losing work.
I haven't taken an OS class or messed around with xv6 (I'd like to), but I think that would help with a different set of things.
Then go back and automate it yourself. Make your own build process, scripts, auto download sources and automatically apply sets of patches. Make your own fakeroot system and assemble all three pieces into a working image file, auto tar the whole thing and make it easily deployable.
That... is how you come to fully understand everything that goes into a basic bare-bones yet fully functional, <50MB Linux Distribution.
Then do it again, but with a cross compilation toolchain (CLFS but using the build tooling you've just created).
I did this year's ago for the OG Raspberry Pi. I was frustrated by the lack of a minimal distro that just had the basics for serving websites and nothing else.
Fast forward 6 months, all the above completed, amazed by how far I'd come... only to realize CentOS had been released for the RPi in the meantime.
I'll never regret that time spent, though. It really does unlock a "next level" understanding of what a distro is. There is no magic, just a lot of hard work.
I did end up with a flawed system with funny bugs. I couldn't access the network with the usual tools but elinks managed to bypass something and get a TCP stack somewhere. But it failed to render properly on tty. No surprise I had no clue what I did wrong but it made me realize how much plumbing there was.
But LFS was a wonderful learning opportunity for me: it allows you to build an understanding of the nitty-gritty bits of the system, but hurdles one has to overcome are very painful for a daily system. I think things have improved a lot in early 2000s when better build scripts were introduced and you actually had links to packages you needed: back before that, there was no way anyone would build a LFS system over a weekend, as I see people reporting here (not the least because most Internet access was dial-up).
So first up: have time and desire to learn. I would probably appreciate re-doing this with systemd to really internalize how things have changed from the inittab days.
Just checked it quick again, the basic steps remain the same.
It still requires a separate partition, I would add a chapter about: how to use loop device to mount a file as a device, then how to get LFS built on that loop-file and run it inside a virtualbox or kvm, to make it non-intrusive and safe to play with for beginners.
Many beginners will install their linux under one partition, and they probably do not have a second partition to try LFS, so a file-based approach, with virtual machine in mind, could be easier to attract users?
I do think this is a good education, although realistically I think you will learn as much (and probably more) on Arch these days, especially if you have to create your own AUR packages for some reason.
Apart from the kudos of saying "I did it." and learning a bit about the construction of GNU-Linux, there is no benefit in Linux from Scratch except as something crossed-off your bucket-list.
These days I 'buy built' and download a distro that many experts have put lots of time into producing something that I, alone, could never compete with.
Was a fantastic way to learn how to grow a system. Horrible gateway drug to Gentoo Linux, of course.
I think is mandatory if you really want to understand how Linux base os works.
http://linuxfromscratch.org/lfs/view/stable/prologue/package...
Buildroot's target systems don't include a compiler - it's more for real-world embedded linux system generation.
https://buildroot.org/downloads/manual/manual.html#faq-no-co...
Here's one example though of someone who went on to create a new distribution: http://www.nutyx.org/en/
To create a distribution, you need a package manager of some sort to manage software updates, and while LFS discusses some possible package managers that's not really its focus.