The Linux Upskill Challenge | Hacker News Reader

The Linux Upskill Challenge

theleo.zone

275 pointsunderanalyzer88 comments

So, in years past, I lamented that people didn’t know basic sysadmin skills and therefore invented complex solutions to what are rather simple problems of configuration within common Linux software. They’d write entirely new pieces of software to deal with something I saw as trivial, and they did this because they didn’t know a solution existed.

These days, I don’t lament this at all. As I see it, people wanted to make new on their own instead of learning what was there. This is just how younger people are. As they get older they naturally want to learn how things used to be, and the cycle will repeat. They will be the ones lamenting, and the younger folks will make it all new.

I don’t remember who said it but: those who don’t understand UNIX are doomed to reinvent it poorly.

I've seen that quote a thousand times and I disagree with it profusely. After 30 years of it now on my part, I know we can and should do better. And that does mean reinventing it, not being resigned to the status quo.

Care to give some examples? I am still too young in Unix years to be wise enough to judge what is good and what is bad.

Top three bits of pain from me:

1. Constant weak reparsing of formats in shell pipelines.

2. Completely dangerous and unsafe API and principal programming language.

3. Trying to remember several different domain specific languages.

> 2. Completely dangerous and unsafe API and principal programming language.

Apart from the fact that these days you can hardly call C principal (yes, it's still used for the Linux and BSD kernels, but quite rarely for any new userland project at all), what are your main gripes with the API? It is well understood, stable, most people know its limitations, it's being used by most servers and clients on the Internet, so I'm curious why you call it "completely dangerous and unsafe".

What?!?! Most of the userland is C and we're still finding gaping holes in it after decades.

As for the API, the entire permissions system is just a shit show (inflexible, setgid hack, no MAC, terrible ACL implementation), ioctl (hack job), signals (totally inconsistent handling), links (hard complexity), open(2) flag craziness, then the semi userland crap particularly PAM recently and anything which involves delegated permissions from yp/nis/ldap. Urgh it's hell.

I'd rather write win32 if I'm honest these days. At least they have the honesty to break the API when it's dead by adding Ex on the end of a new one.

> I'd rather write win32 if I'm honest these days

You might need to train yourself to ignore the ads blinking at you from your start menu and taskbar though.

I’m on windows 11 and haven’t actually noticed any yet. I’m not sure if that’s because I’m immune to all forms of advertising or I turned something off though.

> And that does mean reinventing it, not being resigned to the status quo.

There are 2 options: Either we incrementally improve, or we break with tradition. I'm pretty sure people have been saying that we should break with tradition and do better than unix at least since the 90s, but for just as long the incremental improvements keep coming and keep being perfectly good. Modern unix-family OSs are not the unix-family OSs of 30 years ago; we have crash-consistent filesystems, your choice of improved security systems, assorted jail/container systems for compatibility/scaling/security, better service management, etc. Or perhaps phrased differently:

> After 30 years of it now on my part, I know we can and should do better.

We are - just in place rather than throwing out a perfectly good system and ecosystem.

Sure,reinvent it, but not on top of the previous foundation , replace it.

What usually happens is that people reinvent functionality from layer 3 at layer 47.

> As I see it, people wanted to make new on their own instead of learning what was there. This is just how younger people are.

You cut the divide between "forgers" and "honers" along an age boundary, but that view of reality wrong. It has nothing to do with age, the mindset has other reasons. Also, as with most personality characteristics, they rarely change once developed, I wonder what gave you the idea.

When I was young, I was a "honer" all along and have remained thus. I was interested in the history of Linux and its ancestors, and preferred making use of past and modifying contemporary software rather than writing new software.

I think the next level above this might be https://sadservers.com/

Discussion: https://news.ycombinator.com/item?id=33344142 (597 points | 3 months ago | 128 comments)

hehe..

lsof | grep bad.log

ps -ef | grep (pid)

sudo apt remove python3

kill -9 (pid)

I got lucky:

ps ax showed a python3 process running /home/ubuntu/badlog.py, so I knew to kill that. If it wasn't named so obviously it would've been much harder to find it.

lsof is list open files. Good for finding things like what process is writing to a log file. You could also go dig around in /proc

Oh hi, author here, AMA :-)

I'm happy that a lot of people use and find SadServers beneficial while not costing me a lot of money. Still a lot of features to implement on the website and a shrinking backlog of scenario ideas to materialize (if somebody has ideas for Linux/Docker/Kubernetes etc troubleshooting scenarios, please let me know).

I've just tried several of your challenges and they're all painfully accurate for real-world scenarios. I will definitely point people at these next time I'll get asked how I learned to fix <random Linux configuration problem>!

As for suggestions, here are some random things I needed to do recently:

- resize the boot partition of an OS (don't know how doable this is with your vserver setup, maybe use one of those WASM Linux emulators?)

- set up a systemd service/timer/socket that starts at the right time and responds correctly to reloads/restarts

- set up IPv6 correctly

- troubleshoot why a device wasn't connecting to the WiFi (DHCP service problem!)

- set up a VPN (wireguard/openvpn/etc). Expert mode: make the remote endpoint have an A/AAAA record that the server isn't listening on

- troubleshoot why some of my devices couldn't ssh into a server despite the pubkeys being in the authorized_keys folder (old sshd version didn't understand the most recent key algorithm!). Bonus problem: ~/.ssh had the wrong permissions so the authorized keys weren't loading.

- renew an ACME/letsencrypt certificate in nginx in proxy mode (location / was proxied but location /.well-known/... shouldn't have been!)

- check your preferred smtp daemon to see if it's set as an open relay

- upgrade postgres from an old version to a new version without data loss (hard mode: the partition postgres uses by default doesn't have the free space to make a copy and migrate the data)

- figure out why the firewall isn't blocking port 1234 despite UFW being enabled and a block-all rule being present (it was because of Docker iptables rules overriding UFW rules)

- update a package that has some kind of dependency issue (i.e. an external repository that is no longer needed)

- make Ubuntu shut up about Ubuntu Pro and stop it from fetching ads on ssh login

- alter a systemd service file so that it no longer runs as root (hard mode: set up dynamic users and other hardening features)

thanks! (some of these ideas are not possible to implement in the current setup but I can explore some of the other ones)

I'm very happy to see this continuing. The original creator passed it onto the current maintainer before he died.

He was super nice.

Honestly most of the people that approach me to learn Linux, I have them follow the Gentoo Handbook to compile and configure a Linux workstation from scratch.

This has been my go-to training approach for decades. Many of those individuals are now in high-seniority roles that encompasses Linux Administration.

It is a tried and true method. If you want to be _good_ with Linux (including administration), build it: Follow the Gentoo Handbook, and restart every time something is wrong or fails.

Alternatively, if you want to be a semi-competent user you can go the path of installing a user-friendly distribution (like Ubuntu, etc.) and get various Windows-based workstreams working (Gaming, Windows-Only Software through Wine, etc.) But this is less likely to push you to the same tier.

I assume this is very similar to the Linux from scratch book? https://www.linuxfromscratch.org/

Any advantages or disadvantages?

Gentoo handbook method is a lot friendlier than following LFS, yet you stil have to deal somewhat with linux internals. IMO the main advantage over LFS is that is close enough to day to day linux administration that it serves more as a real experience than LFS which seems more like an academic experiment. OTOH, watching stuff compile is the worst part. I would complement the experience with using something like arch for a few weeks as main driver. Quicker than watch compilation logs, still you have to deal enough with linux internals.

Well, “Linux” is a broad thing that involves a ton of software. I always advocate that people research a problem domain and what software solutions exist for that domain in a standard Linux installation.

I have a friend who's asked me to teach him a bit about Linux and computers in general when we hang out. We haven't done much, basically I let him muck around in a TTY for a bit one day. I've considered just giving him one of my old spare PCs and having him build his own Gentoo system.

Any tips for following this approach?

This is what I did when I was about 14 on my pentium 4 to keep my room warm in winter and sleep with white noise. It also worked great for me to understand where everything was in my install.

Unfortunately when the gentoo wiki got nuked like 12 years ago I was in the middle of a reinstall and so I couldn’t finish it and moved on to other things. Used arch for a while but it became my second job to keep it running and it wasn’t worth it. As things started branching out I no longer understand what’s what. Every time there is some drama over some decision some distro made I feel like I don’t know either. Wayland or no wayland, snap, etc.

I’m on team vim, but it is still interesting to me that the challenge has vim but not emacs. I would think that an introduction to Linux would include both.

Probably because emacs is not installed by default?

TRAMP could also work, depending on the situation.

Regardless of which team you're on or why, the base installs of almost every distro includes vi or vim, but many leave out emacs and nano. Since you have to pick one editor to be the first one taught, it makes sense to teach vim to avoid losing the learner.

This is as it should be.

Emacs is about enlightenment; enlightenment is a path to be walked not a destination; it is a journey you must choose.

Along the way, you might accidentally edit text files. That's not the point.

Anyway, makes sense to me why it's not default included.

> Emacs is about enlightenment;

And that's why I'm on team Vim. I rather not use a tool that needs/uses cult-like slogans to convince people to keep them hooked. I want the usefulness to be the reason, not some kind of belief system.

It was a joke my friend

It's just a bit of playfulness.

Nobody is 'hooked' on Emacs because of slogans. A person uses Emacs because they like it.

But... that's what alcoholics say!

A defining feature of cults is how difficult it is to leave...

:q!

Have you ever tried exiting emacs? It's not easier than vim.

Lol "team vim," spoken like someone who doesn't know what Emacs is at all

I use a vim implemented in Emacs Lisp -- evil-mode -- so I can have access to the rest of the usefulness of Emacs while also having everything model editing has to offer

The GP is joking, but I'll happily use a vim in Emacs while the reverse is impossible, and you can figure out what that means for your belief system

I also just use regular vim sometimes

Tools aren't teams

He said, rooting for a tool.

Surely we can all agree that nano (pico) is a toy compared to either.

Nano has a lot of functionality. Not really a toy.

> enlightenment is a path to be walked not a destination

There is no path without an end.

I don't understand why so many distros ship vi instead of vim. Vim was realeased as an improvement in 1991!!!

To save precious disk space

Including Vi-improved would be complicit in the assumption there was an imperfection in vi.

I am not aware of many distros shipping the original vi today. However, a lot of them ship busybox, which is a self-contained and lightweight collection of standard UNIX tools, including a simplified vi. If you have a UNIX shell on an embedded system, there is a good chance it is running busybox.

vim is an order of magnitude bigger than the entire busybox package. In fact, it may be bigger than the entire system.

Clearly, you would only want to focus on the superior editor. /smile

In this day and age, an intro to Linux should probably recommend an editor which has CUA shortcuts by default, like micro, or most GUI editors.

I don't think an intro to GNU/Linux really also needs to be an intro to a text editor at all. Just teach people the man command, tell them a few of the editors that exist, and maybe mention tools like vimtutor.

To be sorta serious about editors for a min. I think console editors should definitely be in there because a lot of Linux access takes place over ssh without a GUI.

And Vi/Vim is present everywhere, so it is a must for sysadmins.

Sysadmins can install whatever editor(s) they prefer!

Thee ability to modify the root filesystem should never exist on a production system.

I've been using and administering linux machines for decades and I can't remember ever using emacs. I still only know the real basics of vi - how to insert/delete and most importantly how to save and quit or quit without saving. Anything requiring more manipulation (even copy/pasting), I switch to something like Geany instead.

The point of learning vi is that it's nearly always available, even on a broken system and is often a key part of fixing broken systems, so learning the basics of it is more or less essential. Not even just Linux - I've had to bring up vi on an ancient HP-UX system once to get it to boot.

The post mentions using crontab for scheduling tasks. Recently, I needed to use systemd user account job scheduling. I found it pretty nice once I got the hang of it.

What's the best way to run software on an old Ubuntu system that I don't want to upgrade or run apt-get on, and I don't want to run stuff inside a Virtual Machine, and snap is broken, and docker as well? Compiling stuff seems too much trouble because of all the dependencies. I was hoping for Nix to save me here, but it only compiles for a specific libc version.

It sounds like your asking how to install software without using precompiled artifacts and also without compiling stuff.

Well, I want to run new versions of software on a system in which I basically don't want to touch system directories like /usr and /lib etc.

You can download precompiled binaries of a lot of things from git these days. You can modify your PATH to include a location of your choosing and put the binaries there. It probably won't work well though because you'll be missing dependencies.

You could also chase down debs, download those and install them manually and specify your install path. (I don't have the deb switches memorized, so many pages are your friend.) You'll have dependency problems here too but you'll get friendly warnings that can help you hunt down other missing packages and you can work your way up from there. It won't be fun.

I'm glad I started in Slackware and used it for about 15 years as my daily driver. I'm comfortable with Linux The Hard Way and when people tell me how hard it is to run Linux because all the weird kinks, I just chuckle.

Hey, I started with Slackware too, and like you said, some of those installation methods won't be fun. I'm looking for something that Just Works^TM.

Flatpack and appimage are similar to snap but less obnoxious I believe.

> I basically don't want to touch system directories like /usr and /lib etc.

How about touching (creating) a /nix? That'll sit next to the rest of the system just fine, although I don't know how new of a kernel it needs.

Alternatively, there's always the old standby: Can you create a chroot of a newer system and install stuff that way?

Figure out why docker and snap are broken?

---

You could use https://wiki.debian.org/nspawn unless it's really old.

On a technicality I will say you can use podman or distrobox/toolbox assuming you can install them.

More seriously your best bet is to compile things on other systems and move them over.

What's wrong with Nix in this situation? A Nix package's closure will come with its own new-enough glibc, right?

Yes, but an external glibc puts extra constraints on the nvidia driver versions I can use. The driver I have now works fine.

I see how that's a problem.

I know that GPU stuff is an area that requires some impurity on foreign Linux with Nix. I didn't realize that had implications for libc versions. :(

It wouldn't be a problem if it was easy to recompile the entire Nix system with a different glibc version.

To change glibc do you have to override the stdenv? Why is that hard? Do you have to make a new bootstrap or something awful like that?

sudo -u fred -i # to become fred

sudo -u $user -c '$commandtoexechere" to just run something as the user and drop you back to your own shell.

ip a

ss -tnaopu

grep -C 5

grep -E ‘dick|ass’

nano -c foo

grep -A 5

ls -lrth

You can achieve much more learning in 20 days.

Important fundamentals are omitted but some oddly specific subjects are cherry-picked and given an unusual amount of attention like running nmap. Running nmap (a portscanner) can get you fired from your job and there's precedent of this.

The text formatting is inconsistent. Reader mode makes it better, but a simple markdown file would have clearer.

Explaining what a service is before explaining what a task is, is weird.

APT is mentioned, but APT is specific to Debian-based distros. Debian is not mentioned a single time in the article. Other distros use other package management systems.

You should use nftables instead of iptables. iptables is deprecated.

This article just reinforces that "learning Linux" is simply learning some few random commands.

What alternatives do you suggest?

Learn about the file system, permissions, processes, signals, threads, the scheduler, standard input and output, networking, etc.

Yeah, I just haven't seen a single book/resource that does this - maybe I'm not looking hard enough. "the linux command line" from nostarch press was good but, of course, focused on commands

Take a look at "The Linux Programming Interface". It is a thick book that you can use as a reference when needed.

This type of bizarrely hostile criticism would be so much more helpful as a more thorough list of areas to improve AND detailed suggestions on improvement.

Anyone can take shots, but a PR takes guts.

Like you come in hot, then proceeded to not tell me how to learn more in 20 days, you tease.

Just like we are all lectured by sudo at one point, neophytes should be lectured on not using nmap, especially on production Corporate IT, unless they are part of a red team activity.

These are all valid points, the apt and nftables have bitten me as an older timer. While I think one should handle nmap, tcpdump with care I have never seen an organization that fire someone based on it.

I accidentally overwhelmed a brittle network stack doing nmap scans that took down internet overnight for a major resort. It happens.

I may or may not work for a large credit bureau in the US, and I had a legitimate need to run nmap from my laptop to scan a server. 30 minutes later, I get pinged by security asking about it, which I gave an explanation, which they followed up with asking if I sought permission to use the tool (no, why would I?). They made me remove nmap, because that’s what hackers use. A month later, I get an email from someone else letting me know the situation was resolved. The point being, I ran one scan which involved no less than 7 people and wasted a month of time.

If running nmap gets you fired from a job, then that wasn't a job worth keeping. Sheez.

Fired is a bit much but I could see port scanning a prod server or like putting an interface in promiscuous mode generating a call from infosec asking for an explanation. Oh and the reply “I just wanted to see what happened” would not be a good one heh.

If Little Bobby Tables is playing with his new nmap tool, from work and scanning High Security Org, LLCs SaaS range and they see the origin comes from an IP registered to New Corp, they may call New Corps security team. If the security team can identify Bobby as the source, they might fire him for annoying High Security Org because with the initial contact came a Scary Letter with Legal Words.

Sorry Bobby, we can't afford a fight.

It does happen. I could (but won't) tell about similar things happening at FAANGs.

They would rather ask you why were you portscanning and what does have to do with you job, and if they have reasons to not believe you or there is a process in place to deal with that, there might be consequences, including termination.

Curmudgeon alert

Name-calling isn't productive.

Reply on news.ycombinator.com