Hacker News Reader

Comment by simmons | Hacker News Reader

Yes, I remember feeling pride in the stability of my systems when I saw a large uptime. I had a server that had 1000 days of uptime, once. Now when I see a large uptime, I'm terrified of what security patches the kernel may be missing!

View on HN

I still remember the days of servers as pets, rather than cattle, and I was harping about server uptime. A wizened server admin piped in and said he rebooted his servers once a week. Said, if you do it any less frequently, then the odds of catching an error causing change while the person who made said change (possibly himself) is still around and can remember what they did go down precipitously. So, to avoid headaches and potential downtime when it mattered, he would just take servers out of rotation and reboot them, and make sure they came back online.

So true. We have one older, rather large machine in a data center that's been up for.. (checks uptime): 963 days. It has IPMI but at some point something stopped working and now we have to physically go to the data center to restart it. And since we use it every day we can't really afford to lose access to it.

This is kind of like making sure your backups actually work.

You need to test when servers go down, and people who use them should know and understand what happens when the are off.

Live Kernel Patching has been around for about 20 years[-1] now.

Red Hat Enterprise Linux[1] and Oracle (Enterprise Linux) Unbreakable Linux[2] both use it as a selling point.

This feature is still a bit ad hoc because, in most setups, rebooting a system isn't a huge burden and is much simpler than using boutique commands to live-patch it.

[-1] https://en.wikipedia.org/wiki/Ksplice

[0] https://www.ksplice.com/

[1] https://www.redhat.com/en/topics/linux/what-is-linux-kernel-...

[2] https://docs.oracle.com/en/learn/ol-ksplice/

Live patching exists but it tends to cap out at around a year of updates for any particular kernel version. It's not getting you anywhere near 1000 days.

    8:59PM  up 1858 days, 22:51, 1 user, load averages: 1.69, 2.21, 1.60
    dblrabbit@cookie:~ $ uname -a
    FreeBSD cookie.server 12.2-BETA1 FreeBSD 12.2-BETA1 r365618 GENERIC amd64

    9:05PM  up 1859 days, 13 mins, 1 user, load averages: 1.19, 1.32, 1.39
    dblrabbit@mookie:~ $ uname -a
    FreeBSD mookie.server 12.2-BETA1 FreeBSD 12.2-BETA1 r365618 GENERIC amd64

    9:14PM  up 245 days,  8:46, 1 user, load averages: 1.26, 0.97, 0.91
    dblrabbit@dragoness:~ $ uname -a
    FreeBSD dragoness 16.0-CURRENT

Currently serving: vm's, dns, email, mx-relay, and multiple shoutcast radio relays 24/7 and some other miscellaneous stuff. Colocation is fun, do I win?

5years; I'm 37 now, I was 32. Life seemed easier then.

I dunno how much easier life was back in May 2021... a year into the pandemonium, heh.

Easier for some, harder for others. The economy was more unfucked because the government printed tons of free money (mostly to business owners).

I had a box set up as NAT (running amazon linux) when we moved from a local datacenter to AWS in 2012. Shut it down last year. It had not been rebooted. Should have grabbed a screen capture of the uptime. Part of me wanted to leave it to reach 5000 days....

I worked in a place with a lot of Solaris servers with years-long uptime. It would be my job to patch them. Having no idea what config changes that may have happened over the last 3 years which would take effect on boot was always terrifying.

Ksplice came out of MIT in 2008, which updates your kernel while it's running. No need to reboot! Supports Ubuntu.

Thankfully there's livepatching (e.g. https://ubuntu.com/security/livepatch )

6.19 added a new Live Update Orchestrator, which allows significantly more of the system to be retained while doing a kexec / Kernel Handover like transisiton to a new kernel too. https://www.phoronix.com/news/Linux-6.19-Live-Update-LUO https://lwn.net/Articles/1033364/

Systemd added support in recent 2.61. Theres also now ways to have user stores, that survive across switches. https://www.phoronix.com/news/systemd-261

There's also `systemctl soft-reboot` which initiates a userspace-only reboot, which quickly restarts the system without going through the full hardware and kernel initialization process.

But that doesn't give you kernel updates, being userspace only?

Real men use: killall -9

I'm glad to see this. Almost 18 years ago I implemented a similar kexec device+memory preservation for a storage vendor. It was done on a Linux kernel of that day, and it had had a memory reservation and handoff protocol between the two kernels to keep some specific PCI device alive, allowing for state restoration at the application side. I'm proud of the fact that the kernel replacement was just under 1 second in execution (after init process optimization) and the whole kernel+app was less than 10 seconds.

Was hoping this would be mentioned. This version of functionality will probably be more universally accessible to all configurations as compared to the soft reboot released in 254:

https://github.com/systemd/systemd/releases/tag/v254

Soft rebooting was even more exciting for me, but I wasn't able to get it to work on my bluebuild based system (customized universal blue). I haven't tried it in a while though.

I remember installing some new computers for a small shop around y2k, and their NT4 server was acting up a bit when I was adding the new users to the domain controller.

I opened Task Manager to see if any processes was running wild. Imagine my surprise when I saw it had well over 1100 days of uptime!

Reply on news.ycombinator.com