Hacker News Reader

Comment by stsp | Hacker News Reader

+1 -- This is the one and only problem I have to regularly help my non-technical Ubuntu friends (and their friends) with. Every few months they cannot install updates anymore because their /boot fills up and apt fails to install a new kernel package.

The simplest fix would probably be to make /boot large enough by default (in the order of 10GB or 20GB or so -- the current size is 512MB IIRC).

A better fix would be to purge old unused kernels automatically but as far as I understand there were some difficult edge cases around that.

View on HN

> The simplest fix would probably be to make /boot large enough by default (in the order of 10GB or 20GB or so -- the current size is 512MB IIRC).

Sure, I'll just use 1/6th of SSD to store 60 megabytes.

  $ du -hs /boot/
  56M	/boot/

If 512M is not enough space for /boot you're doing something wrong.

If you're running updates weekly it will fill up on Ubuntu. This is a recent problem, and I've only experienced it on my laptop with full-disk encryption.

The update process generates on the order of 100mb/month.

It's not new. It's been happening to me since I started using Ubuntu in the 8.x range.

>If 512M is not enough space for /boot you're doing something wrong.

I don't know what planet you're living on but it's certainly not this one. Between a Ubuntu desktop, a laptop and personal server with multiple Ubuntu VM's on it, all of which are kept up rigorously to date, I fix this problem at least three times a year, every year.

The command line process to fix it[1] is a multi-stage mess of dense bash-foo that comes with a 140 word, two paragraph explanation so that /ubuntu veterans/ can figure out what is going on without resorting to scouring the man page for flags. The friendly GUI process to fix it relies on a third party tool that is no longer maintained[2].

It is not possible to explain to non-technical users what is happening here, which means the only thing they can do when they see this is call their technical friend and cry for help. This is exactly the kind of user experience that makes people think Linux is not ready for widespread desktop use.

This is definitely something the OS should take care of itself. I'm ignorant of the challenges that caused it to be this way in the first place, but in my ignorance I would advocate that:

a) the partition be made larger by default b) the OS auto-purge any kernel package more than three revisions old

[1] https://askubuntu.com/questions/89710/how-do-i-free-up-more-... [2] https://launchpad.net/ubuntu-tweak/

Um, isn't the fix `sudo apt auto-remove --purge`, which autodetects unused kernels? What am I missing?

https://news.ycombinator.com/item?id=14005528

Following the chain of links and answers and explanations, we come to the conf file that says in the comments that it commonly results in two (2) kernels being saved, but can sometimes results in three (3) being saved.

IOW, it does automatically remove old kernels, it just keeps the last 2-3.

So, yes, run "apt-get autoremove", that's it.

If you do not run that command before /boot fills up, and you have a full /boot with a partially installed kernel, then that command fails. So this works fine if you remember to call it regularly, but it does not solve the problem once it occurs.

Interesting. I haven't encountered that edge case. I've many times filled /boot and resolved by doing an auto remove.

It seems silly to me that I need to manage this myself. Why do I need to be worrying about different kernel versions? I just want to make websites.

I think it has solved the problem for me, but still is not a good solution for anyone who would answer "What's a terminal?"

I love having a terminal with bash and use it constantly, but I don't think it should be needed for the system to just go on working.

Here is my old-timey one-liner personal solution[0] for it that has worked flawlessly so far, obscure theoretical edge-cases be damned, because the non-edge case situation is just awfully worse and practically impactful.

(warning, rant inside)

[0] https://gist.github.com/lloeki/520acee8ba3b44c532c7

I've been using Ubuntu either part or full time since 2007. I've literally never encountered this.

Which is not to say you're lying, I'm just sort of flabbergasted that this is an issue for so many people. Do you run autoremove much? Maybe that would solve it for you?

I run ubuntu 16.04 on a laptop, desktop and a TV streamer and I get this all the time. My boot partition on the desktop is 15gig and it gets plugged every now and then.

I've hit this before, but honestly do not think it's a big deal. Sure the installer could default to a larger boot, but it's manually configurable during install. And cleaning it up once in a while is just good sys admin practice.

sudo bash -c "apt auto-remove --purge; apt update; apt upgrade" is what I usually run.

Prefer they focus engineering cycles on actual engineering problems.

Yup! That "something wrong" is installing every single kernel update for two, three, four years and not deleting any of the old kernels.

Super common in enterprise deployments. I ran into this a bunch on my $EMPLOYER-issued workstation.

The installer should handle this. When you apt-get upgrade anything besides the kernel, does it leave the old version lying around?

I understand that it may be wise to keep the old kernel around so the system can be booted in case there is a hardware incompatibility or breakage in the new release, but that justifies only one additional kernel. Ubuntu keeps those kernels sitting there until you `apt-get autoremove`, and that means that unless you're running that command routinely, the boot partition is going to fill up at some point, no matter how big you make it.

This is especially a problem for people who use the unattended-upgrades package. I've autoremoved and had it clean up almost a gig of old kernel images before.

Sorry, but Ubuntu is doing something wrong here, not me. This should be handled automatically. Ubuntu wants to be the system for everybody, but you can't expect people to open the terminal and fix this manually. Making boot 20GB is ridiculous, but 1GB should be no problem, and for me 2GB would be OK if that means that this problem will disappear forever.

And I believe my boot partition is only 256MB, and I didn't set it to that. That was a system default.

Ubuntu is absolutely doing something wrong here and we'll get that fixed. Thanks!

Doesn't `apt-get autoremove` remove those old kernels? Not that it's a solution; it should of course be done automatically! Here's what I get when using it:

    > apt-get autoremove
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    The following packages will be REMOVED:
      linux-headers-3.19.0-79 linux-headers-3.19.0-79-generic
      linux-image-3.19.0-78-generic linux-image-3.19.0-79-generic
      linux-image-extra-3.19.0-78-generic linux-image-extra-3.19.0-79-generic
    0 upgraded, 0 newly installed, 24 to remove and 39 not upgraded.
    After this operation, 1,732 MB disk space will be freed.
    Do you want to continue? [Y/n]

Whenever a kernel is updated autoremove should be called immediately afterwards. It should be called before the restart now / restart later dialog box of update-notifier appears.

Currently, Ubuntu installs a new kernel and update-notifier tells the user a reboot is needed. The autoremove notification only appears when using the terminal which explains why users are running into this issue. Also, update-notifier informs the user another reboot is needed after autoremove is run.

To avoid this mess I’ve commented out the lines of /etc/apt/apt.conf.d/99update-notifier and wrote my own updater using bash and zenity and incorporated needsrestart. It’s not pretty but it works.

Care to share it? Maybe it could help others...

I thought about sharing it but like I said it’s not pretty. It involves editing sudoers and holding back config updates for sudoers and update-notifier-common which might cause problems in the future if you’re not aware. I’d much rather see Canonical address it properly.

absolutely not, automatically running auto remove may lead to bad things; on occasion autoremove flags other more useful packages for removal.

For example I'm using LVM with my installation on my Ubuntu laptop and after updating the kernel and running "apt autoremove" it removed the LVM package leaving me scratching my head shortly on reboot as to why it wouldn't find my root filesystem (frankly i have no idea how it became "unneeded").

A more sensible approach is how Red Hat do it with YUM/DNF, that is, to allow a certain number of the same packages to be installed, "installonly_limit" in yum.conf. Doing this means that when a new kernel gets installed the oldest is removed to keep the the system at the limit specified.

On my RHEL/CentOS machines I tend to narrowly provision /boot to around 250-500MB. set "installonly_limit" to 2 and the system will keep the most recent kernel and one back. it works for me.

I see you’re point, though I too use LVM and haven’t seen that happen... weird. I could have been more exact with my response as autoremove does more than just remove old kernels. Anyway, it would be nice to see Canonical resolve this.

Except, it would be nice to keep a few of the (recent) older kernels, in case things go awry with the new update.

I agree.

I'm running Ubuntu 16.10 currently. A kernel upgrade hosed my setup yesterday, and having an older kernel available saved my butt. I was able to do another `apt-get update` and things eventually worked with the latest kernel.

This already happens: apt autoremove won't remove the package for the running kernel. It'll clean up "old" (N-1 and lower) kernels, but installing kernel N+1 won't allow kernel N to be deleted as long as kernel N is still executing.

Once you reboot/kexec into the N+1 kernel, it'll let you remove the N (now N-1) kernel, bringing you down to one. But at that point you've proven the new kernel works—at least well enough to get to a shell you can run apt autoremove from.

This is why autoremove isn't so auto: if it happened automatically after reboot, it might be running on a now-wedged system (e.g. one that can't bring up the display manager), removing the last-known-good kernel and leaving you with only the broken one.

I think the right middle-ground solution would just be for installing kernel updates to touch a file, and for Desktop Environments to notice that file and trigger a dialog prompt of "you've just rebooted into a new kernel. Everything good?"—where answering "yes" runs apt autoremove. On a wedged system, you can't answer the prompt, so the system won't drop the old kernel. (In other words, just copy the "your display settings were changed. Can you read this?" prompt. It's a great design!)

Wouldn't a good solution then be to run autoremove before installing a new kernel?

That way, you have kernel N running, first autoremove wipes kernels N-1 and older, then it installs kernel N+1, so that when you reboot into N+1, you'll always have known-good kernel N if it doesn't work.

It's a very similar solution to how a good programmer solves an off-by-one error, doing a shift/rotate shuffle on a for/while loop.

What happens when you have a high-uptime system where you repeatedly "apt dist-upgrade" and end up installing packages for kernels N+1, N+2, N+3, etc., all without rebooting into any of them?

I agree that if the user manually runs an apt [dist-]upgrade—or really any manual apt command—that that's a good time to do apt maintenance work. (Homebrew does maintenance work whenever you invoke it and there haven't been any complaints so far.) But kernels usually get installed automatically, so it can't just run then.

Now, if there was a specific concept of a "last-known good kernel" (imagine, say, the grub package generating+installing a virtual package when you run grub-install, that depends on whatever kernel you specified as your recovery kernel, ensuring it remains around), then your approach could work—you'd always have two kernels, the LKG for a recovery boot, and the newest for a regular boot.

Exactly what happens on Fedora.

Fedora/RHEL yum has a much better solution: installonly_limit, defaulting to 3. Kernels which have been updated will only be kept up to this depth. The excess are automatically trimmed during update.

>Doesn't `apt-get autoremove` remove those old kernels?

Of course it doesn't! Why would you assume such a silly thing? /s https://askubuntu.com/questions/563483/why-doesnt-apt-get-au...

> It's better to err on the side of saving too many kernels than saving too few

But Muh Freedoms! I hate to be subject to one man's opinion of things /s

It's also kind of a garbage argument.

People who know they have broken kernels don't keep upgrading them, they stop and fix them.

People who don't know they have broken kernels also don't know they can boot with an older kernel, so they get nothing from the "backup".

We want to leave some time for people to realize their kernel is broken, so keeping three is probably just fine. Honestly, it would probably be adequate to just bump the oldest one off the queue whenever a newer one is requested. If you've got a tiny boot partition, maybe that means only two revisions. If you've got a huge boot partition it could be 20.

But just keeping them all and making people manually uninstall them gains you nothing, it's user-hostile for no reason.

Yes, it does remove old kernels. Read the very link you posted.

I confirm that it doesn't autoremove. I had to empty /boot on some servers lately.

Anyway sometimes one wants to keep old kernels. I have an old laptop that runs OK with a 3.something kernel and has wierd video sync problems with any newer ones. Ubuntu 16.04 keeps running with that old kernel so I keep booting from that, maybe once or twice per year.

However the proper solution would be pinning a package and autoremoving the others.

For Ubuntu Desktop, it may make sense for the package manager to keep only the latest 2 or 3 kernels, and automatically purge the rest.

I had the /boot filling up problem but had thought it was fixed, I'm on 16.04+. I'm pretty sure the last two kernel updates I did removed older kernels leaving me with the current one and previous one ... ?

Nope, still doesn't do it without manually invoking autoremove.

You can configure apt unattended upgrades to autoremove by default, perhaps you did that?

This is the main problem that keeps me from wanting to set up less technical family members on Ubuntu. It's possible to get in a spot where even a simple command won't solve this.

Solus uses https://github.com/ikeydoherty/clr-boot-manager now, which purges old kernels and modules, but keeps the modules for the currently running system so HW still works

> The simplest fix would probably be to make /boot large enough by default (in the order of 10GB or 20GB or so -- the current size is 512MB IIRC).

What? This is ridiculous and unacceptable. I don't use Ubuntu anymore, can someone tell me what is filling up the boot partition?

I'm currently on ArchLinux and mine is 200MB and it's 14% full! I can't fathom what could occupy so much space.

It's the way kernel update come in apt. The kernel update is a new package, not an upgrade of a previous kernel package. Thus the old kernels are left in place and the new ones installed alongside. After about 3 kernels have been made available in /boot the previously recommended size for /boot is full and attempted update to a new kernel fails.

It can be manually fixed by removing older kernels ("sudo apt purge ...").

Perhaps I'm mistaken but i thought a fix was in place for this, maybe it was something third-party but apt definitely offered to remove unused kernel package for me recently.

Reply on news.ycombinator.com