Ubuntu 16.04: “Out of memory” errors after upgrade to 4.4.0-59 | Hacker News Reader

Ubuntu 16.04: “Out of memory” errors after upgrade to 4.4.0-59

bugs.launchpad.net

120 pointsivank169 comments

It's not just a bunch of people setting cm.swappieness to 0 is it? That behaviour changed and now prevents going into swap /ever/ where it used to mean go into swap as a last resort. Changing it to 1 fixes the issue and will act like the old behaviour or only swapping as a last result.

Regardless, it's things like this why we never recommend people use Ubuntu - years of poorly packaged and tested software. If you want a Debian based system use Debian (and accept that there's now a lot of missing packages are barely any proper SELinux or clustering support) or use RHEL/CentOS for a more complete district and add in repos like epel and elrepo as required.

Yeah, Ubuntu still keeps running out of space in /boot by filling it w old kernels in the default partition profile. A bug that I've been hitting since about 2011.

In spite of their best efforts, it is not the "grandma friendly" quality software it hoped to be.

If we're sharing Ubuntu bugs, how about the one where NFS with Kerberos had been completely broken since several years back. You simply can't mount such a share, and you'll always get an authentication error.

I reported it as soon as I upgraded (to 14-something I think) and the bug was marked as severe. Every time a new release of Ubuntu comes out I check if it had been fixed, and of course it hasn't.

FWIW, we've been able to use NFS with sec=krb5 on both 14.04 and 16.04.

My M.O. is:

  # apt-get update
  # aptitude full-upgrade
  # apt-get autoremove

Because as far as I can tell, apt-get sometimes fails to actually update things, whereas aptitude can't autoremove. I'm sure there's a better way.

This is my major gripe with the dpkg ecosystem. There are (AFAICT) multiple different tools to build dpgks and multiple different tools for updating your system, and none of them are adequately documented nor support all use cases.

The Fedora ecosystem is better in this regard. Want to build an rpm? rpmbuild is the only answer. Want to update the system? Use dnf. (Sadly, there's now also PackageKit, and it's not quite isomorphic to dnf.)

AFAIK, the only supported way to build a deb is using dpkg-buildpackage, and all the others just wrap it (unless we're talking about alien, but that's not really "supported" going any direction).

Updating your system is all the same backend, and the Debian-alikes try to tell you to only use one tool for cmdline (apt, now, which just takes the same args as all the various apt-FOO commands did, all of which still work), and one for the GUI (Synaptic, or wrappers around it, unless Ubuntu Software Center has mutated a lot since I last looked).

IIRC they made a big deal about not installing aptitude (a popular alternate text UI for apt-based systems) by default on Ubuntu _because_ they wanted to standardize and fix any deficits, rather than working around them.

> they wanted to standardize and fix any deficits, rather than working around them.

That would work great if would they would take bugs reports seriously. In practice, with Canonical I file a bug and ninety days later get an automated message that the bug was closed due to inactivity.

That's an unfortunate pendulum swing cleaning up from their problem of yester-year - namely, bugs sitting around _forever_ without any comment.

I noticed they changed that policy when I started getting comments on years-old bugs of closed-obsolete. If they're now 90d expiring bugs, I'd _guess_ there's a bit that can be set for your account of whether you're a customer paying for support, and that their support staff have a queue filter that prioritizes those, and the rest get expired if nobody comments.

> apt-get autoremove

You should be doing:

    apt-get autoremove --purge

Otherwise you'll be leaving a lot of clutter around (obsolete configuration files).

apt-get update won't do anything beyond download the latest view of the repository.

you want:

apt-get update && apt-get dist-upgrade && apt-get autoremove

Aptitudes full-upgrade is synonymous with dist-upgrade.

Others wrote "apt-get dist-upgrade" in sibling comments, which should do the trick. If you want to build packages for multiple managers, fpm is a nice front-end for some of them: https://github.com/jordansissel/fpm

I use FPM all the time, but it is not a 'nice' way to package things, it's an 'easy' way to package things ;)

> apt-get autoremove

This is not going to autoremove kernels unless you modify the default preferences in /etc/apt/. For added benefit you should probably add --purge to your autoremove command.

apt-get dist-upgrade is what you nerd to use to upgrade all packages... Don't blame apt-get for your lack of knowledge on how it works

I lost my encrypted backups thanks to the Ubuntu installer. It turns out that when you set up an encrypted volume, the installer overwrites the LUKS metadata block on all connected volumes. I happened to have my encrypted backup volume connected.

As great as Ubuntu is, I've always thought that a project of their magnitude and their resources (which I grant is not a lot) should focus on QA, specifically with these sorts of issues.

Yep, I had to fix a 14.04 server like that last week. /boot filled with old kernels and apt was falling. I had to manually fix that.

I also don't have swap on my laptop (16 GB RAM and SSD) and I rarely hit 10 GB of used space. With no swap I mean no swap partition. I'm getting those out of memory errors in Opera tabs since the new kernel. Luckily my primary browser (FF) is fine. I guess it's trying to swap Opera out even if there is no swap. I'll reboot with the older kernel and see what happens.

Likely, opera asks for a disk-based anonymous memory mapping - this goes on the swapfile if you have one, and fails if you don't.

The /boot bug is unreal, I just ran into that a few months ago.

It's super frustrating. There's no reason I need twenty kernels on my machine :-p

Ubuntu actually only keeps the current and the previous kernel as necessary and give the user the option to purge all the old kernels. These old kernels will be marked as autoremovable and by setting a flag in unattended upgrades config you can make it autoremove those. On desktops this happens automatically.

On RHEL, it happens on servers, too - anything is better than your /boot partition filling up on its own!

It shouldn't, the default is to keep 6 recent kernel packages.

Honestly, the amount of regressions, packaging problems, hacky fixes, non-stardard configuration and poorly set defaults in Ubuntu is disgusting, look at how bad NetworkManager was / is on it, it's not that bad on Fedora (not that I like it that much) and the whole debarcle with them doing Unitiy on top of Gnome rather than contributing to the community projects, or recently when they released an update that broke all USB 2 ports and if you got that update before they pulled ita and rebooted - you had no keyboard or mouse to work with, their own custom (IMO very gross) init system of Upstart when they could have contributed to System which they're now going to move to anyway, their dodgy partnerships with companies like Amazon where they're putting apps / links on the desktop menus to Amazon (who does very little for the open source community also), it's just a joke and I really wish people would just move on to better distros or at least use pure Debian if they can handle the lack of packages. So often at work when devs complain their DNS isn't working or they can't connect to the VPN or they updated and now their machine doesn't boot - our first questions - 'are you running Ubuntu?' it's like the Windows of the linux world I swear.

I think that Gnome3 has a more detrimental effect on community projects than Unity. Why? Because of the client side controls they are pushing, rendering Gnome apps for everything non-Gnome3 either useless or ugly. And all of that for a few pixels of screen space, something that has been solved more elegantly by putting menus in the top panel.

And talking about breakage: I can't count how many times some important Gnome3 plugin broke after a minor Gnome3 update from the distribution. Or how many times I wasn't able to install a plugin that installed fine just a month ago on the same Gnome3 version. Why? Because you have to download them from some obnoxious website that apparently has no archive and no way to properly match the right plugin version to a given Gnome3 version.

Cinnamon copied the same madness. But you have to install less addons to make it at least usable. KDE is somehow still stuck in the previous decade when it comes to desktop design.

If the Gnome3 project with their questionable design choices and holier-than-thou attitude is the future of desktop Linux, then I have to say that the situation is much worse than it was ten years ago.

BTW, Ubuntu doesn't come with Amazon addons installed anymore. And even when it did, you could entirely remove it and look at the source. I think Canonical realised that they cannot pull something like that anymore.

> rendering Gnome apps for everything non-Gnome3 either useless or ugly.

I use i3 instead of Gnome 3 and the Gnome 3 apps with CSD fit in beautifully. There used to be issues where there were gaps and such, but those are resolved by now.

> something that has been solved more elegantly by putting menus in the top panel

Arguable and probably comes down to personal preferences. For some apps (like the screenshot tool) it fits really well.

andrewshadura is right. Here's just one of your items that is bunkum, for an example.

> their own custom [...] init system of Upstart when they could have contributed to System

upstart was developed several years, just under half a decade, before systemd even existed; it was used by Fedora for three years.

It's true though that Canonical consistently "does their own thing" and then fails to establish it as a standard for whatever reason (technical, political, CLA...). Anyone who adopted the new thing then has to migrate.

Eucalyptus vs OpenStack (ever heard of Eucalyptus?)

Bazaar vs Git (Git won)

LaunchPad vs Github (LaunchPad is alive and well, but Github won the popularity contest)

Wayland vs Mir (Wayland won)

upstart vs systemd (systemd won)

Let's see how these work out:

Juju vs Kubernetes/OpenShift (I'm aware that you can run Kubernetes on Juju, probably won't save them though)

Snap vs Flatpak (vs AppImage?)

It's sad, really. I like Ubuntu and Canonical and strongly believe in competition, but it'd really help if they didn't consistently back the wrong projects.

In a lot of those cases (Mir being the obvious exception), the Ubuntu backed solution predated the alternative.

I can't help wonder if this will keep happening as their competition waits to see everything new Canonical chooses to back, then works to thwart it :)

Yeah they invent a new wheel (often poorly) then they fail to test and fix it when it breaks.

I guess they need to buy more project managers...

And if Canonical hadn't required contributors to sign a CLA... https://news.ycombinator.com/item?id=7640930

a short point of order: Upstart existed and shipped long before Systemd existed, Infact Red Hat Enterprise Linux 6 shipped with Upstart. It was the shortcomings in Upstart that led to the development of Systemd. If you're using RHEL 6 you're using Upstart.

So much written, so little sense in that.

Have you actually looked at Ubuntu, have you tried to use it? I'm not even going to go through your assertions one by one to debunk them, as most of them are wrong and unsupported by facts.

Cheers.

he got some details wrong but the frustration is real, especially if you have anything slightly off standard, like a multi monitor setup or, god forbids, a touch screen.

I have a touchscreen. Works great nowadays.

even pressure?

That I wouldn't know. It's not capacitive.

Been using Ubuntu for years in multiple production environments and I agree with him.

Much fewer issues with Red Hat.

My mother, who is literally a grandmother, ran into the /boot old kernel's issue twice already. Very annoying.

I used to use this to clean boot. dpkg -l 'linux-' | sed '/^ii/!d;/'"$(uname -r | sed "s/$.$-$[^0-9]\+$/\1/")"'/d;s/^[^ ]* [^ ]* $[^ ]$./\1/;/[0-9]/!d' | xargs -p sudo apt-get -y purge

What does it do?

I tried it on Lubuntu 16.04; it required modification:

   # mystery one-liner from https://news.ycombinator.com/item?id=13513171
   # dpkg -l 'linux-' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]\)./\1/;/[0-9]/!d' | xargs -p sudo apt-get -y purge
   # my corrected/improved version:
   noncurrent_kernel_pkgs() { dpkg -l 'linux-*[0-9]*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.\)-\([^0-9]\+\)/\1/")"'/d;s/^ii *\([^ ][^ ]*\)[^ ]*.*/\1/' ; }
   # corrections/(possible-)improvements:
   # - dpkg -l 'linux-' # returns nothing
   # - dpkg -l 'linux-*[0-9]*' # moves 'pkgnm must contain a number' rule (last sed cmd in orig: '/[0-9]/!d') forward
   # - pkgnm isolation/extraction was broken (did dpkg output change?)

output:

  kg@kg-VirtualBox:~$ noncurrent_kernel_pkgs
  linux-headers-4.4.0-43
  linux-headers-4.4.0-43-generic
  linux-headers-4.4.0-45
  linux-headers-4.4.0-45-generic
  linux-headers-4.4.0-53
  linux-headers-4.4.0-53-generic
  linux-headers-4.4.0-57
  linux-headers-4.4.0-57-generic
  linux-image-4.4.0-43-generic
  linux-image-4.4.0-45-generic
  linux-image-4.4.0-53-generic
  linux-image-4.4.0-57-generic
  linux-image-extra-4.4.0-43-generic
  linux-image-extra-4.4.0-45-generic
  linux-image-extra-4.4.0-53-generic
  linux-image-extra-4.4.0-57-generic
  kg@kg-VirtualBox:~$ uname -r
  4.4.0-59-generic
  kg@kg-VirtualBox:~$

Sorry about that, the '*' got removed from a non marked up paste. You are 100% correct. Just a little script to remove all but the current unused kernels that I use on 14.04. Yep, just take the xargs off to see what it will remove (which would always be good advice when pasting a random script LOL ).

Take off the xargs part of the command and run it to see. :-D

Just looking at it though, it looks like it'll purge kernel packages that are the currently booted one. I don't have a system handy to check though so take that with a grain of salt.

Just for posterity that should have been "are not the currently booted one".

I downvoted you for "grandma friendly." My grandma remembers when LaTeX was "new" and was one of the first people to professionally use computer-based typesetting. What exactly does "grandma friendly" look like to her?

I utterly agree that it's patronising to use a generic grandma as an example of a naïve user, and we should stop doing it.

However, I suggest that your grandma's long experience with computers probably means that she in would be especially displeased with the crummy state of the Linux desktop in 2017!

I just wanted to say I think it's sad you're getting downvoted. My grandma was also relatively computer literate. If we ever hope to reach equality in our line of work we need to think more carefully about the personas[0] we use, not because our grandmas care but because of how such descriptions affects the people who might one day fall into that user category.

[0] https://www.usability.gov/how-to-and-tools/methods/personas....

Fair enough.

I just use the term because I'm "family tech support" for multiple grandmas. I'll avoid that phrasing in the future.

Thank you! :)

You can run `apt-get autoremove` and it will purge old kernels/headers for you (amongst any other no-longer-used dependencies).

Oh, I know it's easy to fix, I just single that bug out as "I have done nothing with this box but install updates and surf the web. I am Grandma. It breaks".

If you're Grandma you don't have /boot on a separate partition, you have all / on a big 120GB at least, so if you run out of space you simply know you've run out of space.

Not perfect but not a problem for 90% of people I guess...

Ubuntu installer has separate /boot partition by default. If you don't do anything special during installation you end up with it.

No you don't. Ubuntu's default partition map is one big / and one efi(if necessary).

I literally just did a BIOS install and it has a separate /boot.

I assume it changed recently, I've seen fresh 2014 installs with tiny 255MB boot partitions.

This was ubuntu's recommended/default partition plan. I did not customize it. Now, it's possible this is a plan they no longer recommend and have changed since I first installed Ubuntu, but the point stands:

I installed Ubuntu onto a machine, chose the recommended default options, and it set up a /boot partition that was quite small, and its normal update cycle fills it up and fails to fix the problem automatically or prompt the user with information to fix the problem.

My old mother (born 1936) is using ubuntu since 2013. She can read her mail, answer or forward, view powertpoint, video and pictures, web surf, print stuf, write letters (its her most complex task). I use ssh and x11vnc for remote administration. It is the most simple setup I can imagine.

These are autoremoved nowadays.

> In spite of their best efforts, it is not the "grandma friendly" quality software it hoped to be.

Grandma didn't customise her installation to put /boot in its own, small partition.

FWIW, you can run "apt-get --purge autoremove" to remove old kernels since at least 16.04 and maybe 14.04 (I don't recall). Not fully automatic, but at least it is one command. For fully automatic, you can install and configure unattended-upgrades and configure it to autoremove automatically.

I don't remember when I first installed mine... 12.04 maybe? But this was the default/recommended partition plan.

IMHO if you can't not use swap without running out of memory in normal workloads, you either need more RAM or to work on reducing your memory consumption; swapping is for temporary overloads or gracefully recovering from one, not a substitute for RAM.

That's a surprising behaviour change (from the perspective of "don't break userspace"), but looks more like a "fix of previously broken behaviour" as in 0 not being able to disable swap entirely; from that perspective, using 1 makes sense.

I think that the new behavior at zero (documented anywhere?) is a discontinuity. There should be a separate setting for mlocking code pages.

Swap is useful even if you have plenty of RAM. It's better to swap out a cold file-backed page and use that page for cache instead than to pin that cold page in memory for no reason.

Gah. I meant non-file-backed

That's what ZRAM is for.

It's also what swap is for.

Funny, but Linux is the only unix-like OS I've ever run where using swap means you have a problem. Other OSes have much more intelligent memory management.

For me swap is an error. I would rather a server crash then go into swap and run slow.

If your servers are at 75% CPU and start swapping you are screwed. Time to start adding servers. In this day and age you are wasting a lot of money to be sitting around waiting for swap.

zswap makes swap useful on the server as it results in gradual slowdown when the system has used all the ram. And if the system runs at 75% of CPU, then there is room for such slowdown.

I hadn't heard of zswap [0]:

"Zswap is a lightweight compressed cache for swap pages. It takes pages that are in the process of being swapped out and attempts to compress them into a dynamically allocated RAM-based memory pool. [...] Zswap is a new feature as of v3.11 [...] zswap is a work in progress and should be considered experimental."

The return of RAM Doubler!

The naming is initially confusing if you're already familiar with ZFS, especially:

"Zswap makes use of zpool for the managing the compressed memory pool"

[0] https://www.kernel.org/doc/Documentation/vm/zswap.txt

It's actually quite useful on many consumer devices, especially those with multiple cores. Most of the time I'm happy I can trade a few CPU cycles for better I/O performance.

yes, we run ZRAM of every single server we have, its fantastic and we don't have swap anywhere. It's worth noting though that vm.swappiness is set to 0 then ZRAM wont be used, it must be set to 1 or higher unless you're using a really old kernel (pre 3.5).

I have found that zswap behaves much better than zram when the size of the used ram approaches 150% of RAM. In theory zram compresses better as zswap has very simple mapping for the location of its cache of compressed pages leading to space underutilization. But, as it has real backing store, it can evict unused pages allowing bigger chunk of uncompressed memory for running applications.

zram[0] is not the same thing as zswap[1].

0. https://en.wikipedia.org/wiki/Zram

1. https://en.wikipedia.org/wiki/Zswap

It's not the case on Linux either.

Linux intelligently swaps out ununsed pages even if there's enough physical RAM (that is more useful as page cache).

I have FreeBSD systems that use gigs of swap with gigs of memory free. If you have a Linux system with that much swap usage it's usually about to crash (as it struggling under memory pressure)

People should stop monitoring swap usage and start monitoring the amount of page in / page out IO.

Not true. I have plenty of Linux systems that use gigabytes of swap with 100+GB of free RAM.

Monitoring swap usage doesn't make sense on either FreeBSD or Linux.

heh, try using Windows when it goes into swap, Windows both desktop and server additions have the worst memory management I've ever seen and having no ZRAM and a pig of a filesystem like NTFS really doesn't help either.

Going into swap in Linux isn't a problem, it's a symtom of a misconfigured / provisioned server or an unhanled application memory leak, other than that the other only things that should cause it is incorrectly configured use of shared / reserved memory.

Disk based swap should pretty much always be disabled these days and ZRAM used to incresed efficiency.

> having no ZRAM

Windows 10 has had memory compression [1] for quite a while now. On the PC I am on at the moment with 100+ Chrome tabs open, I have 1.3GB compressed out of a total of 16GB.

[1] http://www.thewindowsclub.com/memory-compression-in-windows-...

Works very well on Windows Server 2012 and saves massive amounts of money with some workloads.

Incidentally, at least on Windows, you can save a ton of harddrive space by changing your swap settings - the default is crazy high, especially if you have lots of RAM.

My laptop has 8 gigabytes of ram. I can fit all of my personal files in that at obce, I don't see why that's not enough for everything.

Space used on disk is less than when the data is in RAM. It's not a binary copy loaded. It's binary data loaded then inflated into app-object memory. So much overhead

I’ve run with 4 GB of memory and no swap for a long time. It’s not hard.

Swap makes a lot of sense for a consumer device, where I'm opening a big spreadsheet, then putting that in background to edit this video.

It makes less sense for server workloads, where you usually don't have huge chunks of inactive memory.

I always run with no swap in AWS, especially with the newer EBS-only families.

EBS is a problem because in reality you would be swapping to a network volume. But on a standard VPS server, a bit of swap can really save you some problems. Swap was invented at a time when *NIX was strictly used in a multi-user environment, not far removed from today's webservers but very different from today's consumer devices. It still serves a purpose in that role.

> [...] a bit of swap can really save you some problems.

And cause a bunch more. Have you ever had to debug a race that is only triggered when a process hits swap. Not fun at all.

Yes I did, and the swappy machine started responding just a tad too slowly to other nodes' pings, engaging into a loop of join/parting the cluster, causing replication to kick in all the time and a whole "highly available" critical cluster of 9 nodes was unable to handle traffic (too busy replicating) causing millions in losses. So yeah, don't use swap in servers.

Ironically, if the node had crashed for OoM, the cluster would have survived just fine.

>"fix of previously broken behaviour" as in 0 not being able to disable swap entirely

But you can disable swap entirely by not having a swap partition/file, so I don't think that's a huge issue.

It was indeed a massive, not well discussed change and the kernel maintainers should have vocalised it better (not that I think the Ubuntu maintainers wouldn't understand it properly anyway TBQH) but yes, the behaviour was previously logically incorrect and had to be fixed at some point.

Exactly.

I have stopped using it long time ago, after they introduced the following bug:

https://bugs.launchpad.net/ubuntu/+source/netcat-openbsd/+bu...

Crikey - they did what?!

I recall a fascinating article / thread years ago about swappiness=0 not actually meaning what people thought it meant, and explaining why it was (generally) a bad idea. A low number, yes, but not 0.

Doing some research now, and found a blog written by someone bitten by this change back in 2014-04 -- referring to the commit that introduced this in 3.5-rc1. Not clear why the change was made, though, as a) it breaks lots of people's understanding / expectation, and b) if you want to never swap out then just forego the creation of a swap file / partition.

[1] https://www.percona.com/blog/2014/04/28/oom-relation-vm-swap...

Exactly, people in this thread missing the point of the use of RAM. A perfect operating system always going to use as much RAM is available for caching and there are long running processes that can be swapped out to free RAM for more important use cases. Memory management and paging/swapping is an interviewing topic for SREs/systems engineers, and this is not an accident. Oversimplifying the problem does not help.

It's surprising that many professionals have broken mental models about how an operating system supposed to work. I think they have something like DOS in mind. I was just told last week about how caching static (!) files from web server in redis increase performance in their infrastructure. I was baffled.

Yeah it wasn't well announced when the change was made, esentially if you find any linud machines running any kernel dating way back to 3.5 make sure you change vm.swappiness=0 to vm.swappiness=1

What are the many missing packages on Debian? From my experience Centos looks like a bad joke.

Most SELinux policies, Corosync and Pacemaker and related HA / clustering tools, and a pile of other things and they settled for a really awful stable kernel major in Jessie / 8.x.

It really went downhill with, the release of 8.0, the only improvement we really saw was a slightly newer kernel (but still old) and SystemD (although not consistantly implemented), The Debian community has also fallen through the floor, lots of smart-ass bickering and slack package maintainership and generally unhelpful, rude contributors / owners. By contrast RHEL/CentOS sped up their release cycles after re-in-housing Fedora as and taking a lot of it's rolling updates into stable a lot quicker and the community has been fantastic, Redhat have really stepped up and modernised their game a lot over the past two or there years.

I happen to run most of the latest of things on CentOS (server-side only, though). My approach has been to package and deliver modern versions of various runtimes (python, ruby, jre) into specific directories for my apps. This way, I am not dependent on the system-provided runtimes which are an older version.

For my work with Belenix (revamp not announced yet), I am working on examples where users no longer depend upon system-supplied package versions, and instead choose their own local instances.

Are you aware of the "software collections"? That may or may not be of use to you.

Why don't you use Software Collections to get the newer versions of the runtimes?

Though I had a commercial contract in place for Software Collection, these were not as up to date with what the communities released (e.g. Ruby). Hence my approach of packaging and testing my own builds of these.

Not sure what you mean, if RedHat/CentOS is good for Amazon to run their 2M linux nodes, it is probably good enough for me to host my services. The stability and production readiness of the platform beats all the other Linux distributions, especially Ubuntu and I would argue Debian too (hello purify complains bug).

As someone who is using all of them, I'm inclined to agree.

CentOS only includes packages that Red Hat builds and supports on RHEL. Extra packages are maintained by the community as part of Fedora EPEL.

and a number of other useful community / product maintained repos such as PostgreSQL's official yum repo, ELRepo, Docker's official repo, Sernet's official Samba repo, Nginx's official repo etc...

What the hell? What happened to "don't break userspace"?

Yeah I know right, Ubuntu breaks userspace all the time, Debian isn't quite as bad but it's much worse than it used to be thanks to so many missing packages, try building a cluster / HA system on it - you actually can't!

RHEL has had a similar behavioral change during lifetime of a release so this particular case is not actually a good example for recommending RHEL over Ubuntu for stability.

https://www.percona.com/blog/2014/04/28/oom-relation-vm-swap...

Can you please share your experience a bit more ? How much SELinux is really required ? What about grsecurity.

My use case is custom build Django/Postgres app.

SELinux is an absolute must in todays world, simply put it prevents applications and namespace from doing thing outide of their area of interet, for example a web server should never be allowed to execute files from /tmp so it prevents that, another great example is when you have any form of multi-tennancy - for example you might have several docker or LXC containerised apps running on a host SELinux will / can ensure that even if of those apps is compromised and someone popped a shell in it, they couldn't go beyond anything running in that namespace, i.e. to the host itself or other apps running in containers on the same host.

It prevents a lot of sitations where an app might behave badly, for example perhps you're running a PHP app and a 0-day PHP flaw is discovered, there's a good chance that SELinux restrictions could prevent that exploit from being useful to the attacker by not allowing the app to do unusual things.

We have a rule - if it's not running SELinux in enforicng then it's not going into production.

We use to run GRSec as well, however with modern kernels a lot of the GRSec code base has been merged in so it's often not required.

I agree with you except for the GRSec part - most GRSec features are still missing except for a few low hanging fruit.

grsecurity is enhancing the security of the kernel itself, SELinux is for user space applications.

That's an oversimplification. Grsec protects both kernel and applications, although mostly via separate systems. Selinux mostly protects applications, but can also stop malicious interactions with kernel code (syscall and virtual file access restrictions)

It's not -- I'm getting OOM on some java processes and my swappiness is set to 60.

Java has its own memory limits that it will trigger OOMs at.

I didn't mess with swappiness, but I also didn't have a swap.

Centos/rhel are on a pretty old kernel at this point though?

We run the latest mainline kernel on all our 200-300ish CentOS hosts after it's passed standard testing and is available in Elrepo: http://elrepo.org/tiki/kernel-ml

Back when we ran Debian everywhere I wrote a CI project that would build the latest mainline kernel, applying GRSec and adding some SELinux policies in as best I could (backported from CentOS/Fedora funnily enough) and build them into a Debian package for deployment across our hosts, see: https://github.com/sammcj/kernel-ci

For Centos 7 its 3.10 according to good old http://distrowatch.com/table.php?distribution=centos

and it's whatever is the latest, stable, tested version of the kernel when using Elrepo's Kernel-ml repo: http://elrepo.org/tiki/kernel-ml

Don't use EPEL in production, it is an unstable mess. If you need more packages than are in the core of CentOS, Debian is the best choice.

How is EPEL an unstable mess?

I stopped using Ubuntu after Amazon affiliate marketing showed up on the desktop. I stopped using Arch after I read the source code for SystemD. FreeBSD had quite a learning curve, but now it's on my laptops and is my preferred server OS. I love DTrace. I love ZFS. I love pf. I love Jails. I love ports, even though I generally use pkg. I love how easy it is to compile a kernel. The FreeBSD handbook is awesome. All the JetBrains IDEs work well with it. Hardware support is spotty and battery life is terrible. The BSD license means you can incorporate FreeBSD into commercial products without infecting yourself with the GPL.

So you came from Arch, and thought FreeBSD had quite a learning curve? Hmm maybe I should rethink FreeBSD and stick with Mint.

Neither are that hard, FreeBSD has the additional challenge of overcoming Linuxisms. Just takes a little reading, and the FreeBSD Handbook is an excellent work.

FreeBSD has been a learning curve for me in terms of learning kernel development and things like kqueue and ZFS. I didn't find it difficult in terms of getting started, since it is a lot like Arch in terms of setting up things like X windows.

FreeBSD is actually a much cleaner operating system than other unix-like clones. I have used it for a decade but my work required me to use linux.

Best parts already mentioned in this thread (pf, jails, etc.) I would add one thing to the list:

https://www.freebsd.org/cgi/man.cgi?hier(7)

Since Cieplak mentioned systemd:

* https://freedesktop.org/software/systemd/man/file-hierarchy....

May I ask what's so bad with the SystemD source?

I wouldn't say that it's bad. I just find it to be quite complex, e.g., why does it need its own web server? I was also troubled by the number of Debian maintainers who left the Debian project over the intense controversy, as well as Linus's comments.

Systemd doesn't embed a webserver. It provides a service separate from both the logger and the main systemd process, which allows remote logs access. There are many issues I have with pulling various projects into one umbrella organisation, but saying systemd integrates webserver is just incorrect.

https://www.freedesktop.org/software/systemd/man/systemd-jou...

Precisely what I described. It's a web server that serves journald events/entries. It's separate from journald, and it's separate from systemd as a service. It's completely optional. It's no more embedded in systemd than nginx.

It does exist in the same umbrella project as systemd, (and I think that is an issue) but embedded is the wrong term.

> embedded is the wrong term

Don't use it then. Anyway, since systemd-the-project doesn't have a separate name from systemd-the-init, I don't see any problem with Cieplak's statement.

I'm calling it out because there are many people saying completely wrong things about systemd project. Both the init itself and other services. There's a lot of things that could be criticised about systemd, but if people constantly attack it just because they misunderstand it, it's going to be much harder to agree and change anything about the legitimate issues. Being specific helps.

So the anti-systemd trolls who harassed and attacked Debian developers until they quit are a point against systemd? Interesting perspective.

OpenBSD seems to do better on laptops driver-wise.

(I run Open and Free, and started with Net, I have no dog in this race)

I've always been curious about *bsd on laptops.

What laptop are you using and exactly what does/does not work for you?

Why not trying any systemd-less Linux distro?

E.g. Guix is a very cool Nix-like distro.

Alpine Linux is pretty awesome as well.

https://forum.alpinelinux.org/forum/general-discussion/alpin...

Small review of FreeBSD (I used it for a very short period of time, so I could be off the wall, though).

PREFACE

I've been using linux since the mid-90s. I know how to use the console, I compiled kernels, played with the old xfree86 config files and all that back in the day.

THE GOOD OF FreeBSD:

* Stable. It feels like a Unix out of the 80s. If you know what you're doing, it'll get the job done.

* man pages - Haven't used them much on Linux recently, felt that they were on the level of command --help, but on FreeBSD they felt like reading an authoritative blog post. Clear and concise.

* Cross-update versions - better than Debian. I personally never successfully had a seamless update between major releases, while FreeBSD did.

* No systemd (for now) - Unfortunately, systemd is taking over Linux because the largest block of user-space OSS programmers work for RedHat, which pushes for systemd. IIRC, the reason Debian went with systemd is because their maintainers simply have no resources to commit to remove it from gnome, etc.

However, as systemd grows in scope, FreeBSD will have to either play with it, re-implement it in a normal way (which will let other Linux systems find a way out), or stop being compatible with gtk, gnome, et al.

THE IRRELEVANT:

* BSD license - unless you're modifying and then distributing your modification, it doesn't matter.

* "Fragmantation" - True, "FreeBSD" is less fragmented than "Linux", but "BSD" is more fragmented than "Linux" and "Debian" (or RedHat or Ubuntu or Arch...) is the same not fragmented as FreeBSD. Look at the Distro as the OS rather than the kernel.

And it doesn't matter -

If you're distributing OSS, then release it as POSIX and configure; make; make install, and let the distro figure out packaging.

If you want to make life easier for the distro or releasing closed source software, Linux (on the server) has an order of magnitude more users than FreeBSD (https://en.wikipedia.org/wiki/Usage_share_of_operating_syste... ), and if your program works with Debian, Ubuntu and RedHat/CentOS, you'll probably cover the vast majority of users.

* Ports - Complicated to update, takes a long time (unless you have a fleet of servers), and can't be automated. And pkg is no better than apt.

* ZFS - ZFS is being ported to Linux, and btrfs may one day become stable. Either way, in a few years you'll be able to run the same ZFS on Linux as on FreeBSD.

THE GOOD IN LINUX

* Update - On Debian, you run apt upgrade && apt update. And it updates everything. And if you're on a stable release, updates are only bug-fixes, so (short of bugs/regressions), it can be automated.

FreeBSD, on the other hand, doesn't have enough maintainers to keep a stable ports tree, so every update can break your setup. So every update has to be taken care of manually.

* Security - Yes, I know that "experts" like the FreeBSD kernel better and everything, but the FreeBSD kernel has bugs of its own, so practically the key to secure both is keeping them up to date. And most of the time you're using ports, which isn't really run by FreeBSD and subject to their code review. So as Linux has an easier time updating (see above), it would be more secure.

* Directory structure - Why would you want a "blessed" vi? Why should a distro manager decide that /usr/bin/vi is (say) vim while neovim /usr/local/bin/nvi.

If anything, Linux is a lot more "Unixy" (I do know that FreeBSD is the "traditional" way, but I think that Linux is a lot more Unixy) - Why shouldn't I be able to write my own libc without having to maintain my own kernel? On Linux I can replace glibc with muslc. Why can't I do the same on FreeBSD?

If anything, this is what everyone's complaining about systemd: "It's not modular". Well, neither is the FreeBSD base. (Though in truth, FreeBSD seems much more careful about basic software hygiene than systemd).

* Default software - Maybe I'm unique, but I kind of like bash. I don't see why it shouldn't be my default shell except "Well, if it didn't exist in the 80s, it won't exist now" (It could also be a licensing issue, but as a user I don't really care about that).

Others will scream "ShellShock". Well, yes. Though ShellShock shouldn't be a security issue if you don't rely on userspace software to maintain security.

The OS should assume that software will break, and defend against it. So things like pledge or SELinux are much more effective (and sustainable) than treating userspace bugs as security holes.

Though in truth, I can't see why Apache should ever run bash honestly.

If I may comment on a couple of these (have been running Debian at home and FreeBSD at work for a long time):

> And pkg is no better than apt.

Apt is pretty good, so being no better than it isn't that bad.

> * Directory structure - Why would you want a "blessed" vi? Why should a distro manager decide that /usr/bin/vi is (say) vim while neovim /usr/local/bin/nvi.

> * Default software - Maybe I'm unique, but I kind of like bash. I don't see why it shouldn't be my default shell except "Well, if it didn't exist in the 80s, it won't exist now" (It could also be a licensing issue, but as a user I don't really care about that).

It helps to think of FreeBSD base as a minimum workable install: the goal is not to install everything you need, it's to get enough software that you can a) build whatever you need from ports, b) rebuild the kernel and the world. That's why /usr/bin/vi is vi (not vim) and the vi you like is in /usr/local/bin/, it's pretty helpful to have a screen oriented editor, so they've given you a basic one. I personally am not sure why they ship tcsh in base, but it's probably because they always have; philosophically they should just let you install tcsh if you need it, just like bash. Running a simpler shell than bash for /bin/sh is just good sense, Debian uses dash for /bin/sh too.

> Maybe I'm unique, but I kind of like bash. I don't see why it shouldn't be my default shell except "Well, if it didn't exist in the 80s, it won't exist now" (It could also be a licensing issue, but as a user I don't really care about that).

Where's the beef? If you like/want it, then just make it your default shell. All it takes is "pkg install bash" and "chsh". I happen to find the default /bin/sh (Almquist shell) provides just about everything I care about in bash, while being much lighter weight and less full of security holes, but if I ever want bash all I have to do is type "bash" from my sh prompt.

> doesn't have enough maintainers to keep a stable ports tree, so every update can break your setup

You mean frozen package versions + backported security fixes as in stable Debian branch? I think Debian is pretty unique in trying to support the whole world.

Just to clarify, FreeBSD as a project only supports the base system. Ports tree doesn't have any notion of releases and is intended to be always kept up to date, akin to Gentoo portage or Arch linux packages.

>You mean frozen package versions + backported security fixes as in stable Debian branch? I think Debian is pretty unique in trying to support the whole world.

>Just to clarify, FreeBSD as a project only supports the base system. Ports tree doesn't have any notion of releases and is intended to be always kept up to date, akin to Gentoo portage or Arch linux packages.

Yes. I understand the amount of work Debian admins put in (which is why I stick with Debian, even though nixos/guix seems much more interesting), but it's a strong advantage.

Errr.. the linked article means no, it is not safe to just cron an apt-get upgrade. In theory Debian is maintaining ABI/API. In practice there are issues. Rolling releases in practice have better tradeoffs, but require continuous deployment. On a workstation there are really no significant issues. On a production infrastructure, you need to use poudriere to maintain your own repos and orchestrate a branching and installation strategy.

btrfs seems like an elaborate hoax. ZoL is fine, but support and integration still go Illumos > FreeBSD > ZoL. There are enough commercial products using FreeBSD ZFS that it is robust.

UNIX uses the shell for a lot of things, interactive users are just one infrequent process. Bash is harmful as a non-interactive shell. Change your interactive shell to bash or zsh, problem solved.

>Errr.. the linked article means no, it is not safe to just cron an apt-get upgrade. In theory Debian is maintaining ABI/API. In practice there are issues. Rolling releases in practice have better tradeoffs, but require continuous deployment. On a workstation there are really no significant issues. On a production infrastructure, you need to use poudriere to maintain your own repos and orchestrate a branching and installation strategy.

There's a difference between manually logging in once a day and typing apt-get update && apt-get upgrade, seeing warnings, responding to them and applying and re-configuring/recompiling ports.

One can take five minutes. One can take an hour.

>On a workstation there are really no significant issues.

Yes, and on linux you can run testing. But I want more stability on a server.

> ZoL is fine, but support and integration still go Illumos > FreeBSD > ZoL. There are enough commercial products using FreeBSD ZFS that it is robust.

For some reason, all of a sudden everyone thinks that ZoL is legally safe to use. I'm not a lawyer, but whatever. The point is that it's still new and not tested significantly, but it should be more stable soon.

>UNIX uses the shell for a lot of things, interactive users are just one infrequent process. Bash is harmful as a non-interactive shell. Change your interactive shell to bash or zsh, problem solved.

So write shell scripts against /bin/sh or dash.

>Bash is harmful as a non-interactive shell.

How is it harmful? I mean why are your putting your shell in an untrusted environment?

Moreover, I understand if you like sh, tcsh, zsh or bash. My point is that it seems to me more of a "we always used tcsh/sh, so that's how it will stay forever" rather than "let's see what's the best default shell now and use it".

And why can't both shells (or zsh, which is MIT licensed) be included in base?

>For some reason, all of a sudden everyone thinks that ZoL is legally safe to use. I'm not a lawyer, but whatever.

That reason is because it's free software, even the FSF thinks it's fine to use ZFS on your system. The issue is whether it's fine for the ZFS module to be distributed with the Linux kernel.

And automatically updating ports is easy. Set up Poudriere somewhere to build every night, log in, check security notices, and install the already compiled packages.

Can you provide any good entry points to the BSD world?

There are several, depending from what one wants. TrueOS, formerly known as PC-BSD, is a good entry point if one wants a desktop-style system. Or one can try vanilla FreeBSD. Or OpenBSD. Have some user guides and Handbooks to read:

* NetBSD Guide: https://netbsd.org/docs/guide/en/

* FreeBSD Handbook: https://freebsd.org/doc/handbook/book.html

* DragonFlyBSD Handbook: https://www.dragonflybsd.org/docs/handbook/

* TrueOS User Guide: https://www.trueos.org/handbook/trueos.html

* PC-BSD User Guide: http://web.pcbsd.org/doc-archive/10.1.2/html/pcbsd.html (viewable off-line directly in both PDF and HTML forms in /usr/local/share/pcbsd/doc/)

Are you coming from Linux, Windows or OSX?

I am very good with Linux and Windows. So either of the two are fine.

I would perhaps spend some time running it in a VM, and then if you decide you like it and think it's worth the time investment, maybe try installing it on your laptop. I run it on a Thinkpad X1. The handbook is a great resource:

https://www.freebsd.org/doc/handbook/

The book, Design and Implementation of the FreeBSD Operating System is quite good if you like technical reading.

Do you Firefox? Because Firefox has Google as the default search engine which got them to receive hundreds of millions of dollars per year. Without that money,firefox would have been a lousy browser mainly by volunteers. Ubuntu tried to do something with amazon but probably did not earn enough money and was scrapped for more than a year.

You're mistaken, FF is using Yahoo! right now.

Not in the EU...

We are affected by this. Our elasticsearch process were getting killed even though we set xmx to half of the ram. Now reverting the kernel.

"I don't always upgrade my kernels, but when I do it's in production"

You have to do it some time.

Is the problem with just those users that changed the snappiness value to 0?

I finally switched my desktop to Xubuntu this week (from Win 10) after using it happily on the laptop for months.

Stability is, unfortunately, not great. Games exit unexpectedly, Firefox tabs frequently crash.

uname -r prints this exact kernel version so I assume I'm affected.

If you want something similar to Ubuntu, but something that is much more stable, I highly recommend Linux Mint. I've been using it for about 4 years now and it's a delight. Very stable and doesn't give me a lot of issues after upgrades. Their upgrade-schedule and the upgrades itself are very conservative.

They've based off a Ubuntu LTS, but I think they are starting to move away from that.

Linux Mint has issues - http://www.techrepublic.com/article/why-the-linux-mint-hack-...

Yes, a failure at safely distributing software is a pretty big failure when it comes to distributions.

Mint, elementary and the other "user friendly" distros should focus on creating the desktop and leave the distribution part to projects that have the resources and the staff to maintain the infrastructure.

Well, I'd argue just use base debian. All the same benefits, none of Mint's security issues.

Reply on news.ycombinator.com