Hacker News Reader

Comment by NelsonMinar | Hacker News Reader

I love the bit on slide 16 where initializing 8GB of RAM takes 100MS, so they boot with only 2GB and then hotplug in more memory later.

View on HN

There are some drawbacks to that. Memory flagged as hotplug can't be used by the kernel itself (because otherwise unplugging the RAM would risk crashing the system). That includes things like network buffers and a number of things which use the total RAM size as a guideline.

There was a bug in a proposed kernel patch a while back, that we were testing. It ended up designating a good chunk of the machine's RAM as hotpluggable. Test suite was passing, but the benchmarks showed a drastic performance drop. Took a while to track down why that was happening.

Isn't the solution introducing another type of memory abstraction in the kernel next to "available since boot" and "hot pluggable", namely "later added but won't be removed"?

This solution didn't introduce it. The concepts being used have existed in Linux for quite some time.

Here's a patch of mine that added (some of) it from 2005: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

I think you may be confusing some of the aspects of hot-add with hot-remove. Memory which you want to later remove should be onlined as "movable" and it has some content limitations. But, not all hot-added memory must be onlined that way. You can very easily just online it as "normal", which is (believe it or not) normal and can have any allocation type placed there.

Just last weekend I was talking to one of FreeBSD's VM gurus about doing exactly this in FreeBSD too. You have to be careful about it though -- some data structures are autosized at boot, and if you're not careful you can end up with systems "running out of memory" because they have too much memory.

Reminds me of Windows 95 "running out of memory" on systems with too much RAM.

https://devblogs.microsoft.com/oldnewthing/20030814-00/?p=42...

> some data structures are autosized at boot

It's probably a good idea to audit all the things that are autosized at boot by ram size; some of them are probably much too large on a system with 128 GB or so, given some of the sizing was written when 128 MB was big. I know I've run into the size of IP fragment buffers being way too big, but not sure what else.

On Linux, most (maybe even all) of the data structures that must grow with memory size can be carved out of the memory being added. The main culprit is 'struct page' which typically needs 64b of metadata for each 4k physical page (on x86 at least). Much of the code to do this stuff is in Linux's "sparsemem" infrastructure: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

A lot of embedded systems have been doing this for years.

We've been progressively loading webpages for years. It's like the 7th layer of the stack is working its way down.

Lazy loading is not unique for the web or in anyway a new idea. The web is in fact one of the last places it has appeared in.

Indeed. In apps lazy loading is something you add when you need it. But for kernels, it’s the first problem you tackle: how to pick yourself up with your bootstraps. Booting is tricky now, and it always has been. Right back to the early machines that were booted by manually entering machine code via switches.

Interestingly AWS virtual machines still boot in 16-bit mode, then bootstrap up through 32 and then 64 bit modes, last I heard.

All Intel-compatible and emulated Intel-compatible processors do this. You'd have to run ARM to avoid it.

I thought x86 started in 64-bit mode when using UEFI?

More like the UEFI puts it into 32-bit/64-bit mode before it gets to the kernel, the hardware didn’t change.

Reply on news.ycombinator.com