Hacker News Reader

Comment by randyzwitch | Hacker News Reader

I have the 3970X and use it with Ubuntu. It's fun to have a lot of cores, but in the beginning it's highlighted some areas where open-source programs aren't assuming you're going to use that many cores (similar to what they talk about for Windows). So you see OOM errors with make, see program crashes from defaults being too low (such as creating too many Postgres connections), things like that.

Not taking away from those developers, just sharing my experience. When you do get something that works really well (the newest version of Julia makes it easy to write multithreaded code), it's hilarious running htop and seeing 64 threads of green running.

View on HN

I've got a 256 thread machine under my desk at the moment, I find myself frequently disabling threads because i'm out of memory.

I don't think people fully grasp that lots of applications memory requirements scale with the number of threads. Just running gcc against C make files can burn 1/2 a GB a process, multiply that by the 8 cores most people have and 16G of ram leaves plenty of memory for Firefox/etc.

OTOH, 128GB of ram in a 256 thread machine, is borderline out of ram. Build a project where its more like 2G of ram per process (looking at a lot of recent big public google projects) and its more like I need 1/2 a TB and a bunch of swap to keep the project from OOMing.

So at this point I might use that as a general rule, 2G per thread. So your looking at another ~$1.5k+ or so in RAM for this machine for most purposes.

I second this. My experience with many core machines (32-64 cores) is kinda dissappointing so far.

Yes, running "make -j" on these machines feel amazing at first, but you will soon either 1) run out of memory, or 2) notice that you don't need so many cores anyway due to build dependencies.

Hmm, my experience has been resolutely great; we just upgraded to 64 thread 3970x's with 256GB of RAM, and pretty much everything worked smoothly out of the box; I'm very pleasantly surprised by the lack of tuning necessary.

We did do the math and decided that for us 4GB per thread is likely safe up front; 2GB per thread might have been pushing it. Another reason to avoid the 3990x (for us) was the tricky scaling of the previous generation's 2990WX; We don't have faith that all of our code would run well on the more memory-channel constrained machine.

I mean, if you're just running a single fairly small build, sure, it's likely overkill...

Interesting heuristic. That means my 6 core-12 thread CPU will need to have 24GB of RAM. I am on 32GB so it's cool.

> htop / 64 threads

It sure is fun to see the entire top half of htop being taken up by individual CPU bars, but this is another program whose assumptions have been overtaken by the current silicon: At some point it is not that helpful anymore and makes looking at the processes themselves harder.

Sadly the development of htop seems to have stalled, patches/PRs for that and many other issues remain unmerged and unanswered.

you can easily change the CPU bars to be quarter sized.

If you press 'F2' and look at the available meters you should see half and quarter sizes.

Even then -- https://gamozolabs.github.io/assets/afl_256thread.png

Note that this CPU has 128 threads, so half of that picture. For my usual terminal size I might get 0-2 processes listed below that...

I’ve found htop becomes less useful after about an 8 core count. Is there a reason you’d run htop over top in a normal use case?

Htop is IMO better overall with more helpful coloring, filtering, etc.

You can replace all of those with a single Avg bar.

At this number of processors, it starts to make sense to plot min/median/mean/90%ile/99%ile/max instead of gauges for each cpu.

I would love to have this kind of problem :)

_edit_: or just all the deciles and 1%ile + 99%ile

What in the world are you doing when that picture was taken??

> It sure is fun to see the entire top half of htop being taken up by individual CPU bars,

You should have seen it with the Xeon Phi - the Phi could do 4 threads per core and the big ones had 72 of them, for a mind-blowing 288 threads. It was, however, much less forgiving than either of these beasts - L2 cache was Atom-sized and cache misses were not cheap.

IIRC, the SGI Origin 2800 could go up to 512 MIPS R14000 processors. It's one thread per socket but, with 512 sockets and a single memory image, it'd most likely end up crashing htop. ;-)

That is definitely not the CPUs problem to worry about. Operating systems need to catch up.

Yes, that was generally my point. Even if available code already took advantage of multiple cores, my experience has shown that currently these chips bring out some latent assumptions on what hardware the code will be running on (whether that's OOM or UX or whatever)

That's why we have things like Rust - so we can write multi-threaded apps without shooting ourselves on the foot.

Reply on news.ycombinator.com