Hacker News Reader

Comment by Keyframe | Hacker News Reader

I must admit I only glanced at the article and in the past when topic appeared. How does this work, the speed of 1.7s I mean? Taking a look at input, let's say average entry is 16 bytes, there's billion of it; That's ~16GB. Average read speed of SSDs is what, ~500MB/s? Scanning alone at max throughput will take half a minute. This must be relying on DDR4+ read speeds which would probably come in under a second in certain cases. Is that's what's going on, RAM disk?

View on HN

I used to benchmark a lot on an enterprise-grade SSD 10 years ago, and that was already at 2 GB/s. Today, even my laptop's SSD supports multiple GB/s.

But you're right about the contest -- each program was measured five times in a row, and there was enough RAM to fit the entire file into the page cache.

The best time using all 32 cores (64 hyperthreads) on the evaluation machine was 323 milliseconds.

I went to the original repo, it's indeed a RAM disk. That makes it clear then. I was already almost excited about some IO wizardry going on.

Results are determined by running the program on a Hetzner AX161 dedicated server (32 core AMD EPYC™ 7502P (Zen2), 128 GB RAM).

Programs are run from a RAM disk (i.o. the IO overhead for loading the file from disk is not relevant), using 8 cores of the machine.

Running without RAM cache would be a great followup to this challenge. I think a time around 2-3 seconds should be achievable. But, it would be highly sensitive to the hardware setup and how well the disk-reading code is placed on cores relative to the connection to the disk. Not sure what it would take to allow hundreds of contestants to benefit from the best arrangement.

PCIe 3.0 x4 SSD runs at ~3500MB/s, PCIe 4.0 x4 SSD runs at ~7000MB/s and PCIe 5.0 x4 SSD runs at ~14000MB/s (yes - megabytes, not megabits)!

blazing! I had ye olde SSD SATA bricks in mind though.

1 KIOXIA SSD on PCI 5 can do that 14000MB/s and 2M IOPS on 4 lanes. Some servers have 100+ lanes. You can have some serious speed if you want!

14000MB/s? That's around 14,000,000 KB/s !

Reply on news.ycombinator.com