Hacker News Reader

Comment by tremon | Hacker News Reader

Probably the #1 area that can produce results is avoiding/conquering the processor-memory gap: while processor performance has been growing exponentially, memory (bandwidth) performance has basically grown linearly. There is now a factor 1,000 difference between processor/memory speeds compared to 1980.

One of the areas that I have much hope for is near-data processing: since processors scale so much better, pretty much every peripheral device already has its own microcontroller. The idea behind NDP is basically to offload some data-heavy processing to the data layer. What if your disk layer could already preselect your data so the database wouldn't have to read and discard so many rows for each query? What if the network controller could evaluate your firewall rules itself, so dropped packets wouldn't have to interrupt the main CPU?

View on HN

> What if your disk layer could already preselect your data so the database wouldn't have to read and discard so many rows for each query?

My impression is that that the process of filtering DB rows is sufficiently complex to need a full libc type execution environment. But taking a big step back in perspective, a famous example of filtering on processors connected to disks is Map-Reduce, aka Hadoop.

> What if the network controller could evaluate your firewall rules itself, so dropped packets wouldn't have to interrupt the main CPU?

Yes, this is a real thing: https://duckduckgo.com/?q=nic+packet+filter+offload

So NDP is essentially what Commodore did with their 1541 disk drive. That disk drive had a 6502 in there to complement the 6502 in the actual VIC-20.

From what I remember, the IBM System Z mainframes also do this sort of thing and have dedicated IO processors that can decode XML on the fly for you and other fun things like that.

Every modern hard drive is a computer in its own right.

Seagate had a cool project where each hard drive ran linux and they used the physical sas cable to run a 2.5 Gbit network (or two actually) per drive.

So you could use that as block storage for luster, hadoop, or similar and enable things like direct disk to disk copies.

Cool idea, seems unlikely to hit a reasonable price point though.

As a matter of fact, I think this concept dates at least as far back as the IBM 360.

https://en.wikipedia.org/wiki/Channel_I/O

And you can find similar concepts in a standard PC today. The GPU is an example of offloading a workload to a specialized processor.

In addition, if each individual unit of memory hardware gained mini-CPU-like data processing capability, then you would have additional faux-CPU power that scaled linearly with memory; this would allow you to do some (embarassingly parallel) things much faster than just having one additional CPU per peripheral.

This is first time I'm hearing it but NDP seems really cool. I like the idea of augmenting the memory and peripheral devices with intelligence.

Is Kryder's Law also dead? (2x memory every 13 months) If not, we could plan for petabyte local memory by 2030 and exabytes 2040s.

Reply on news.ycombinator.com