Hacker News Reader

Comment by com2kid | Hacker News Reader

The 7F52 has 16MB of cache per core.

I'd love to see what is possible with a tiny runtime/OS in the kilobyte size and running a microservice written in a native language off of each core, everything out of the L3 cache.

I imagine the throughput would be amazing. Single thread per core, e.g. cooperative multitasking. Do this for stream orientated workflows, or even for processing data that is in reasonable sized chunks, it might be screaming fast!

View on HN

When I was younger, I was very pleased when I stumbled across the trick of putting all of a DOS system on a RAM-disk, which made the system fly. Then I got to college and discovered in my CS classes that CPUs now had multi-megabyte on-CPU cache, which immediately led me to wonder how hard it would be to make a DOS system that ran completely out of cache. I'm not convinced that it would be practical for any number of reasons, but it still blows me away every time I look at what modern CPUs have in-package:)

Run windows 95 in L3 cache and use this to store the install: https://github.com/prsyahmi/GpuRamDrive

This is effectively what you do with some of the smaller MCUs -- you're running out of a megabyte or two of on-die static RAM. This is more to eliminate latency in RTOSes than it is to improve throughput but it's the same idea.

More than eliminating latency I guess is done to eliminate latency variance, which is inherent in the relatively unpredictable behaviour of caches, and is required to characterize worst case scheduling of realtime systems.

Megabyte of RAM? That's a very big MCU for most of embedded programmers. 20KB is plenty for most, and some even go to zero RAM (ATtiny11).

Working with embedded systems. Different MCUs' memory ranges from bytes to gigabytes. What's plenty for one project isn't for another.

Yeah I've worked on projects with Toshiba MCUs with 10's of kB of RAM, and the last truly embedded project I was on used a safety-critical MCU from TI with about a megabyte of internal ECC RAM.

Coreboot does something similar to this. Very early in the boot process the RAM has not yet been initialized and so cannot be used. You need to execute some code to set up the RAM before you can use it. One approach to handling this is to write that initialization code in assembly language and make sure it only uses registers and doesn't touch any memory.

But that's inconvenient. You'd much rather be able to write your code in something like C. Any C code compiled by an ordinary compiler is going to access memory (at least for the stack). Writing and maintaining a custom C compiler for this is a lot of work and still comes with a lot of limitations on the C code that it will accept.

So they set up the cache in a way that it will never try to write its contents out to RAM. Then you can use the output of an ordinary C compiler that uses memory accesses and those accesses will all be served by the cache and never touch RAM. They call this "Cache as RAM".

With some work you could probably boot into DOS like this.

https://www.coreboot.org/data/yhlu/cache_as_ram_lb_09142006....

Not just coreboot, pretty much any BIOS uses cache-as-RAM.

Wow that is so cool. Most of us never have to dig that deep and so weird stuff like that usually flies completely under the radar...

Thank you for sharing!

Interesting that no one to my knowledge has built an architecture where sram is ‘memory’ such that it would be transparent to C code but no slower dram is used. 32MB is more than my 1998 computer had.

Many embedded systems have a block of SRAM directly addressable.

modify tcc maybe

This is a common approach for most bootloaders, in fact for several microprocessors (the IBM 440 comes to mind), it is the only way to boot.

Is there a way to mess with the cache or MMU and prevent it from _trying_ to write out to DRAM, and be able run a minimal system purely from cache with the DRAM slots completely empty?

> I'm not convinced that it would be practical for any number of reasons, but it still blows me away every time I look at what modern CPUs have in-package:)

The optimism is premature. A CPU may flush cache even when it pages exactly the same page. Modern CPUs have too much of completely unfathomable prefetch, and cache coherence logic

But they're also configurable via various config registers, and can be set to not touch RAM, as most early-boot firmware uses.

No, all tricks to control cache behaviour are not 100% reliable on modern CPUs, or at least intel ones

Ha. I still use ram disk for local training of machine learning models. You can get a SOLID throughput boost by just loading up the dataset in RAM... My work desktop has something like 192Gb, so can handle a pretty sizeable dataset this way.

Isn't this what 5-eyes have been doing with MiniX3? etc

There is an interesting argument to be made for DSP like workloads for these chips. With x8 PCIe 4.0 lanes to carry inter-socket traffic.

If I had infinite money to spend I would build a hypercube out of these bad boys. each "atom" having 6 x8 PCIe connectors (up,down,left,right,front,back) with corner nodes dedicating a x16 PCIe port for I/O in and out of the cube.

Immerse that bad boy in flourinert and contemplates the deepest secrets of the universe :-)

That would just be a cube, right?

OP's referring to an old architecture which was called "the hypercube" which involved a couple (I can't remember if 16 32, 64 or what number) of microprocessors connected to one another in a certainly-not-2d topology.

Edit: oh a bit of searching and it was in the order of thousands of microprocessors. It was called "Connection Machine", you can read about it in Wikipedia[0].

[0] https://en.wikipedia.org/wiki/Connection_Machine

One of my favorite essays ever is on the subject of the Connection Machine and Richard Feynman: http://longnow.org/essays/richard-feynman-connection-machine...

Thank you for this, that was a really excellent read. Nothing better than old computer architecture and some early ML!

TMC was one of the coolest companies of all time; I really wish they were talked about more often.

Nominally hypercube architecture is 'n' dimensional[1], which relates to the number of channels at each vertex. Assuming you took the front and back plane of this thing and connected them together it would qualify as having 6 dimensions (6 channels per vertex). The first such system I had a chance to play with was the Intel one[2] which was order 6 using Intels iAPX432 processors.

[1] The (original?) Oak Ridge National Labs paper -- https://www.osti.gov/biblio/6487986-parallel-computing-hyper...

[2] https://www.nature.com/articles/313616b0.pdf?proof=trueIn

Oh, I see. If your cube is exactly 4x4x4, and you loop it in each dimension, then it's topologically equivalent to an order-6 hypercube with a node on each vertex.

In the general case, you need log2(N) connections in each direction to turn an NxN or NxNxN topology into a hypercube, so it's not as simple as "left/right". (For example, for N=8, you might connect each position to position xor 1, position xor 2, and position xor 4.)

I learned from an electrical engineer that works on systems for the power grid that running the whole OS in CPU cache is a requirement for some of those systems. I guess that makes sense when you are working with things that move at the speed of electricity.

No, the system is probably using tightly coupled memory; the CPU has single cycle access to all of system memory, effectively making a cache redundant.

And the reason is likely "we are building something realtime, we can't afford variable latency".

For the electricity grid in today's world, that's over-engineering things. The 60Hz electricity grid doesn't really need anything being changed at more than 600Hz, and considering your CPU core runs at 3,000,000,000Hz, you can totally waste a lot of cycles before you start missing deadlines.

Even if your CPU cycle budget is large, designing a realtime system requires proving thet it won't be exceeded, by adding up the worst-case running time of the worst-case sequence of events. Avoiding high latency memory gives a few orders of magnitude of margin, which could be needed to afford good functionality on a reasonably cheap CPU; it's not a mere useless improvement from "fast enough" to "as fast as possible".

And yet, so much software on modern CPUs takes seconds rather than milliseconds to perform and output simple calculations. Especially with the mantra "developer time is more expensive than computing power", few developers nowadays have recent experience of optimizing programs for sub-ms reaction times. You could probably make it work with standard hardware. But since developers have to be retrained for that field anyway, why not use hardware that's battle-tested and gives you more leeway?

In addition, the advantage of optimizing at every point is that you can use hardware not designed at the limit of what's possible, likely giving you lower error rates.

"This meeting is to announce we've settled on an architecture for the next gen grid optimization program: Electron UI with a Ruby on Rails app underneath the hood that calls to AWS Lambda for anything that's cpu-heavy. State is persisted to Firebase as serialized YAML, which should scale really well. And to our normal crowd of performance zealots, don't even start with your normal "latency" and "outages" griping. We're comfortable with this stack and besides, we can waste a _lot_ of cycles before we start missing deadlines!"

In hard realtime systems, the problem is not "we have enough time to do the computation" but "we can prove the computation will run in the available time".

The complexity in modern mulit user operating systems (many processes, a run queue, spin locks) along with modern processors (multi tier memory architecture, micro-code, branch prediction, long stalls for memory access) mean it's very hard to prove that anything will run in finite time.

IMO the terms "hard realtime" and "soft realtime" are unnecessarily confusing to beginners. I think that "deterministically bounded latency" and "probabilistically bounded latency" are more descriptive. But they're long and difficult to type, so we'd inevitably call them DBL and PBL, and be back at confusion.

Would KolibriOS fit the constraints to do something like this? Per the description it "requires only a few megabyte disk space and 8MB of RAM to run." but has a recognizably usable GUI and office suite.

https://kolibrios.org/en/

Better the original MenuetOS which supports now 64bit, though this is not open source anymore.

Unikernel as a c++ header would be a good candidate: https://github.com/includeos/IncludeOS

kdb+ [1] used to advertise already about a decade ago that their database code fits into L3 cache, making it extremely fast in execution. With 16MB, you can probably fit the core of a pretty much fully featured database in there, accelerating data driven jobs that GPUs are suited for.

[1] https://kx.com/

This is kind of what the SPUs of the PS3 were like, except with way way less RAM.

Besides cryptographic heavy services, I'm having trouble imagining a microservice that would benefit from this.

Disk, Network, or in-memory artifact reference would seem to be much bigger bottlenecks.

I'm thinking of simplicity. No VMs, no containers, just a single binary.

The majority of microservices I've written would fit into a system like this just fine.

NodeJS is built around the idea that a single thread of execution is enough for a stupidly large % of backend code. On top of that is ton of infrastructure to deploy slews of 100 line JS microservices that each do one thing.

So, back to basics. No VM, no disk at runtime, a single statically linked binary running on top of nearly bare metal.

Available memory is whatever space the program code doesn't take up.

Ignoring l3 being unaddressable, I wonder if UEFI applications are powerful enough to pull something like image deployment off.

This is probably all less efficient than the current Russian doll system of VMs and containers in use right now, but it is fun to imagine!

Basically unikernels on the Jailhouse hypervisor.

> Ignoring l3 being unaddressable

If you set the right config registers, you can map the cache to a linear address space and use it like RAM.

I think to do that, you loose access to actual RAM though, so it isn't awfully practical.

I can see this being used in hft houses

My eyes read that as

>I can see this being used to lift houses.

My evening has been made that much more interesting.

I thought they all use custom hardware by now? But it's probably useful for applications that go beyond simple algorithms, work on time series data could be much faster.

They use custom hardware for the basic kind of latency arb but software is used for more complicated stuff.

I don't understand why you would want to run tiny systems for that. Properly configured Linux processes will do the same.

I'm not sure with cache aliasing you'd be guaranteed that it would stay all in L3.

It's have to be a special setup for sure, a board with no memory even hanging off of it.

In theory all you'd need is some tight integration with the NIC (or a few of em!) and tiny driver that knows how to DMA to the NIC.

Larger problem of course is that L3 isn't addressable. I wonder how much money a cloud provider would have to pony up to AMD to get an exception to that? :)

A huge # of workloads would fit in 16MB with no storage hanging off. Heck I think every microservice I've ever written could manage that.

There have been some efforts of late to make a cloud first OS, honestly once you remove everything except for "talk to network controller" and "parse JSON", well so long as you don't mind writing in a language that isn't JS, a few MB is plenty for lots of workloads.

Apparently there are ways to treat the cache as a static scratch pad. It is used by BIOSes during early startup, but I'm not sure whether the magic incantations are publicly known. It is second hand knowledge though, so it might just be folklore.

If you allocate one big block it would fit even in a 1-way cache. Modern chips appear to have 16-way L3, so you'd have to be trying to spill.

For most general purpose work, you’d probably lose throughout. Chances are you’d rather have the heap be in the cache rather than the code segments.

kdb

I'm not sure why you are getting downvoted, because kdb+/q would be a perfect candidate. Too bad kOS is no longer still a thing (or maybe it never was).

>640k ought to be enough for anybody

Reply on news.ycombinator.com