While it takes ~100 picojoules to do a double precision floating point operation on a Ivy Bridge Intel processor, it takes 4200 picojoules to move the 64 bits from DRAM to your registers. Most people assume that the huge power usage is because you need to move data from off the chip, but the reality (and surprising fact to most people) is that over 60% (~2500 picojoules) of the energy usage of moving the data is consumed by the on chip cache hierarchy. That doesn't mean the SRAM caches themselves, but all the additional logic that makes it hardware managed (TLBs, etc) that give you functionality like virtual memory translations, cache coherency, etc.
Getting rid of all of that cruft that has been added since the 80s to make programmers lives easier would actually reduce power consumption and latency significantly... My startup is working on that problem by removing all of that additional logic from the hardware and instead having it managed at compile time. The best thing though would be having programmers really think about locality when writing their programs though.
Maybe a naive question. How do you foresee your approach with this succeeding given the failure of Itanium's similar approach?
1) As for the traditional VLIW problem, the simplified explanation of why it is difficult for most systems is because it is difficult to know exactly when a functional unit will actually receive/have access to a piece of data (either due to data hazards, latency due to the memory system, or many other factors). We solve this at the hardware level by being the first architecture to be able to guarantee latency between any location in memory. Once you can guarantee this in hardware, your compiler has a lot more information to be able to make decisions with and does not need to needlessly insert nops that hurt performance.
2)When it comes to software memory management, we have some new proprietary techniques for determining memory usage at compile time plus runtime tools. For obvious reasons I can't go into too much detail on how they work, but we will be publishing on it in the near future.
To summarize, we think that the reason others have never made a "sufficiently smart compiler" is because the hardware never gave enough data to the compiler and vice versa. We decided to have virtual memory (which we think is unnecessary) and instead opted for having all of our cores have access to a shared memory space, which simplifies both the hardware and makes memory mapping easier for the compiler. Hardware features that guarantee the latency for both operating and moving data along with the entire system being non blocking is what really gives our compiler the information necessary to efficiently pipeline things.
This is going to be interesting from a security point of view. It also sounds like users will need a new OS to take advantage of this?
If you have fully software managed memory it sounds like any binary running on the system has full access to any other memory? This is kind of the opposite of ARM "TrustZone".
Edit: I'm just asking these questions because novel architectures tend to sink without trace and the small-system world is currently dominated by ARM. You need a real "wow" factor to get people to change their tooling.
For memory protection, the most traditional way would be leaving it up to a RTOS or microkernel. Something very small and verifiably secure like seL4 is something we want to port.
We have not made it a huge priority to start of with as our customers have a small number of applications that are being ported and are isolated on the system. As each application needs to be recompiled for our architecture, we think that memory segmentation done at compile time is good enough to start with (in these limited cases).
> having it managed at compile time
Uhm, been there, managing scratch memory statically can only work for very high level (and very domain specific) languages. Hardly possible for C.
>managing scratch memory statically can only work for very high level (and very domain specific) languages. Hardly possible for C.
Well, we've solved that. Looking forward to sharing that in the near future.
If you really did, you've got much, much more than merely optimising a memory access. Consequences for the static analysis can be enormous. Looking forward to seeing your publication.
I speed up programs all the time by reorganizing how memory is layed out and accessed. Many time by factors of 12x or more.
I would think that making SIMD, parallelism, and multiple simple loops instead of one bigger loop much easier to program around would be much more realistic. Something like a fusion of ISPC, Rust, C++11, and Julia.
Our scratchpad (the analogous term for software managed memory, in comparison to a traditional hardware managed L1/L2/L3 cache system) for instance has single cycle latency along with zero bus turnaround. Along with our ability to guarantee memory latencies between any locations in memory, our whole goal is to try to never have a wasted cycle.
What about functional programming? IMO the biggest benefit from programming without state is that order of execution does not matter. Thus programs can be parallelized trivially. Under the hood you end up with "multiple simple loops" without really even trying. I think when more people catch onto this, we're going to see a rise in functional language usage because of how easy it makes parallelism.
You should talk to the Rust compiler folks :)
PS: Tell me more!
And it's not reasonable to abandon trees and graphs and everything just for the sake of cache locality. Algorithm first, CPU optimization second. Especially because you can control allocation very easily with something like an object pool, which will minimize cache misses.
Of course "cache friendly" is relative; any hashmap has pseudorandom memory accesses as a core part of its design so again, array scans will beat it below a certain number of elements.