Forth Inventor Chuck Moore's $20 144 core CPU now in full production
greenarraychips.com
greenarraychips.com
That being said, I've heard him talk about this chips for years and it is great to see it finally see the light of day. If you were familiar with the Transputer technology, this is a better take on that, if you are familar with FPGAs you can think of it as an FPGA with processors instead of CLBs.
Things that it can do are similar to what Intel is doing with Larrabee or some of the CUDA stuff that nVidia has done. It doesn't have a GDDR3 interface to a GB of memory so you can't custom build your own GPU, but you could do your own PhysX type engine with it.
It also makes a helluva differential cryptanalysis tool, or a signals analysis tool in general.
I found it very easy to write XC correctly. I think that the CSP/transputer idea is really the best way for most parallel programming to happen. MPI and pthreads allow more flexibility, but in most (say, 80%) of cases, that's not needed for your application.
A cursory look at Go reminded me of some time I spent writing OCCAM code.
From the site: "With instruction times as low as 1400 picoseconds and consuming as little as 7 picojoules of energy, each of the 144 computers can do its work with unprecedented speed for a microcontroller and yet at unprecedentedly low energy cost, transitioning between running and suspended states in gate delay times. When suspended, each of the computers uses less than 100 nanowatts."
How does this instruction time compare to other modern processors?
It sounds to me like some of the benefit to be had here may be from the low amount of power necessary per core. Power and associated cooling is a MAJOR source of cost for datacenters, and so if this really is a significantly lower power consumption (again, I don't know how much power the alternatives use), then it could have a big impact on the cost of commodity computation power.
My colleague uses it for research purposes and he said that it is pretty mature.
Our architecture explicitly omits a clock, saving energy and time among other benefits.
Also, they quoted instruction execution time as being as low as 1400 picoseconds, so 700MHz equivalence is probably the best case, not the average case.
Edit: They also say in the architecture paper that their goal is one billion operations per second, or 1GHz.
[1] http://www.greenarraychips.com/home/documents/greg/PB002-100...
As a random comparison, that's 1.4 nanoseconds per cycle. The PIC24H 16bit PIC microcontroller form Microchip runs at 40MHz, making it 12.5 nanoseconds per cycle. Now, I don't know enough about Chuck Moore's processors and haven't read the docs yet, so I don't know, for example, how many cycle a typical instruction would take, or how much work it takes to synchronise cores, but assuming 1 cycle per instruction, that would make these 144 core $20 processors 8.9 times faster per core. I know its not realistic to assume linear performance speedup from additional cores, but if we do 144 things at once, this makes them 1285 times faster than the PIC24H microcontroller, yet these processors only cost 4 to 6 (depending on amount of RAM) times the cost of a PIC24H microcontroller.
Note that the PIC24H microcontrollers are the more expensive, high performance (but not the highest) model of Microchips 16-bit PIC microcontrollers.
I've done analysis of GA144 before: http://news.ycombinator.com/item?id=1810641
Most of the Chuck Moore design (I reviewed several, starting from M17) can be described by quote from Devil Wears Prada: "the same girl- stylish, slender, of course... worships the magazine. But so often, they turn out to be- I don't know- disappointing and, um... stupid". Chuck Moore designs are all slick, slender, stylish, worship Forth, but they turn out to be disappointing and stupid in the end. The only beneficiary often is Chuck Moore itself. You just cannot apply his experience to other places in the world of computing.
Let's see what we have here in GA144.
The memory inside all cores is way too small for general purpose programs, even if you split your program into 144 parts and spread them across cores. 64 words of RAM, ie, 128 bytes. 128 bytes times 144 - 18Kbytes. Same for the (program) ROM, and you should factor communication code in there. Communication cost affects RAM as well.
They offer no compiler from high-level language like C. You had to learn a specific dialect of Forth and some bizarre (albeit small) assembly language.
The only benefit for general population from this affair is the relative ease of the desing of asynchronous hardware.
The GA144 is so different from the classic way of computing that it requires new approaches of development. One very interesting feature of GA144 is that all 144 cores can share instructions over I/O lines. That means every core can send instructions to their neighbors who execute them directly without conversion.
I/O ist fast enough. I guess it should be possible to have (someday) an external interface to SRAM which circumvents the low memory problem.
> They offer no compiler from high-level language like C.
That's right, and that's the real weak spot of GA144. Almost every microcontroller board today comes with C or alike. Mr. Moore loves Forth but I doubt that there are many developers out there who like to be forced to learn Forth just for this single platform. I know Forth, it was one of my first languages I ever learned. It is perfectly suitable for embedded systems but you have to learn a lot to master it.
http://thesz.mskhug.ru/svn/hhdl/previous/HSDF/CoreletTest.hs
The link above contains some simple "Hello, world!" program, in five "big instructions" which contains 21 corelet instruction in total. 8 of those 21 instructions are send and front advancing instructions - their only purpose is to establish communication between program parts. My machine sends pointers to "big instructions", up to 32 bytes long (up to 32 instructions if you're lucky) while GA144 could send only 4 instructions max.
30% of 4 instructions is 1 instruction. Another one instruction from those 4 is a loop or jump or something like that. So you have two instructions to perform program logic. And this is best case.
So I again express dislike of GA144 as a computing machine. And I again express my gratitude to Chuck Moore for proving that clockless design works.
Unfortunately I don't have time to dive into your design but AFAIK the GA144 doesn't need 30% preparation code because every instruction can be executed immediately by neighbor nodes.
That means (correct me if I am wrong) if core X has to evaluate a Forth function of say five arguments then it could pass all five arguments to its neighbors (without any preparation) by sending them the code addresses of the arguments, wait until they have finished and then use their results to compute the function result. These neighbor nodes themselves could evaluate (or delegate) subexpressions to other (free) nodes and so on.
This form of parallelization would require an efficient shared memory access. This problem needs to be solved because AFAIK I/O ports are accessible by the edge cores only. It doesn't make much sense to transport each shared data through several columns or rows of cores.
You can send a word, ie, "big command" composed from four MISC command. One of MISC commands in "big command" can retrieve data from other core.
So most of the time you will wait to send a command or to receive some data.
FORTH doesn't have one to speak of. No dynamic memory management, no garbage collection. It's about as bare-bones as you can get.
PostScript, the JVM, the CLR and so forth are all backed by runtimes with powerful functionality.
LLVM SSA, modulo phi nodes, encodes expressions as DAGs in a completely straightforward sense: every operation names its arguments, and those names serve as unique references to a sub-DAG, since they cannot be reassigned. So given the final operation, you can follow the arguments recursively all the way through, completely transparently. Stack machines like the CLR and JVM encode expressions as trees serialized in a post-order - and to the extent that they use 'dup', they also encode expressions as DAGs.
A symbolic interpretation of a CLR or JVM instruction flow, or LLVM SSA instruction flow, can reconstruct the source expression DAG/tree; at which point you can re-encode it using the other approach.
And producing output targeting LLVM is even easier than it appears, because you don't need to worry about SSA yourself; just allocate stack locals (with alloca) and use the mem2reg pass to turn it into valid SSA.
Make it easy on yourself.
With that said, Factor is really cool.
Assuming someone wanted to produce a prototype board, but didn't want to purchase 1000's of CPU chips, are there many CPU chip makers who offer low quantities like this?
The design makes more sense when you stop trying to think of it as an ARM or AVR competitor, and more like a replacement for FPGA's.
If someone can make many-core, in this form, do something useful that can't be done elsewhere (unlike DSP and GPUs, which are already many-core), it will fundamentally upend computing.
.. and the laser(!)
http://www.greenarraychips.com/home/documents/greg/WP002-100...
Don't picture each core running C programs. Think the whole chip replacing something you would use an FPGA for, but more flexible, capable of adjusting its own behavior according to the environment.
The thing is, it's just a computer. The real value of this is hooking it to hardware to actually do something. So I suggest starting here instead: http://www.greenarraychips.com/home/documents/budget.html and learning how to work with hardware as well.
"suggestion for a complete system"
Some possible ideas
====================
- Sound processing (think 144 cores preforming Fourier Transforms)
- Pico Satellites (This chip has a surprisingly extreme low power requirement).
- Wireless communication (small power requirement means small batteries)
- Computer Vision processing. Imagine toys or tools that can process visual information faster than an Xbox.
- Basically anything that can be done in parallel, that is suited for small size and low power, but doesn't require much on chip memory.
On the other hand, picochip sells a 200-300 core DSP , that is being used for the wireless industry.
"SUITABILITY: The GA144 is designed to support the largest and most demanding computing challenges that can be addressed with a modest sized die in a relatively inexpensive and easy to use package while still using well less than 650 mW in most practical applications. The geometry allows for generous numbers of parallel paths and/or pipeline stages, or for complex flowgraphs in control, simulation, or DSP applications. Clusters of nodes devoted to functions such as cryptographic algorithms are easily placed in the large array, and the cluster needed to control external memory and run a high level language from it is well out of the way but has good surface area for interaction with other functions. Use it also as a universal protoytping platform for applications destined to run on our smaller chips. "
Can this replace a physics dedicated processor? What about latency?
Can anyone explain to a nonhardware geek?
http://www.greenarraychips.com/home/documents/pub/AP002-OSC....
The picture I get is that the boundary between computing components and control/sensor electronics is dissolved.
Gate arrays are vast arrays of logic gates, which can be wired together in almost arbitrary patterns by a sea of "fuses", typically controlled by state stored in on-board SRAM. They are real time and blindingly fast due to their massive parallelism, achieving supercomputer type speeds when applied to the right type of problem and programmed well. They are more difficult to program than a microprocessor. One way of looking at an FPGA is as an array of tens of millions of very simple computing engines.
Over the years, the number of transistors on an FPGA has been rocketing up. Generally these transistors have been put to use by providing more and more simple logic blocks. We are now to the point where we have almost more gates than we know what to do with, and the chip is being dominated by interconnects. This has seen a move towards including a limited number of elaborate hard wired blocks, such as CPUs and multipliers, in addition to the array of logic.
The logical evolution is to stop providing more blocks, but make each block more complex as transistor counts go up. Eventually we will see arrays of tens of millions of microprocessors, rather than tens of millions of logic blocks. There will be no distinction between a multicore CPU and FPGA.
It's worth noting that the first Xilinx FPGAs, thirty years ago, provided arrays of around 144 logic blocks, similar to the processor count in this chip. Extrapolate 30 years and we will have an array of 10 million microprocessors.
If anything defines an "FPGA guy" is choice of language. Historically FPGAs have been programmed with an HDL, such as VHDL or Verilog, because these languages are concurrent meaning they handle parallel systems.
My prediction is that as mainstream multicore CPUs develop we will see the rise of concurrent versions of mainstream languages, which will supplant HDLs. Just as DSP's are now programmed in (almost) vanilla C/C++/..., we will see FPGAs being programmed using the same languages as micros and DSPs. It will then be possible to write code and (almost) seamlessly run it on any platform, depending on how fast it needs to run.
I guess my feeling is that there's a fundamental difference between the sequential execution inherent in something like C and the ability to describe concurrency that is fundamental to HDLs, and it's going to be a hard bridge to build.
Side question: are "SW guys" satisfied with the tools/languages available for multi-core development? I've always thought things like OpenMP and CUDA were steps in the right direction but still hacks. It seems to me like there's still problems to solve there as well.
Not a SW guy, but I'm pretty sure the options can be boiled down to "pthreads or HDLs"
I'm not saying that such languages currently exist, but I do think they will come into being, as programming comes to grips with multicore CPUs. It will be essentially HDL synthesis, with the synthesiser/compiler being smart enough to hide all scheduling issues from the user.
As someone with a foot in both camps, I'm not happy that I have to choose early in the design process whether to run my code on a CPU or FPGA.
For example, I've seen a meaningful subset of Haskell in which the compiler would only accept programs which would provably terminate. The compiler output could then be sent down the FPGA synthesis toolchain.
Side question: are "SW guys" satisfied with the tools/languages available for multi-core development?
Yes and no. The new C++11 memory concurrency model and the atomics and multithreading library support are a huge step in the right direction. But it's nothing particularly slick and it still feels clunky. The fancy tools still seem to be not-perfectly portable and open source.
I've always thought things like OpenMP and CUDA were steps in the right direction but still hacks.
I'm guilty of not having used OpenMP. I should try it.
CUDA seems well-designed and has a decently well-supported toolchain. But it's a vendor-specific technology. OpenCL seems to be almost as fast and supported by more vendors.
It seems to me like there's still problems to solve there as well.
I remember as a kid hearing my father talk about the computer they were building at the university for research on high-level parallelization. That was the Illiac IV, completed in 1976.
There's nothing inherently special about Verilog/VHDL as concurrent languages--I imagine a very slick-looking Clojure module could be built as well. MyHDL let me reason about the program much more easily than a Verilog equivalent, all around the block in the cases of synchronous execution, blocking/non-blocking statements and integers ( http://www.myhdl.org/doku.php/why ). Also, (mostly) painless simulation and tests!
Though I would tend to agree that a C to Verilog system would be a step back; MyHDL makes heavy use of decorators and generators and other goodies that come from the functional programming world. C doesn't really work well there; you can hack it in but at that point you might as well just use Verilog.
It's like Erlang, maybe not as HDL-like as you were thinking. But, I don't see why you couldn't model every little gate as a process (obviously impractical), so it theoretically could fit.
I really like Intel's Threading Building Blocks for multicore programming in C++ and I really like Clojures take on time and concurrency, but overall, I feel language support is very limited and library support doesn't mesh as well with the languages as hoped. I would like to see a practical dataflow-centric language.
In (synthesizable) verilog, you have tiny state machines communicating via explicit channels (clock+wires+buses), and functions that get turned into gates. A higher level language would give you ideal channels (mapping onto fixed hardware channels or synthesized to HDL). Depending on their complexity, the functions at each state-node would also be transformed to use more general blocks and intermediate states.
limits of von Neumann architecture (and its underlying mathematics of recursive functions) as a mental framework for our thinking about computing.
Von Neumann architecture is what almost all computers use today: you have (very roughly) an ALU (arithmetic logic unit) hooked up to a memory bank which stores both program data and the instructions the program consist of.
Now you can add a couple of cores to that, but you pretty soon start to run into problems -- threads which try to access the same data, race conditions, etc.
But the biggest problem is that under the Von Neumann architecture all memory is shared so any thread can access any other threads memory. This puts rather drastic limits on how much benefit you can get from new cores.
You also run into issues like the limited speed it is possible to access the main memory banks with, etc, etc. This is possible to compensate to some degree with caches, but they have their own problems.
But the fundamental problem with them is that they were from and of an era where the clock speed kept increasing and increasing.
Today we have a situation where transistors gets smaller and smaller. But if you are trying to use this to make a traditional CPU with these new transistors, all this gets you is a really small chip.
What we need is an architecture inspired by something else. Personally I am kinda hoping it will be some form of message sending -- you run a lot of small (green) threads which each have their own memory as well as the ability to send and receive packages of information to/from the other cores of on the CPU.
You can have access to a (comparatively large but slower) shared memory bank too (like RAM today).
I like it because it works well with how you would design a cluster of computers (where you cannot afford the illusion of shared memory), how computation is organized under the actor model (which I prefer to threads) and it would be possible to implement with not that much new changes in the CPU.
If I may attempt a paraphrase: CPU caches stop being a bandage for slow access to RAM and become a valuable first class citizen for each core of the CPU when coupled with the actor model.
Did I understand you correctly? Again, thank you.
But if the cores of the CPU starts to communicate with the actor model then you wouldn't be using the memory close to the cores as a cache but as a storage area for messages that haven't been sent/processed yet as well as possible for thread local storage.
Also, check the site for the arrayForth stuff.
On another note, it would be interesting if something like these existed in the 64-bit size. Larrabee is interesting, but if it was a simpler stack machine at around this price point then perhaps some work could be done on different ways to do parallelism.
The technology itself was extremely interesting, though, and would probably have been a good fit for what I was doing. Also, the cost of the dev board included technical support and from speaking to the guy, I got the impression that they actually help you port your code to their platform.
On a more serious note, what we need much more of is not the processing speed. What we need much more of is what I call "memory processability", which roughly means "how many times per second can you process the whole memory" - or something like that. Basically how much CPU is there per RAM. Indexing is a great hack, but a hack it still is. Memory and processing cells must be merged into a single, massively parallel chip.
Users recall them fondly. In the end, clusters became too cheap for custom hardware like this to compete.
I might be misinterpreting, but I think the phrase you're looking for is "memory bandwidth".
However, I believe you would have to write completely new programs if it is possible.