A 32nm 1000-Processor Array | Hacker News Reader

A 32nm 1000-Processor Array

vcl.ece.ucdavis.edu

148 pointslossolo104 comments

It is surprising the paper makes no comparisons with GPUs. So I will do it.

For starters it looks like they are talking about integer operations (I only skimmed the paper and it mentions an ALU, not an FPU), whereas my GPU numbers below are single precision floating point numbers. So it is apples vs oranges.

So, a modern 14-16nm GPU like the Tesla P100 or RX 480 does about 5 to 10 trillion ops/sec at 200-300 W, and approximately 30 pJ/op. So GPUs are about 5x less power efficient. The paper authors did a good job. However GPUs are not optimized for absolute power efficiency, but mostly for performance per $.

Another big difference is that most GPU architectures are multi-lane SIMD (so single instructions acting on multiple data but multiple sets of those) whereas the linked architure is MIMD.

In simpler terms: these processors all execute independent code whereas a GPU tends to have multiple cores but a (sometimes much) smaller number of threads of execution.

Nvidia switched to MIMD in their Fermi architecture. I assume that they are the reason why jacquesm says most.

Eh, not really. 32 'threads' share a program counter on Fermi; it's still actually pretty heavily SIMD.

Also important is the memory bandwidth. You can do all the TFLOPs you want, but if you can't access the data fast enough, then there's a bottleneck.

(Dynamic) Power consumption goes up with square of voltage, so a comparison with a gpu's max power efficiency point (almost certainly underclocked and undervolted) could change the comparison significantly.

You are very right. I should have mentioned it in my comment. Back when I was mining bitcoins on AMD GPUs, undervolting and underclocking would easily double or triple the power efficiency.

Power consumption goes up with square of voltage

This needs to become common knowledge amongst nerds and tech types.

It was more widely known, when every geek overclocked; no more.

I was underclocking when everyone else was overclocking. Because I wanted quiet!

Cursory Google search shows Pascal GPUs at 10pJ/op for double precision (these guys did int, which should be 10x faster)[1,2]

[1] http://wccftech.com/nvidia-pascal-volta-gpus-sc15/

[2] http://prism.sejong.ac.kr/download/PRISM3_paper/PRISM_Ajayku...

So how do you program such a beast? What progress is being made on that front?

Cache coherency seems really hard to give up on, and even CPU-GPU cache coherency is becoming the expected norm, with even ARM delivering it.

It's very hard.

In the mid 1980s there was a CPU called a "Transputer" [1] made some of the people who moved to ARM. These CPUs could be connected together in huge networks and directly talk to each other.

The network of CPU's could auto-discover its topology, but coding for so many CPU's was difficult. Some specific algorithms scaled well with the number of CPUs, but most did not.

[1] https://en.wikipedia.org/wiki/Transputer

Occam did it quite nicely.

I think the reason transputers didn't 'make it' is not because they were super hard to program (it was only a little bit harder than programming a regular computer), but because the price premium you paid for a transputer set-up was too high and x86 got faster very rapidly.

This is right around the time when the first 386 machines were launched and in a very short time we went from 12-20 MHz 286 boxes (and some 68K machines for the lucky ones) to 33 MHz 386 machines with a ton of RAM.

So the advantage that transputers had was eroded very quickly and I don't think INMOS was ready to match pace.

Now that we've reached the end of the line for that kind of speed increase we are seeing a renewed interest in multi-cpu fabric architectures, of which the transputer was an instance.

Yes, but in the areas I saw at the time, transputers lost out in particular to i860 for floating point systems and to m68k for things like our mostly-integer "streaming map-reduce" -- it seems! -- system. (Being able to house 68ks in a blade system, i.e. VME/VXI crates, helped.)

I think the reason transputers didn't 'make it' is not because they were super hard to program

You could replace "transputers" with [insert-architecture] and wind up with a nice overarching theory to explain the history of computer hardware.

>> x86 got faster very rapidly.

Is ARM ( or something else) going to surpass x86 (amd64 whatever you want to call it) in the near future?

Just a layman, but i get the impression as of late that things are running into thermal and feature size issues.

And that these will affect all ISA equally.

I was involved with a system porting it from an old transputer based system to a PowerPC&C/C++ based one.

Transputers never really scaled into this sort of league, they were more 8/16/32 type scenarios..

For the interested, there was also the Atari Transputer Workstation (ATW).

...Although interesting, they were expensive and didn't perform compared to the equivalent sequential machine. Given development tho, who knows where it could have gone.

transputers had four links and comms beyond that required routing in s/w. I wrote such a thing for a transputer machine.

Also, the auto-discovery wasn't really auto... That was the boot code probing for CPUs on other ends of links and propagating itself to connected CPUs, building a map of the network in the process (which is kind of cool).

Wow, that reminds me of this[1] game. Lots of tiny processors messaging between each other. I didn't realize there were actual implementations.

[1]http://store.steampowered.com/app/370360/

This is "the" question. I suppose one could look at the Connection Machine: https://en.wikipedia.org/wiki/Connection_Machine The CM-2a had 4K or 8K processors and was reputedly very difficult to program.

Looked great though: http://www.corestore.org/cm2a.htm

The Connection Machine was a SIMD design and the languages available for it (StarLisp: https://omohundro.files.wordpress.com/2009/03/omohundro86_th... and C*: http://people.csail.mit.edu/bradley/cm5docs/CStarProgramming...) were actually pretty good compared to OpenCL.

The Connection Machine Lisp programming language described in Daniel Hillis' PhD dissertation was essentially going to be Lisp with parallel map/reduce but AFAIK was never done being implemented.

I think the big problem was that most of the SIMD algorithms were yet to be discovered at the time. For example this paper by Hillis and Steele was a very big deal but looks kind of basic today: http://uenics.evansville.edu/~mr56/ece757/DataParallelAlgori...

Guy Blelloch did a lot of work on Connection Machines and basically wrote the book on SIMD programming: http://www.cs.cmu.edu/~blelloch/papers/Ble90.pdf He also made a very nice programming language for parallel computing (NESL), as did Gary Sabot who worked at Connection Machines (Paralations). When you compare those to Hadoop or OpenCL it really is a wonder where we went wrong and what the designers of the latter were thinking (or not).

The CM-200 looks even cooler (at least in the brochure…): http://www.corestore.org/cm200.htm

Erlang/Elixir anything running on the BEAM

You program a machine like this much like you work with the unix command line when you string together a series of commands using 'pipes'. Only now the pipes are the comms links between the CPUs and the programs are the pieces of software executing on each of the CPUs, rather than that they all run on the same CPU.

So it's more of a data flow model than a function-call model, and every node will transform its input rather than that it will return results to callers.

This means that not every problem is easily mapped to such an architecture, but when there is a good correspondence between the topology of the network of CPUs and the problem then the throughput can be very good.

After all, there is no bottleneck where all of these have to access a shared resource (RAM) so when it pays off it pays off big.

This layman finds himself reminded of the Cell CPU used in the Playstation 3.

High performance computing relies on message passing -- if not pure and simple -- normally with the MPI standard. There are applications which scale at least past petaflops and 1000 cores is unexceptional.

The developers of the PETSc system wrote a position paper eschewing the typical hybrid (MPI+threading) techniques for trundling towards "exascale" -- if with MPI's shared memory support, which doesn't require cache coherence. (Apart from remote memory access, MPI departs from actor-/CSP-like systems in supporting things like collective communication (and i/o), and dynamic process management.)

I've read the first paragraphs, so take my comment with a grain of salt.

There are 12 memory modules on the chip, so I assume each is shared by less than 100 cores, which is close to a beefy multithreaded server chip. Each core also has a small local program memory that's not shared between them. It'd be not unlike programming a networked cluster of processing nodes each with a separate chunk of memory, except that the cores could be assigned to neighboring workloads.

Sounds fun!

Effectively you have to design for message-passing between microservices as a replacement for function calls. Not only do you have to step back from cache consistency but you have to step back from uniform memory access altogether and head towards a streaming architecture. It's very alien, but that's the only way to really use a system which has such a high ratio of processing power to external memory bandwidth/latency.

These people have some grand claims in that regard:

  - http://texasmulticore.com/

I keep failing to get plans to match up to talk to their CTO on how this would stack up against what a rebuilt (differently optimized) Erlang VM could do...

And, yes, I know: Not open source. No one needs to make that clever observation...

This reminds me of the Ambric am2045 with 336 cores from 2007. They put 40 of these into an X-Ray machine for a total of 13,000 cores.

https://en.wikipedia.org/wiki/Ambric

http://www.embeddedinsights.com/epd/Diagrams/nethra-am2045.j...

There's (1) talking about getting ~60 tflops from ~130k compute elements by 3d assembly of 128mb ram . But it requires microfluidics cooling so it would be very expensive.

But in another paper(2), the authors say that offering this chip, even expensively ,could start building the ecosystem and and environment to start a Moore's law like race to solve the heat issue via photonics .

(1)https://www.semanticscholar.org/paper/Fft-on-Xmt-Case-Study-...

(2)http://drum.lib.umd.edu/handle/1903/17153

Yeah whatevs. This reminds me of the ICL DAP with 1024 processors from 1979...

https://en.wikipedia.org/wiki/ICL_Distributed_Array_Processo...

I think that http://venraytechnology.com/ is the only high CPU density design I've seen that seems useful for general purpose, large-footprint computing, but their launch model (sell to a DRAM manufacturer and cash out) didn't go anywhere.

So many of these tons-of-cores chips can't really do much per chip, and are only suited for streaming algorithms like encryption, data packet routing, video stream processing, etc. They also have nowhere near the memory bandwidth to compare to GPUs, or to feed those many processing units with unique data per unit.

Would somebody please think of the memory requirements? :-P (or Venray, please seek investment and start pushing your designs yourself)

Thinking out loud.

Instead of imagining applications per processor, I imagine this device could map threads or message handlers to processors. It could work better with a functional language or at least some language that didn't explicitly manage parallelism in code but rather in the runtime. Offload the app writer to just coding algorithm and not thread synchronization.

E.g. imagine each timer wait being a processor spinning; each I/o loop being a processor that blocked/looped on an I/o pin state. With so many processors to schedule, it wouldn't stall application progress to spin or block an individual strand (until you ran out or processors). To make this efficient, they'd want interrupts and semaphore state to be hardware-supported. Instead of polling a memory location, block on a shared register masked where each bit was a condition. So instead of a 'kernel call' it'd be an opcode or small loop. Latency of wakeup then becomes about 1 machine cycle.

I imagine with the right runtime support this could be a useful device for a large I/o server. It could reduce latency of processing each client message to just the execution time. No time burned in kernel calls, process switching, stack copying, interrupt/event latency.

I'm an ex game engine developer and I bristle anytime anyone thinks any existing functional language is better for multicore. Specifically garbage collection alone will make any language an order of magnitude slower generally per a single core. Also the C/C++ game development community at least has great approaches to multicore which makes C/C++ linearly scale with scores to boot, see for example: http://www.gdcvault.com/play/1022186/Parallelizing-the-Naugh....

I love functional languages, more for thinking in them, prototyping ideas, especially compilers/visualizers, and etc. But for any language that adds garbage collection, immutable data structures (way more operations per write and crazy memory thrashing/alignment issues), unless used sparsely or in a mixed paradigm (ugh, except maybe scala/clojure) are going to pay a magnitude of performance loss.

Mind you there are tricks around using more system languages (C/C++/Rust/D etc) for a lot of the heavy lifting with the application core being functional that gets you closer to the best of both worlds.

Garbage collection is an idea. It can be slower, or faster, than other memory management techniques depending on implementation and specific usage.

Functional programming languages do not _require_ a GC. They just largely have it.

"Fibers" like you linked in the presentation (m:n green thread scheduling) have been in use for decades. Many, many languages other than C++ have had them for over a decade. Go is built on them.

Functional languages _can be_ better for multicore because of referential transparency. As a game dev you are used to working on 4-8 cores. Some of us work on 40-80 cores * 10k machines and have been for years. Much of your complaints such as immutable data overhead make sense if it let's you work on 10x more cores at the same time. I will also point out that immutable data _really_ is not 10x slower, unless you think those Haskell micro bench marks are all lies.

>Functional programming languages do not _require_ a GC. They just largely have it.

No they just require Infinite Memory [1] XOR GC.

Pure Functional programming has no concept of Alloc/Delloc. Let alone the concept of binding/assignment can fail. These are real. To quote James Michens [2]

>Pointers are real. They’re what the hardware understands. Somebody has to deal with them. You can’t just place a LISP book on top of an x86 chip and hope that the hardware learns about lambda calculus by osmosis. Denying the existence of pointers is like living in ancient Greece and denying the existence of Krackens and then being confused about why none of your ships ever make it to Morocco, or Ur-Morocco, or whatever Morocco was called back then. Pointers are like Krackens—real, living things that must be dealt with so that polite society can exist.

[1] Infinite memory simply means more memory then the program can ever consume... But the halting problem exists so you can't actually know how much memory your program will consume :P

[2] http://scholar.harvard.edu/files/mickens/files/thenightwatch...

PreScheme was a LISP to replace C for low-level programming. Used manual, memory management instead of GC. Very fast and efficient.

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.3.4...

Formally verified plus a Scheme48 interpreter as part of VLISP project.

https://en.wikipedia.org/wiki/PreScheme

ATS has no garbage collector that I'm aware of. I've seen it used in device drivers and 8-bit MCU's.

https://en.wikipedia.org/wiki/ATS_(programming_language)

LinearML is GC-less, functional, parallel programming.

https://github.com/pikatchu/LinearML

So, functional languages neither require GC's nor infinite memory. Two also combined low-level efficiency with easy, formal verification vs C programs. So, that's higher mapping of idea to code, high efficiency, and better safety all at once.

"You can’t just place a LISP book on top of an x86 chip"

I believe I just did with PreScheme. Microsoft goes further to straight-up use a theorem prover to do x86 coding.

http://research.microsoft.com/en-us/um/people/nick/coqasm.pd...

Well yeah but these languages aren't purely functionally which was what the original discussion was centered on. If you throw imperative elements into FP they're very useful yes. But you've broken the functional paradigm.

Also if all these things are amazing why aren't anyway using them?

Lambda Calculus and Turing Machine are equivalent in power. You can, as many academics have, model imperative programs as functional ones. You can compile functional programs, as almost all compilers do, to imperative code in C or assembler. We often make the distinction on syntax and structure a bit but they're both passing state into functions that optionally produce state. Edit to add that my forays in hardware show it's all functional (analog) underneath: mathematical functions running continuously without memory emulating abstract machines.

"Also if all these things are amazing why aren't anyway using them?"

Social and economic factors as usual. See Gabriel's essay Worse is Better:

https://www.jwz.org/doc/worse-is-better.html

Just take C language. I have its history in detail and with citations. It was literally an engineered language chopped up to run on bad hardware, chopped again with arbitrary alterations on bad hardware, and slightly extended for bad hardware again. Most people had bad hardware. Worked good on that. Spread like a virus with gradual improvements. Still nowhere near what engineered languages can pull off in various tradeoffs to consider. Yet, almost everything is written in it now thanks to it working on half-assed hardware, a MULTICS chop called UNIX doing so, UNIX distributed freely, and UNIX written in C. Social & economic factors spread it like a virus plus improved it to approximate solutions designed under cathedral model with better properties.

http://pastebin.com/UAQaWuWG

Meanwhile, alternatives sprang up that kicked both their butts in capabilities. The LISP machines, functional's answer to whole systems, had a flow and consistency you still can't match with modern stacks. B5000, a HW/OS combo designed for safe languages, would've given hackers hell. Amiga's combined SW and HW offloaders for excellent performance... like today's servers & game consoles. BeOS screamed in concurrency, multimedia, and ease of use while popular Windows and Mac boxes couldn't do but a fraction of it. AS/400's and VMS boxes ran, ran, and then ran some more with at least one person forgetting how to reboot them haha. Some in high-security survived NSA pentesting while things that get easily smashed by amateurs prevail today for security-critical work. I think it's clear a language or system's technical superiority has almost no causal relationship with mainstream adoption by laypeople or technical people.

Actually and sadly, it's usually better to bet on Worse is Better approaches with slight improvements for success or adoption. Occasionally, you can bet on The Right Thing with a win as Mozilla is doing with Rust. Heck, even Burroughs B5000 lives on in Unisys mainframes. OpenVMS lives on in Windows NT family as it cloned it for desktops minus strong focus on quality (sighs). ZeroMQ is a good example in middleware. Nix applying database-style principles to package management. Technically superior stuff occasionally mainstreams but not often. Human nature usually wins. :(

Pure functional programming doesn't require any special memory management beyond the stack, if it avoids any data representations that use reference semantics and have indefinite lifetimes. Lazy evaluation and higher order functions with environment are pretty much out. But even C can be functional:

   int (*pg)(int) = f();
   int x = h() + 2*z;
   int w = pg(y);
   /* ... etc */

Here, we just introduce new variables instead of assigning new ones, don't malloc anything and indirect only by means of dumb function pointers carrying no environments.

There could be some higher level (though nonetheless quite primitive) assignment-free language for specifying tasks for the 1000 cores of this chip, instead of programming them in assembler.

>Pure functional programming doesn't require any special memory management beyond the stack,

Yes. Just an infinite amount of stack. Why? See my previous post.

If we pretend for a minute we live in the real world... Oh guess what your stack can still over flow and assignment/binding can still fail.

Also in a pure stack based language you can't preform multithreading. If you do... Well now your building a whole OoO super-scalar functional VM on top of a physical processor just to avoid doing GC.

Infinite stack. Why? Oh, because there is no iteration so recursion has to be used? Iteration rewrites to tail calls though; they don't require stack. As far as real tail-calls go, you avoid the algorithms that blow stacks: stick to strict divide and conquer, to keep the stack depth logarithmic.

About multithreading: this language can describes a computation done by each of up to 1000 nodes. Those implicitly run in parallel.

Garbage collection is not slow.

Actually it's probably the fastest dynamic allocation method. Allocating data with a copying garbage collector is O(1), which is as fast as allocating something on the stack, while malloc() is usually O(log(n)) with n being the number of live objects in the heap.

Running a collection on a such GC is usually O(n), with n begin the number of live objects. This is way faster than calling free() on each allocated object, but slower than using the stack.

They idea that languages that rely on a garbage collector are slower is not really due to the garbage collector, but to the fact that they allocate way too much on the heap (I'm looking at you, Java). C/C++/Rust are awesome because they allow you to control way more where data is allocated. Also, it's theoretically possible for a compiler in a functional programming language to decide to allocate on the stack instead of the heap.

Another issue is that the time is will take to execute the collection can be unpredictable. It can happen that the GC will stop the entire program for a few milliseconds every few seconds. This is highly undesirable for realtime applications such as video-games.

The only problem with GCs is the stop the world pause. I'm seriously wondering why not more languages have a gc per "process" like erlang.

The only problem with "stop-the-world" GC's on hardware not built for them. There's "pauseless" collectors out there where that either doesn't happen or happens so fast you don't experience it. Some do microseconds. One was in a Scheme machine where they put it into the memory subsystem. So, the program just allocated, deleted, whatever with a parallel, hardware GC managing pages in the background.

Many things one can do in GC's. The only one I know with mainstream success is Azul's:

http://www.azulsystems.com/sites/default/files/images/c4_pap...

https://www.azul.com/products/zing/

Note: No affiliation with them. Their Azul systems and pauseless GC were simply the best stuff I found researching Java & GC hardware. Assuming they match marketing claims. ;)

> I bristle anytime anyone thinks any existing functional language is better for multicore.

.. and have they read the paper for this multicore? Though it has a large number of processors, there are severe resource constraints per node, with respect to how large a local program can be and how much memory is available.

But virtualizing such things is what OSs are all about.

> I bristle anytime anyone thinks any existing functional language is better

The parent comment doesn't say that, though. I imagine current functional languages are about as ill suited as current imperative languages for running on a 1000 core machine.

Something new might be needed.

I think that the concept of purity, present mostly in functional languages, helps a lot for writing parallellizable (?) code. My naive assumption would be that pure code could be adapted for a 1000-core machine with only changes to the compiler and runtime environment.

That said, few things in CS are written with this kind of parallellism in mind: even most algorithms' pseudocode is written in imperative style, and assumes the ratio of data to execution cores is on the same order as the data size itself.

We definitely need something new. Both on the algorithmic front and on the architectural front. I suspect that on this scale, you could easily need more than 10% of the computing power simply to feed the right data to the correct cores. A programmable memory management unit might be helpful.

> We definitely need something new.

Well, if only there would have been some ideas around…

like TTA[0] or Dataflow architecture[1]

These are just things that come to mind when thinking about an architecture like this. You'll probably still need something new, but this, instead of some bottle neck computing would be my starting point.

Admittedly I'm still waiting for processors like thisat consumer prices, but those ideas seem to have been forgotten at a time when I was still figuring out how to tie shoelaces and such.

[0]https://en.wikipedia.org/wiki/Transport_triggered_architectu... [1]https://en.wikipedia.org/wiki/Dataflow_architecture

> My naive assumption would be that pure code could be adapted for a 1000-core machine with only changes to the compiler and runtime environment.

Perhaps for use cases where current, garbage collected languages are suitable.

For areas where (today) asm/c/c++/rust is a must, you would need a functional language that can give you guarantees about garbage generation, so you can be sure you won't need to collect garbage, at least in specific areas of the code.

Perhaps a stream-oriented language would be suitable? The runtime could spin up more cores as needed depending on back-pressure.

> For areas where (today) asm/c/c++/rust is a must, you would need a functional language that can give you guarantees about garbage generation

That's the entire point of the "purity" idea. Pure FP languages give you very strict guarantees about garbage generation.

The only question is if everyday code has enough pure code to fill a 1000 cores processor.

> Pure FP languages give you very strict guarantees about garbage generation

I thought the only guarantee is they don't have side effects.

They are immutable, there's no implicit cross-dependency between data upper on the calling hierarchy, and any shared data can be recalculated as many times as needed.

That's what I get from the top of my mind. There are probably more features that will help. Purity is a very strict guarantee.

> So instead of a 'kernel call' it'd be an opcode or small loop.

Like a Wait For Interrupt instruction? If all idle processors sleep, I imagine it could save some power as well.

Yes! And an interrupt is any bit in a large (128bit?) register. Some bits are real hardware interrupts, some are local semaphores, some are global semaphores, and maybe one is a 'my timer register matches the system time counter'.

Existing OSs use a kernel call to wait, because they want to manage the (tiny number of) hyperthread(s) closely. But with essentially unlimited hyperthreads, they can manage themselves! And avoid the whole kernel-round-trip latency.

It's important for people to know these are eight bit CPU's. Least, that's what every other incarnation has been.

https://en.m.wikipedia.org/wiki/Kilocore

Asking what to do with 1,000 32- or 64-bit cores != 1,000 8-bit cores. Suddenly the value proposition doesnt seem so great. Such designs were used successfully, though, in both neural networks and genetic algorithms. Could probably handle stuff well that normally goes on DSP's.

This sort of thing, taken to even greater extremes, could be an interesting match to Actor languages, with an actor mapped to a processor. The overhead would be likely be ridiculous, as most actors would be blocked most of the time.

However, if it is true that we are now able to put a lot more transistors on a chip than we can power, that might actually be a feature rather than a bug. Assuming that power management is awesome and these actor-processors consume little or no energy when blocked, the power/performance ratio for most application should be awesome.

Does anyone have a link or the name of another weird architecture that was posted a while ago? (~3 months maybe?) I remember that there were a large number of cores that all communicated with each other in some weird way and that they like didn't have main system memory or something like that...

Pretty sure you mean us (REX Computing; http://rexcomputing.com), though I would not consider our network on chip "weird" :P

Our big difference that you were trying to remember was our use of scratchpad memory which is simply stated as this: We can radically reduce power consumption, increase density, and increase speed of on chip memory (SRAM) by removing the traditional hardware caching system. We instead use a purely software managed memory system, through some very fancy (or do I dare say "smart") compiler techniques that are enabled by a a very simplified architecture and the ability to guarantee latency for all memory operations. We can still have main system memory (meaning DRAM), it is just instead of having a bunch of complex hardware that burns a lot of power and wastes a lot of space in order to automatically fetch pages out of DRAM, we structure the code for each core to efficiently pull only the data necessary when it is needed.

Perhaps you've noticed the architecture of the new top top500 system. There's currently an HPC guessing game on filling in blanks in <http://www.netlib.org/utk/people/JackDongarra/PAPERS/sunway-... but of course CPU comparisons stop at Knights Landing.

One of the things I wondered about is how Linux is adapted to such an architecture. I couldn't find anything from REX on operating system support.

Yep, have seen it... it seems to not be exactly what was described by the Chinese at ISC or SC last year (their original description being much more in line with a DSP), though there is not much data available. All of the articles I am seeing today are calling it DEC Alpha "like", though the only real source for this seems to be the wikipedia article for the family of processors, with the latest version being in 2013. Dongarra even specifically says in the paper you linked that it was NOT related to the Alpha ISA, so it seems like all of the media articles are incorrect ;)

As for REX, as dnautics said, we've been mostly focused on running raw compute kernels on the current simulated versions (software and FPGA), and for the soon to be in hand silicon (coming this fall)... one of our projects internally is to port the L4 microkernel, and a telecom focused RTOS, but that is as far as our operating systems plans go for the near future. I'd also love to get a Plan 9/inferno demo running on it for fun, but we've got more important work to do at the moment.

Oh, so I guess the Sunway thing isn't as much like Neo conceptually as it looked to me from what little information I had. (I wasn't thinking about the ISA.) Of course the media are mainly useful for indications to chase primary sources.

The point about the OS related to the Linux-based one for Sunway, but maybe it only runs on the management processor anyhow, with just offload to the others. I'd commented in that respect that we really don't want something like Linux in an ideal world, so I'm pleased to see mention of L4.

Thanks for the comments, and good luck.

I don't think you're not supposed to run linux on the current rex architecture. I think you're supposed to (pre/) compile a program for it, issue it to the chips, pass the data on over, and then sic the programs on the data.

There's Tilera, and Parallella, but they're older than just a few months.

Could it be the mill cpu that you were thinking about?

Videos:

https://www.youtube.com/channel/UCKdGg6hZoUYnjyRUb08Kjbg/

The Mill CPU Architecture – The Compiler [video] (youtube.com):

https://news.ycombinator.com/item?id=9856334

Wiki:

https://en.wikipedia.org/wiki/Mill_CPU_Architecture

Could be a number of things. The Connection Machine? GreenArrays?

I think it was GreenArrays. Although in my searching Adapteva also seemed like it, but GreenArrays seems like the more likely candidate, plus it's come up more often here. I'll have to check when I get back on my desktop. Thanks much!

Maybe this? https://news.ycombinator.com/item?id=11845698

No :/ Thank you both for your suggestions though, they are appreciated :) If it helps it had the same sort of feel as the Mill architecture, in that it was a small project being developed by very enthusiastic people.

Rex computing neo chip? I know the founders and did a bit of contract work (virtual hve) on it and I think they wouldn't like being compared to mill :)

Maybe a Systolic Array thing? https://en.wikipedia.org/wiki/Systolic_array

Maybe a Massively Parallel Processor Array thing? https://en.wikipedia.org/wiki/Massively_parallel_processor_a...

Multiclet? https://en.wikipedia.org/wiki/Multiclet

Sutherlands 'Fleet'?

Doesn't look like it... there was an established website for the project. It seemed to have some big names from processor design and such behind it as well (at least from what I gathered on their wikipedia pages ;) )

For future reference, here is a link talking about the arch mentioned so that no one else has to wade through google's results for "fleet architecture":

http://arc.cecs.pdx.edu/publications

edit:

previous (2009) discussion about fleet: https://news.ycombinator.com/item?id=723882

That one?:

https://news.ycombinator.com/item?id=11845698

Dug around a bit for the paper, found this 2 page PDF:

http://vcl.ece.ucdavis.edu/pubs/2016.06.vlsi.symp.kiloCore/2...

The paper doesn't really explain how the its suppose to handle memory. You have a 1000 processors and 12 modules with 64KB ram. Now if you have to access real ram how will that be done? You would need a massive bus like a GPU so you could keep all the chips with data to do work. How do you keep the processors starving for data.

I understand its an academic project, but I wish they would explain what the real life use of this thing would be?

Here is another project I found: https://en.wikipedia.org/wiki/Kilocore

Not the same thing, but I wonder how much they sold? More like like 1000 small processing units instead of full cpu.

I'm certainly hopeful this will lead somewhere useful. Intel tried with Larrabee but apparently couldn't get the perf they wanted, especially with comparisons to GPUs

It would be great to know what the trade-offs are with this architecture

you mean xeon phi. I think it's still around. It is plenty fast, but not much better tdp than a gpu (which are commodity (read this as cheap)).

Xeon Phi is around and used in various financial computations. The problem with GPUs is not TDP, but programming flexibility. Programming a GPU is very hard to do well for generic computations, whereas x86 like CPUs have millions of programmers who can do a pretty decent job.

The Xeon Phi was never meant to compete with GPUs directly. The first version was underwhelming and expensive but used in order cores. The next version is actually available to buy as some sort of limited release. It has out of order cores and 144 512 bit SIMD units (2 per core * 72 cores). The full box costs $5000 last time I checked.

Would a language like Erlang make sense for a machine like this? Run 1000s of communicating processes without any shared data.

I don't think Erlang's VM is that optimized. Currently (as far as I remember) the upper limit is to run 1024 schedulers (threads) but I imagine it certainly wont be 10x more efficient than running 100 schedulers on 100 core machine

Architecturally it's a perfect match for such CPUs. If you listen to Joe talks he was talking about designing for 1000 core cpus more then 10 years ago. It is his stated opinion that Erlang is perfect match for such situations.

He's also mentioned FPGAs in similar terms, but I've not seen any news lately on that topic. Is it not practical? Not worth the effort?

This is a different ballgame but interesting, a recent Raspberry Pi 2 experiment [1] demonstrated good scaling of an Erlang application on multicore ARM up until the concurrency limit inherent in the application (around 128 processes).

[1] - https://medium.com/@pieterjan_m/erlang-pi2-arm-cluster-vs-xe...

Of course it's (much) better than anything else... though my experience is that it's quite hard to scale your application that much... You always need some shared resource and it gets ugly. And I also suspect Erlang has some implementation quirks (for example the algorithm of deciding which scheduler gets which process) that will prevent it from scaling that much. Yeah, I do believe that some day we'll reach such scalability, probably just not today.

Each scheduler has it's own run queue there is higher level migration logic to balance run queues based on statistics collected.

link to press release: https://www.ucdavis.edu/news/worlds-first-1000-processor-chi...

AIseek[0] had 10,000 processor chip in 2006, with simple connectivity and simple ops that were perfectly matched for graph algorithms. Their demo[1] would not be impressive today, but doing it in real time in 2006 was quite a feat and basically impossible without specialized hardware. Unfortunately, they're no longer in business.

[0] http://www.extremetech.com/extreme/75766-new-ai-chip-would-m... [1] https://www.youtube.com/watch?v=VeSQI2hinp0

> KiloCore processors ... store data and instructions inside i) local memory, ii) an arbitrary number of nearby processors, iii) on-chip independent memory modules, or iv) off-chip memory.

Ah, like GreenArrays' GA144, which contains 144 discrete cores - each with a small amount of memory - that can communicate with each other. The rationale with this chip is that you store data in the memory of one or more neighboring core(s) if it won't fit in the ~64 bytes (IIRC) of the core you're working in.

Where can I buy a sample?

Arguably the first 1000-processor chip was the CM-1, made in the 1980's: https://en.wikipedia.org/wiki/Connection_Machine

They also bury the lede somewhat. Apparently the KiloCore is a 1.78 Terahertz chip:

> The energy-efficient “KiloCore” chip has a maximum computation rate of 1.78 trillion instructions per second

Edit: they later say it's 1.78 GHz. I guess "trillion" is just a mistake?

Saying 1000 cores running at 1.78 GHz are a "1.78 THz chip" is a bit like saying 1000 cars driving at 65 mph are a "65,000 mph car"

It is. But while it sounds absurd, there are cases where it makes sense. For loads that are parallelizable, both the 1.78 THz chip and 65KMPH car make sense for similes.

I wouldn't say they are a 1.78THz chip or 65KMPH car, but like them.

It's a modern version of the megahertz myth: https://en.wikipedia.org/wiki/Megahertz_myth#Modern_adaptati...

I think this makes even less sense as we go to 1,000 processing elements -- even Amdahl's law is too optimistic for the workloads deployed in these scenarios (i.e., having an embarrassingly parallel computation workload doesn't help if we still need to access memory, including going through the shared interconnect(s) to do this, while also keeping cache coherence in mind--and most workloads have phases that have to be synchronized, assuming we eventually want to write the results of our computation somewhere): http://blogs.msdn.com/b/ddperf/archive/2009/04/29/parallel-s...

Instructions Per Second is different from the clock speed. In this case, it looks like each core is ~1 Instruction Per Clock, so 1 * 1.78 GHz * 1000 cores.

The trick is that getting 1.78 trillion IPS distributed over 1000 cores is very different from getting that over a few cores.

According to the link, the CM-1 had 16 processors on a chip, but there were a lot of chips (up to 4096 to yield up to 65536 processors). Still, the programming paradigms that were used for the Connection Machine might be useful for the massively parallel chip described here.

Maybe they mean "effective" speed. 1000x 1.78GHz = 1.78THz.

Assuming a job is embarrassingly parallel it might be an ok metric. Realistically, probably not.

... 1.78 GHz * 1000 processors ?

That wasn't a chip.

Reply on news.ycombinator.com