A 1000-processor chip powered by an AA battery (2016)

This is reminiscent of Parallella [1]/Epiphany (which unfortunately failed to gain traction). They didn't release any designs with core counts like that, but the entire point of the design was to enable large numbers of cores with in-core memory and predictable latency for accessing the memory of cores elsewhere in the grid.

Epiphany did have (slower) access to external memory too, but the challenge of these architectures is finding problems that requires too much branching to be able to take proper advantage of GPUs, yet so parallel that relatively low single-core performance is outweighed by throwing more cores at it and/or where the low power usage of the Epiphany would be worth it, and it seems they just didn't find enough of a market for it.

I have two of the kickstarter boards sitting around. Wish they'd gotten larger scale versions out.

[1] https://www.parallella.org/

Reminds me of the PS/2 CELL Architecture, with the SPUs having limited local memory, and needing to DMA stuff around in a streaming fashion to build up larger programs.

Yep, at KnuEdge we built some processors with thousands of cores in that style architecture. It's a bit different to reason about but you can do a lot of wild computing that way. Too bad the company folded before any of our processors were available.

Yes, "Cores operate at an average maximum clock frequency of 1.78 GHz, and they transfer data directly to each other rather than using a pooled memory area that can become a bottleneck for data."

This makes sense, everyone loves debugging MPI programs, so of course we'd want to replicate that experience on the desktop.

yep each transputer had 4 serial links, which could be connected in matrixes, and allow data to be moved across the array directly. They used a language called OCCAM, and there was an OS called TAOS that allowed you to run various transputer topologies.

In addition to the memory access I was wondering how they got to 1000 cores and not 1024.

> The chip is designed as a massively parallel processor array, with 992 cores arranged as a grid 32 by 31. Eight additional cores are found along with 12 memory modules of 64 KB SRAM each (for a total of 768 KB). Communication between cores is done via a dual-layer source-synchronous circuit-switched network and a very-small-area packet router (see wormhole routing). The circuit-switched network supports communication between adjacent and distant processors, as resources allow, with each link supporting a maximum rate of 28.5 Gbps. Maximum throughput is 45.5 Gbps per router. Both network types contribute to an array bisection bandwidth of 4.2 Tbps.

That's a good reference.

I had assumed there were 1000 instruction memory SRAMs in there but of course that'd be impossible considering the leakage numbers and there's actually just 12 x 64KB SRAMs shared by all the cores:

     Per core
        640 bytes (128x40-bit) local instruction memory
        512 bytes (256x16-bit) local data memory
    768 KB SRAM on-die
        12 shared SRAM memory modules, 64 KB each

Those local per-core memories are tiny and presumably implemented with registers. But can you really do much with just 128 instructions? The following makes it sound like they might be instruction caches:

> Instructions may come from the local instruction memory or they may be fetched from one of the independent memory module

But then goes onto say:

> the KiloCore's processors do not contain traditional caches.

Hopefully their paper has more info.

The survey that the "World’s First 1,000-Processor Chip" claim is based on (http://vcl.ece.ucdavis.edu/misc/many-core.html) doesn't seem to include any network processors, some of which are technically programmable. But even if it had, they wouldn't have found any of the information required for the table and definitely not a datasheet!

Maybe one can build/run Tensorflow stack on top of this with just customize the tensor sw on each node?

Or is the 16 bits CPU too limited for training operations?

This is kinda wild. It seems like one could implement this in a large FPGA today and reasonably expect to hit a similar clock frequency.

It's sad that OpenCL development seems to have lost its way. I have some OpenCL workloads where a running on a CPU is faster and some where GPUs are much quicker. The beauty of OpenCL is that I can choose between the two with zero code changes.

But CUDA and ML have steamrollered everything!

It's simple to do what you write, multiple languages already do just that. It's the class of problems that is called "embarrassingly parallel" (https://en.wikipedia.org/wiki/Embarrassingly_parallel). However, it turns out that it places some rather serious restrictions on the kind of calculations you can perform this way. In general for two pieces of code to be executed in parallel, they must be independent of each other. I.e for the following

    1: tmpA := A(input)
    2: tmpB := B(tmpA)
    3: tmpC := C(tmpB)
    4: result := D(tmpC)

it's impossible to calculate a line before the previous line has finished. You cannot do this calculation concurrently. On the other hand if it was like this:

    1: resA := A(input)
    2: resB := B(input)
    3: resC := C(input)
    4: resD := D(input)

You can calculate all 4 concurrently and achieve a great speed-up if you have resources to do it. Except if B, C, or D somehow access the result of a previous line. For mathematical notation this is simple, but for general programming languages there are plenty of ways to do this.

We know that an optimal optimization, i.e. finding the fastest way a program can be executed is known uncomputable. Not difficult, not brute forceable, but impossible to compute with traditional computers. The best we can do is identify _some_ concurrency optimizations, but so far it has not been a fruitful adventure for general programming languages. So what we can do is use patterns and constructs that rely on dividing the problems into concurrent pieces and use proper guarding on shared state. This is however notoriously easy to get wrong.

MSVC's C++ Parallel algorithms library gets close [1], and so does Rust's Rayon crate [2]. C# has parallel for loops [3], and apparently so does PowerShell [4]

Though none of these are quite as convenient as I think they could be. I suspect there is a path for Rust to add parallel iteration syntax in the future [5]. There are quite a few steps needed to get there, but the result could look something like this:

    let mut listener = TcpListener::bind("127.0.0.1:8080")?;
    println!("Listening on {}", listener.local_addr()?);
    
    par for stream? in listener.incoming() {
        println!("Accepting from: {}", stream.peer_addr()?);
        io::copy(&stream, &stream)?;
    }

[1]: https://devblogs.microsoft.com/cppblog/using-c17-parallel-al...

[2]: https://docs.rs/rayon/1.5.0/rayon/

[3]: https://dotnettutorials.net/lesson/parallel-for-method-cshar...

[4]: https://devblogs.microsoft.com/powershell/powershell-foreach...

[5]: https://blog.yoshuawuyts.com/parallel-stream/#future-directi...

Rust's Rayon library is pretty good for these use-cases https://github.com/rayon-rs/rayon

    use rayon::prelude::*;
    fn sum_of_squares(input: &[i32]) -> i32 {
        input.par_iter() // <-- just change that!
             .map(|&i| i * i)
             .sum()
    }

I'm not sure this is the best link, but I don't have anything better in mind and it was the first thing that popped up, so...

Anyway, there's a language called ParaSail[0] that's doing stuff in that direction. A little bit back the guy behind it got hired by AdaCore (Ada compiler folks) and they seem to have adopted it and continuing with it. I haven't heard too much about it though.

[0]https://adacore.github.io/ParaSail/

Occam has exactly this. SEQ executes statements sequentially. PAR executes them in parallel, though the language is based on CSP and Cooperative Multitasking, so the idea of it doing 'the right thing' with a PAR is a little different.

Think of Occam programs as big Factorio factories.

https://en.wikipedia.org/wiki/Occam_(programming_language)

Alot the ideas that came out of Occam and CSP have been brought into various coroutine libraries though. Kotlin Coroutines, Goroutines, etc. So its definitely something that is explored.

> Is there any language where you can just go:

Nothing prevents a map function to run in parallel.

There are other things that can happen in parallel, however - the whole instruction flow is reordered in the CPU before it is executed and if you have enough execution ports and enough in-flight instructions to keep them fed, you can accomplish a lot in a single clock cycle before your source code even needs to acknowledge that instructions don't happen exactly in the sequence they are written.

You can get that without a programming language revolution by reformulating the problem a bit.

For instance 3D-API shading languages do just that under the hood, just without the "parallel-for". Instead you tell a 3D-API what code should run per-vertex or per-pixel in (usually) a traditional sequential C-style language, and the GPU driver and hardware care about the parallelization and scheduling.

Yes, SQL. Write your query and it will be magically parallelised without you knowing it.

#pragma omp parallel for

While reading the article i thought of Erlang.

Multiple independent programs sending messages to each other fits well into multiple independent cores sending messages to each other.

In scala (https://docs.scala-lang.org/overviews/parallel-collections/o...):

Serial:

  list.map(_ + 42)

Parallel:

  list.par.map(_ + 42)

In C# (https://docs.microsoft.com/en-us/dotnet/standard/parallel-pr...)

Serial:

  for (int i = 0; i < matARows; i++)

Parallel:

  Parallel.For(0, matARows, i =>

(Function takes a lambda that gets called for each value)

Swift has DispatchQueue.concurrentPerform (https://developer.apple.com/documentation/dispatch/dispatchq...). That is a bit less clear and less flexible (won’t iterate over an array’s values, only over its indexes) but also fairly easy to use.

There probably are many more languages with similar constructs.

Rust is much safer for parallelism and has a lot of safety rules around closures, so this would probably be achievable in Rust without much pain via a clever template library.

The problem is that not all problems are easy to parallelism at the algorithmic level, and for some it may be impossible.

Alongside massively multi core chips I would also love to see more efforts to push the envelope on single threaded performance.

Java has this since 8:

    Arrays.asList("hello", "world").parallelStream()
        .forEach(s -> System.out.println(s));

If you want to calculate stuff, you can

    List<String> suffixed = Arrays.asList("hello", "world").parallelStream()
         .map(s -> s + "_suffix")
         .collect(Collectors.toList())

q has this out the box: https://code.kx.com/q/basics/peach/

It's builtin into prolog's logic and its derivates, alas they are rarely used in the industry :'(

D does this as a library i.e. just add parallel to your foreach aggregate.

As to languages built around this, there are quite a few, e.g. Intel have Data Parallel C++

This architecture is called a manycore processor. There are several programming paradigms that are researched for it, among others actor languages, where the idea is to run parts of your program on different cores that communicate with each other.

https://ne.wikipedia.org/wiki/Manycore_processor

MATLAB has parfor, but that's barely a language..

ADA is quite good at this.

But not every problem (i.e. the Algorithm for the problem) can be divided easily without any coordination (i.e. communication) between the chunks. This overhead can quickly become significant.

There is a reason why desktop software (particularly the computationally intensive) is having a hard time exploiting multi-core benefits (besides the programming language and habits in general).

C# has it: https://docs.microsoft.com/en-us/dotnet/api/system.threading...

Of course when you start doing things in parallel there are automatically a number of caveats and risks.

Dont have many languages already some sort of map implementation that parallelizes pure function calls? Even python has ThreadPoolExecutor.map and ProcessPoolExecutor.map.

It gets difficult once you have side effects and/or synchronization (join) of the results are required.

In python, with numpy, you can use `np.vectorize`, there exists a jax equivalent that can enable accelerators to do your parallel stuff. Cuda exists, but it isn't as simple as `parallel-for`. For trivial for-loops, you can use OMP.

Hardware description languages are parallelized by default.

Pony? https://www.ponylang.io/

Parallax's Spin lets you do this.

>Isn't parallelism complicated and practically dangerous in most mainstream languages?

Yes, but functional languages solve this problem.

(Reminds me of this other chip - built by a "giant" of computing - that was designed to be -very- energy efficient, while having a very large number of very simple cores ...

... ran a special FORTH if I am not mistaken ...

I cannot remember where I found a very nice presentation the gent gave about it ... :/ )

Yeah, thought it was interesting that the subject chip didn't have a heat sink/fan and one supporting? chip on the board behind it did.

It does exist in various forms. One I'm familiar with is Kalray MPPA (massively parallel processor array): https://www.kalrayinc.com/technology/ Their current generation of processors have 80 cores exposed to the userspace (the previous one had 256), and they are mainly used as PCIe acceleration boards for all kinds of applications.

But as another poster said, the difficulty with those kinds of architecture is the memory hierarchy: you don't want all the cores to access the DDR directly, as that would be a massive bottleneck. And this becomes a larger issue as you increase the number of cores.

Those are called GPUs! Or, if you like, GPGPUs.

Readily available in The Cloud(TM) and everything.

https://docs.aws.amazon.com/dlami/latest/devguide/gpu.html

Useful comment upthread :)

- https://news.ycombinator.com/item?id=26143976