Show HN: Turing Pi 2 – a compact ARM cluster with 32 GB RAM | Hacker News Reader

Show HN: Turing Pi 2 – a compact ARM cluster with 32 GB RAM

turingpi.com

109 pointsnsky-world71 comments

Today we are thrilled to announce the Turing Pi V2. The Turing Pi V2 is s compact cluster in a mini ITX form factor with 4 x cluster nodes, 2x mini PCIe (Gen 2) ports, 2x SATA (Gen 3) ports, and new Raspberry Pi compute modules 4 support.

The Turing Pi 2 cluster is architecturally similar to the AWS Graviton clusters. The CM4 processor uses the ARMv8 architecture. You can build images and applications for AWS instances of Graviton 1 and 2, host cloud native apps locally or at the edge and more.

For the mini PCIe, do you mean they will be PCIe 1x slots, or something like an M.2 connector?

And will it be set up where each board gets one of those slots (e.g. 2 boards get single PCIe, 2 boards get single SATA)?

m2 connector is something we are still experimenting with. The first two nodes are exposed to mini PCI express port, the third node is exposed to 2-ports 6 Gbps SATA controller. You can use the network file system within the cluster to access hard drives from other nodes.

I guess it's possible to adapt the mini PCI express port to M.2 pretty easily too. Having it PCI express lets you do other interesting things though too besides just storage.

Would it be possible to install a modern graphics card on this (e.g. some ati Radeon)? That would require some physical adapter on the one hand, and driver support for armv8 on the other.

> The Turing Pi 2 cluster is architecturally identical to the AWS Graviton clusters.

Graviton2 uses Armv8.2-A, as opposed to the Armv8.0-A on the CM4. Graviton2's cores are "Neoverse" vs CM4's Cortex A73, which is also a significant difference. A lot of code will be portable, but these are significantly different cores.

Probably more accurate to put "architecturally similar".

Noted!

I'd like to know more about the PCIe Gen2. From your comment it sounds like the root port is capable of Gen2x2, or is it 2x Gen2x1 ports? If I wanted to attach a PCIe accelerator card with a standard PCIe connector, is there some kind of adapter cable I would need to buy?

2x Gen2x1

This is very neat.

I’m sorry if I missed it, but have you announced pricing?

We are expecting the price to be cheaper than the V1, but the final price will be announced later this year.

Excuse me, whats the price of V1? Cant find it on your website.

V1 is $189

IT's amazing device!

For now i'm running a few RPi devices and i was dream about such device to put them in for more processing power.

On the page you mention CI/CD applications and we are considering using Raspberry Pi for our CI infra (for the lack of a more powerful option). While there are various possible setups, one fairly common arrangement (especially in a more public-facing services) is to start a throw-away VM for each CI request. This is how our CI service works and you can see it here[1] in action.

And while it would have been nice to use Turing Pi, it does not appear to fit our requirements very well because we need to have high-performance storage attached to each node (where we store/run the VMs). An ideal board for us would be a network switch plus for each module a PCIe (or better yet, NVMe) slot and USB 2.0 ports (at least 2x).

[1] https://ci.cppget.org

Are those USB 3 ports? or USB 2? Do they belong to any particular node?

I spent a bit of time testing the v1 board, and while it isn't without fault, it's the best SBC-based cluster project I've used (haven't personally tested the SOPINE Clusterboard though).

Since the CM4 is 2x-3x faster than the CM3+ in pretty much every aspect[1], I'd be very excited to be able to upgrade my Pi Dramble[2] 4x Raspberry Pi 4 model B cluster to a Turing Pi V2!

[1] https://www.jeffgeerling.com/blog/2020/raspberry-pi-compute-...

[2] http://www.pidramble.com

Thank you. As always, your review of the new CMs is awesome!

Besides academic uses to perhaps teach distributed systems on these are there any real world use cases for these clusters?

- cloud software developers .. (testing/developing with/without internet connections)

This is a great case study of how the Turing Pi helps with CI/CD for ARM infrastructure https://synyx.de/blog/turing-pi-gitlab-ci-ansible/

Besides that the main direction here is edge computing.

Pretty anecdotal use case, but when trying to do some type of neural architecture search for ML models that are ultimately going to be run on the edge, running some performance evaluation on a raspberry pi is a decent proxy. I have seen teams build their own type a raspberry clusters for this to be able to evaluate many models in parallel. It's basically a testing proxy for ARM architecture.

But this thing only has 4 RPI per box so I don't quite understand which problem they are targeting either.

I use mine to host some non-revenue-generating websites (they are super cheap for that purpose), and also for some ARM image builds that I integrate into my Docker/Kubernetes pipeline.

I haven't yet moved to ARM CPUs on AWS for some of my projects, but may be able to more quickly if I can have a local cluster like this for testing that isn't as slow as the CM3+-based one the Turing Pi V1 supported.

i often see people claim raspberry pi cheap but in reality it's not. a cluster like this is probably not cheap and performance wise, it's slower than, say an AMD x86 system.

Main use case might be doing whatever you would do on 10 year old laptop reused to be your home server. Similar performance, similar power usage.

You'd be hard pressed to find an old laptop with 32GB of RAM, and a laptop doesn't provide the same operational conditions experienced by a cluster of 4 computers communicating over a network connection.

Scaling horizontally presents entirely new classes of challenges that aren't experienced while scaling vertically.

Couldn't you buy four 10 year old laptops?

Anyone can buy a brand new raspberry pi 4 with 4GB of RAM for around 50€, which fits on the palm of your hand. You can even get them delivered to your doorstep with a few mouse clicks.

The same can't really be said for 10 year old laptops.

Yes, for the former you have to walk across the room and dig in the closet, optionally ask friends.

ThinkPad W530 fits the bill quite well. Broken screen puts it at around $100-150, $300 gets you fully working one. Less tyhan 4 CM4 modules, and what about Turing Pi 2 case? power supply?

>Scaling horizontally presents entirely new classes of challenges that aren't experienced while scaling vertically.

Pay more for challenges? If you really want to fight with bad scaling just run a VM.

>>Besides academic uses to perhaps teach distributed systems on these are there any real world use cases for these clusters?

A considerable amount of people is already assembling and running clusters of raspberry pis, including hosting companies. A cursory search through eBay or Amazon can provide you with an idea of how many people are already operating these small clusters and spending money on this sort of setup. I'm sure there's a significant demand for a COTS cluster solution that allows people to avoid having to MacGyver their own cluster just to get Docker Swarm or Kubernetes running on a small cluster of raspberry pis.

I know people are building these the questions is for what, CI/CD for ARM like others mentioned is plausible but only if you are targeting the Pi or something very similar just due to how “fragmented” the ARM ecosystem is with its multiple uArchs.

If you are building for mobile then it doesn’t makes sense to run CI/CD on the Pi and if you are targeting your own bespoke edge then likely having dev boards which can represent your edge hardware more accurately is probably going to be a better way.

Running CI/CD for ARMLinux apps is pretty much the only use case I can think of and even that might be limited if you are using extensions.

> I know people are building these the questions is for what, CI/CD for ARM like others mentioned is plausible but only if you are targeting the Pi or something very similar just due to how “fragmented” the ARM ecosystem is with its multiple uArchs.

I don't understand what point you tried to make. These hosting services, and also clients, are quite intentionally targetting the raspberry pi. More specifically, they do target Debian running on Raspberry Pi. I don't understand the confusion. I mean, is it also hard to understand why AWS users pay for ARM instances running Amazon's custom linux distro? In both the Pi and Graviton you can run nginx, nodejs, Python, ssh... So, what's the problem again?

Outside of doing it for the heck of it (which is a valid enough use case , IMHO), what practical use do you see out of this ? To me, it looks like I can do all of the same things mentions (perhaps even waay faster) on a x86 processor in the same size factor.

So your question what's the advantage of cluster comparing to one processor unit?

You could get an off-the-shelf x86 machine in the same form factor and run the cluster nodes in VMs, for instance.

Maybe you could argue that four RPis on a network are more reliable than a single x86 with VMs, but that doesn't seem to be the point of this machine.

That's one of the point too. We have industrial customers who deploy apps in the places with a limited access to the internet, so increased reliability something important to them. Another thing is that they can host apps with different purposes on different nodes in containers. Besides that all the infrastructure could be managed with a tool like Kubernetes and organically work with existing cloud infrastructure. And the last but not least in a cluster like that you can combine general purpose modules with accelerated compute modules.

That's really nice. There are high-end products out there with redundant backplanes (dual or triple) and redundant PSUs (2 or more). To match that level of redundancy it seems you would need 4x Turing Pi, to be able to survive a simultaneous failure of 1x arbitrary PSU and failure of 1x arbitrary backplane (ie. the integrated RTL8370 GigE switch).

The 2x external GigE ports on the Turing Pi helps a lot, in that each cluster can connect simultaneously to 2 external GigE switches, in case one of those things fails.

Could be an interesting exercise to set up and get it all working, in particular K3s + dynamic routing might be a fun challenge.

Theoretically yes, but there seem to be only 2 PCIe (and probably x1 slots). And the interconnect will be quite slower (I suspect) btw. the modules. All those GPIO pins are gonna be wasted and if I am going to commercially deploy containers for customers, I'd rather get a solid industrial Ryzen/Intel fanless solution and call it a day

There is nothing in the schematics about multiple power supplies or battery backups, etc. If you are hooking up all 4 rpis to the same power source , might as well replace it with one x86 processor.

Any thoughts on keeping accurate RTC time on these things at sites with poor internet connectivity, weird latency or restrictions on outbound NTP?

I'm thinking a nice addition would be a small module to plug on top of the 40-pin GPIO connector with a GPS receiver in it and an antenna attached. The attached node could run NTP and pull accurate time from satellites. Throw in 3 of the modules, and NTP can use quorum to keep the time accurate even if one GPS module fails.

I can't think of anything better than a GPS receiver.

About 10 years ago we had a legacy system that was using a weird shaped (think UFO spaceship) FM radio receiver to get time. I never understood why our provider chose that over a simple USB GPS receiver.

PCIe switching would have been really interesting to see, even in a non-concurrent mode.

One also desperately wishes that multi-host network adapters had a low-end market. A 1 x 25Gbe connection shared between 6 hosts would be epic. Or if sharing the connection is too much, just a 4 x 2.5Gbe NIC chip would be acceptably interesting & useful to have. I forget, maybe the gbe is free on the Pi4's chip & this is more of a non-issue.

PCIe switching would be awesome, for sure, but so far too costly. We want the Turing Pi 2 be cost friendly.

PCIe switching sounds nice, but let me tell you that it is a terrible nightmare on every level.

Hardware: these PCIe switches are finicky as all get out; will it link up to your device reliably? Who knows, depends on how the switch and device manufacturers interpreted the PCIe specs.

Software: there is no nice easy standard for connecting host-to-host (which I’m assuming is what one would want on a multinode system Like the one linked). There exists the idea of a PCIe non-transparent bridge, which if I recall correctly, allows CPUs on either side of it to do blind memory reads and writes to each other (hope your IOMMU is set up correctly to protect you from ugliness that could occur here).

Long story short there’s a reason PCIe is a local bus; it was never designed to do anything else. Anything else is largely a proprietary bolt on.

Totally agree

I was definitely thinking of NTB, for host-to-host connectivity. As you say, it doesn't provide a whole lot of capabilities. However, there's been a pretty competent ethernet driver available on Linux since Linux 3.9 released April 2013. There's out of kernel NTRDMA if you want ibverbs like communication that will make really good use of the PCIe bus too.

In addition to mapped memory, there are some other modest capabilities: support for ringing the peer's doorbell (one way), & scratchpad configuration. There's a pretty good 2016 presentation on NTB in Linux here: https://events.static.linuxfound.org/sites/events/files/slid...

Interesting to note, Nvidia has docs for settings up NTB on their Pegasus & Xavier systems: https://docs.nvidia.com/drive/drive_os_5.1.6.1L/nvvib_docs/i...

I for one think more people should be investigating & harnessing the open-source Linux NTB tools. Switches are a bit expensive, but compared to ethernet switches their cost/throughput is great. Alas, yes, the whole ecosystem is woefully undersupported, under-developed. Isn't PCI part of the whole PC thing, this great revolution in computing that allowed interoperability to rise? Such a pity that all the PCI SIG's seemed to fizzle out, that not a lot of people participated. Sure we got some SR-IOV but that feels like as far as we got.

Out of curiosity, does the firmware in the Compute Module allow for PXE boot?

Would be a neat way to handle corruptions of the SD / eMMC. Flick the power off through the Turing Pi's I2C, have one of the other nodes act as a PXE server, flick the power on again to the affected node, and have it pull in a tiny kernel and init-script through PXE that flashes the SD / eMMC with an image from the PXE node.

Forgive me if this is a stupid question; but I see that it has SATA support. Does that mean this can be shoehorned into NAS service?

I have an irrational desire to create a cluster, but I would like to use it for something practical like a NAS. Bad idea?

You can. Why not?

Yes, but I would opt for the newly released HC4 if two drives were fine. If you are looking for four to five drives, I'm sure someone will make a PCB at some point that uses A PCIe connected SATA controller.

Price becomes a real problem though. The Pi foundation carrier board is dirt cheap, everyone else's boards seem to be a lot of money. Probably due to low volume.

Yeah that's what I kinda figured. It's interesting but doesn't make sense from a financial standpoint.

Long term request for the next Turing ( mixed CPU cards together - in a Box! )

+ ARM cards

+ X86 cards

+ RISC-V cards

Are there more detailed characteristics of the ethernet switch?

We will post switch datasheet later

Stupid question, how do these 4 SoCs share the 2x SATA ports? Do you assign them to a specific processor somehow? Are the other machines booting off of PXE boot?

The blog post[0] answers that: peripherals spread between nodes.

[0]https://turingpi.com/turing-pi-2-announcement/

But like, how? Mounting a volume in read/write mode on more than one machine is obviously going to corrupt it

Lots of cool things people were asking for, but I’m a bit sad that they decreased it to 4 slots.

Rack mount case would be nice.

I assume DIN-mounts, more so than standard 19-inch mounts, given the size of the Turing Pi.

The Turing Pi comes in a standard mini ITX form factor

I don’t understand the claim about 32GB RAM. The diagram shows slots for four RPi compute modules, each of which has 1GB RAM.

there are thirty two flavors of CM4, some contain up 8gb of ram.

The new compute modules comes with up to 8 Gb of RAM. Here is a detailed review of new CMs https://www.jeffgeerling.com/blog/2020/raspberry-pi-compute-...

Reading https://turingpi.com/turing-pi-2-announcement/, they have designed a “Raspberry Pi 4 Compute Module (CM4)” with 8GB RAM.

https://www.raspberrypi.org/products/compute-module-4/?varia... also lists it as for sale.

Brings to mind the Slashdot-era refrain "Imagine a Beowulf cluster of these!"

What's the price?

not announced yet.

> The Turing Pi 2 is more functional than V1 and we also expect it to be cheaper to manufacture.

Rack mount form would be nice. Please do it!

Reply on news.ycombinator.com