BitScope: 3000-core Raspberry Pi cluster computer | Hacker News Reader

BitScope: 3000-core Raspberry Pi cluster computer

raspberrypi.org

79 pointsNicoJuicy49 comments

Another neat use-case for this hardware is scale testing of systems management tooling, eg. provisioning, configuration management. HPC centers are looking at the possibility of having to manage 100k+ nodes in a single cluster.

Additional links:

http://cluster.bitscope.com

http://www.lanl.gov/discover/news-release-archive/2017/Novem...

My favorite form factor to go build a Pi-like Cluster is something like this: http://www.friendlyarm.com/index.php?route=product/product&p....

- Gigabit Ethernet MAC on-board, Pine64's SOPINE and Raspberry Pi Compute Module both require per node networking components to be on the carrier board.

- Basic headers used for connecting to a carrier board

- Carrier board "only" needs an embedded switch and 5v power. However, I've not yet come across an embedded switch yet that has a non-blocking ratio of 1GbE ports to 10/25GbE uplinks.

- Carrier board should probably be mini-ITX or other standard form factor to fit in existing chassis. Form factor and embedded switch options are going to limit the number of nodes per carrier board.

I'm just looking at the friendly arm website at some of the other NanoPi's, which I'd not heard of before, I'm quite tempted by the Gbps ethernet. How do you find the software/OS support, are they pretty reliable, with longish uptimes?

i have some nanopi neo(1)s and they're rock solid - my rule of thumb with arm boards is that support is good enough if it has an armbian _debian_ (vs just armbian ubuntu) release

Cheers I might look at getting one then, as I had problems with my Pi's network connectivity being slow compared to a laptop's using the same ethernet over power line adapter.

no idea about PoE on either board but if i/o is a concern you might want to take a look at the $25 rock64 with gbe and usb3 - i/o on my neo1 maxes out somewhere just below 20mb/s but my rock64 has no problem doing 80mb/s with some ancient spinny disks and some armbian devs have done ~200mb/s with ssds iirc

os support on the rock64 is not quite there yet however but given the features@pricepoint i expect it to catch up

I've used some of their NanoPI M1 and other similar boards by other vendors (Orange PI PC and Zero) with great success, although not in any cluster configuration (mainly SDR and other "normal" stuff). The power/price ratio is generally more favorable compared to the RasPI boards, and there is some good information on the Armbian forums on how to get fine CPU throttling, which would probably be of interest in such a scenario.

It's a neat way to teach people how to program distributed systems at this scale but I don't understand why they wouldn't use the Raspberry Pi Compute Moudle instead of normal Raspberry Pis. Using the Compute Modules would, I'd think, make for a much easier to integrate system, even possibly allowing higher density of processors per unit volume.

Raspberry Pi Compute Modules don't have ethernet built in - that would require quite a non-standard networking system, wouldn't it?

The Broadcom SoC doesn't have Ethernet built in, it's a USB connected Ethernet on the normal Pis. The USB bus from the SoC is brought out onto the SODIMM connector on the Compute Module so one could easily add Ethernet on the carrier board if needed. The Ethernet used on the normal Raspberry Pi 3 is a Microchip/SMSC LAN9514 USB hub/Ethernet part: http://www.microchip.com/wwwproducts/en/LAN9514

There are likely other methods of networking, if one wanted to avoid paying for a USB/Ethernet adapter chip for each Compute Module. Such as running a USB gadget on each Compute Module within Linux and have all the Compute Modules connected as USB devices to a USB SS host. If you used USB SuperSpeed hub chips and did the network switching on a non-Raspi SuperSpeed host within Linux (software defined networking!), this could likely easily get high-ish speed (ie: 100+ Mbps) networking working cost effectively, as the SuperSpeed hubs should allow upstream to run at USB SS rates while downstreams all run at USB HS rates, 7 port USB SS hubs aren't that expensive of chips. Microchip sell the USB5807 7-port USB SS hub chip for about $5, which is only slightly more than the LAN9514, as then you'd only need one hub for every 7 Compute Modules instead of one LAN9514 (or similar) per Compute Module.

http://www.microchip.com/wwwproducts/en/USB5807

Any comments on using Pi Zero as base line module?

  Supposedly only $5 per PCB from  
       https://www.adafruit.com/product/2885


  Power and network can go thru usb. (usbnet)

Like stan_rogers says, you're limited to buying them one at a time from pretty much any retailer. And it's been this way for years, so it doesn't seem like it will change anytime soon.

You can find more on eBay, but then they're usually priced $15-20 each and it would still be difficult to buy them in the hundreds.

You'd need to buy them one at a time.

35U? That’s a full rack. The cluster looks like a 8U server case.

One of the comments mentions that it is a 6U server case. So presumably there are five of them, with 150 rPi3 units in each case.

Correct. 144 nodes + 6 spares in one box.

I bet the IO throughput sucks. Is the ethernet bus still shared with the usb/sd card bus in the rpi3?

yes

Now, to port lisp and C to it.

The interconnect will be the most interesting part of this cluster.

You forgot the star reference. :)

No I didn't, HN mangled the text to italics and I did not spot it. Too late to edit. That's why lisp and C are in italics :) How do you make an asterisk in HN anyway?

Like this * (3 for 1)

Ah, that's the trick. I tried two and that didn't work. Thank you!

waste of (taxpayer) money using the (overpriced) rpi's gimped (no aes, usb3, gbe) soc

the rpi value prop is the community support but that would mean that this cluster is running a 32bit os - so what really is the point of using these instead of something smaller and cheaper or same size and more powerful for the same money

Good that thing national laboratories and research in general don’t have to show profitability then. Because otherwise we’d get nothing done and funded due to the likes of people like you.

i don't even know where to start understanding this remark

i am questioning why they are using immensely popular and commensurately overpriced but yet woefully underspecced components instead of something better and/or cheaper

is it just because 'raspberry pi' makes for a hot title in a press release?

Your question might be answered if you offered an example (or two) of something more capable and/or cheaper. I'm really interested in what you think would be cheaper.

Keep in mind the goal seems to be to build something with a high node-count, rather than just core count so small size is important.

A Nanopi Neo2 is 1/3 the size, similar CPU, has GbE and is 1/2 the cost @$18. [1] I use one for a seedbox/VPN and it is a very nice piece of kit.

1. http://www.friendlyarm.com/index.php?route=product/product&p...

Because someone who understood the available resources and the actual user needs performed an analysis and determined that this would be sufficient. Commodity hardware eliminates the need to spend a year or two engineering a custom board, dealing with vendor BSPs, etc. It also eliminates the FTE that would be required to support all this.

They could also have chosen to use a reference design from another manufacturer but it's basically like using RPi but 100x more expensive. There's a good case to be made that this is the most cost-effective design for what they're trying to accomplish.

They could also just have bought 12 Epyc 64 core Servers to get equivalent performance assuming they are exclusively compute bound. If they are IO bound even just 3 nodes with 100gbit NICs would be enough to outperform the Raspberry Pi cluster.

If they really wanted to develop a competitive cluster they'd need at least a SoC with 4 A72 cores, 10gbit NIC, 8GB RAM, and a local 128GB SSD.

Edit: I misread the article it's not a cluster with 3000 raspberry pies. It's just 3000 cores. 3 Epyc Nodes are faster than this cluster.

It's not necessary about speed, it's about how to write code that runs efficiently, concurrently and/or in parallel on that many nodes.

IIRC, that's kinda how ethernet came to be, ther were working on the computing world of the future at xerox PARK, they had to create clusters to emulate the cpu power that would be available in the coming years. Looking at the current trend, from phones to servers, they go from two cores to I-don't-have-enough-fingers-except-if-I-count-in-binary cores. A 3000 cores raspberypi cluster can be an emulation of the computing environemnt of tomorrow, not in term of raw power, but in term of distributed computing, and lead to unforseen invention as the ubiquitous ethernet.

pretty sure it's just for its press releasibility but ok

Please do tell us about these better specced, better supported off-the-shelf commodity components!

Unless they're planning to fab 100 of these, chances are high that the retail margin on a Pi still leads to one of the cheapest BOM for a one-off project like this

you do understand that 'better supported' in the case of the raspberrypi means that you are running a 32bit os? that's all they officially support - and have no plans to change

do you really think something or better and cheaper is impossible? the rpi is _the most overpriced sbc on the market_ particularly if you're not using it for raspbian (32bit only) and the module ecosystem

nanopi neo2 and rock64 are better and cheaper alternatives

If you think it's overpriced, you're welcome to build your own one. In fact, you're welcome to provide costed alternatives in your own comment.

you can buy at least 2 nanopi neo2's for every rpi3b - they have gbe and a non-gimped soc, and you can fit in at least 2 for every rpi, maybe even 4

there are also more powerful alternatives of the same footprint and at same or slightly cheaper price point (rock64)

Ugh, it has one of those godforsaken Allwinner chips on it. good luck getting 750 of those working at once. The RPis are more expensive up front, but they don't require the researchers to learn Mandarin to get tech support or read the docs, on the outside chance such things are even available.

Engineering teams rarely trifle over price of parts when reliability and confidence are greater concern. Sure, the neo2 is cheaper, and the rock64 is more powerful, but I haven't heard of either. What is the probability of encountering a weird not-before-seen bug on these platforms, vs Pi? Now what's the cost of dealing with that bug, including the risk-cost of having to scrap the entire platform because it's unscalable? It's probably at least $20x750 = $15k.

Pi has at least 10x more community eyes crawling over the whole system than the next three combined. You can't put a price tag on that.

Also consider the ecosystem for these boards. Do you want your researchers dealing with the non-upstreamed support around Allwinner chips? Additionally consider time to solution, this machine is based on BitScope's Blade: http://bitscope.com/product/blade/.

32-bit vs 64-bit is largely inconsequential when the nodes only have 1-2GB of RAM. The ecosystem I'm referring to means you can run a vanilla Linux on a RPi with no extra work. You can google a problem and have a reasonable chance of finding a solution around the RPi and so on.

From what I understand you can see a decent improvement in speed by going 64bit on ARM because they used the crossover to dump a fair bit of legacy braindamage in the instruction set that was preventing them from making certain optimizations.

On the other hand, I look at the 3000 core figure and think that it's roughly on par with high end GPUs. The clock rates aren't terribly different either. The range of applications where this beats out GPU solutions is probably fairly narrow, especially given the terrible IO bottleneck on the RPis.

For comparison, a $7,500 TITAN X has 3072 CUDA cores clocked at 1Ghz. This cluster has 3,000 CPU cores clocked at 1.2Ghz. On the TITAN card all of those cores share the same 12GB of memory with 336.5GB/s of memory bandwidth. On the cluster every 4 cores shares 1GB of memory with (I think) 3.6GB/s bandwidth. Of course communication outside of those 4 cores is restricted to 0.0125GB/s at best.

no it's not inconsequential

for one, why are _researchers_ using largely obsolete technology; for another, many high performance computing tasks perform significantly faster on 64bit (e.g., lmdb)

Performance isn't actually the objective of the Pi cluster; the people using it have a real supercomputer next door. It's a testbed so they can validate programs before transferring them to the expensive supercomputer.

is their real supercomputer 32bit? because if it's not then i'm not sure how they are validating anything

I would imagine going from a 10-node to 100-node system is more overall complicated than going from 32 to 64. Sure the instructions change, but that should basically be all abstracted away by the toolchain. However job management, allocation, data logistics, queues, cache invalidation, bottlenecks, etc, are all key issues that compound non-linearly with scale.

Chemists still use thin-layer chromatography, a technique 100 years old, day-in-and-out, in every lab in the world, even when HPLC and NMR exist. Why? It's cheap, fast, and works well enough.

if we are 'considering the ecosystem' then that means these are running a 32bit os, as mentioned in my original comment

The Raspberry Pi 3 model B uses the Broadcom BCM2837 64-bit A53 ARM processor and has been supported since Fedora 25.

Here's the link to the AArch64 server distribution:

https://dl.fedoraproject.org/pub/fedora-secondary/releases/2...

Having said that I agree the Pi is overdue for a refresh; it needs gigabit ethernet and usb 3 at a minimum but faster interfaces would be great. I think people execute these projects because the Pi is a great reference architecture that can be bought at scale and that has been proven by the large user community.

I'd love it if the storage and Ethernet weren't hanging off of the same USB. Or on USB at all. The USB is a perennial bottleneck on the RPi.

There are a lot of boards that implement it correctly in hardware, but then make a hash out of the driver support with binary blob drivers that are fixed to a particular kernel version and crash from time to time.

It's probably not technically feasible currently, but I'd love to see a board where all of the hardware is open (even the 3D acceleration!) and already mainlined into the kernel so you could just install whatever distro you want on it and available at a price point under $50. I'm really tired of "you need to dump this proprietary binary blob into the graphics chip before the rest of the board can even start to boot." Why is it taking so long to come up with a universal boot solution, something that could be integrated into GRUB so you don't need to program magic offsets into the bootloader to make it work? PC hardware manufacturers more or less solved this problem 30 years ago, and I'm not taking "but the hardware is so specialized that you can't do it" as an answer anymore. The SBC world is absolutely crammed with stovepipes for no good reason.

I may be mistaken, but didn't they say that there wouldn't be more releases (seems insane to me, but apparently their goal is to drive education and where the board is, I guess, is enough)? Hoping that isn't the case.

Reply on news.ycombinator.com