Let's Build a 28-Core Raspberry Pi Cluster | Hacker News Reader

Let's Build a 28-Core Raspberry Pi Cluster

ikarus.sg

210 pointsColinWright150 comments

Hello, original author here. I saw a bunch of traffic from Hacker News and decided to drop by.

Thanks for sharing it here! I'd be happy to answer any questions.

Fascinating, pardon my ignorance but I’m new to this home brew clustering thing. How is the coordination between the nodes handled for a use case like serving a wordpress blog? Or are we talking about n-1 nodes behind a load balancing master node?

Kubernetes would be my guess... Especially k3s, which has a low footprint is suited for rbpis.

Hey mate. The rig looks awesome. How is the performance output compared to a regular desktop with Ryzen-5/7 or Core-i7 processor in it? Can it run Spark? Apologies if my questions are irrelevant. I'm not sure if Spark/Java works well on the ARM platform yet. Although it'd be foolish on my part to expect too much performance from something like this that costs around $600! It'd be an interesting setup to test parallelism and learn how things work.

Imagine clustering 7 desktops from 2006, same performance.

Hahaha. Yeah, but probably an order of magnitude less power consumption though.

I think the Pi4 would be closer to a 2006 desktop (Core 2 Duo). Pi3 is more like a 2002 or so Desktop, perhaps.

Oh, I didnt even realize it wasnt 4. 4 is on par with 2005 Athlon 64 X2, 3 will be ~half that, somewhere around 2003 Athlon 64.

> How is the performance output compared to a regular desktop

Sorry, but often budget (junk) CPUs are only 10% of the performance of current desktop-class processors.

You can use "openssl speed" to test and compare performance.

When I was experimenting with Intel's Atom netbook chip, it was exactly 10% of a Xeon thread. So it could be used for httpd static content, but not java.

> It'd be an interesting setup to test parallelism and learn how things work.

Long-term, they can be used for DNS, PXE, etc. but most people will not get their money back. If you plan to use docker or k8s, verify they have the right CPU options first.

https://www.openssl.org/docs/man1.1.1/man1/openssl-speed.htm...

https://en.wikipedia.org/wiki/Intel_Atom

Are you sure that it wasn't due to the fact that Intel Atom doesn't have AES instruction set[1]? I think openssl uses it a lot so it would explain this huge difference (since looking at the clocks, AVX/SSEs etc. I would expect at most ~50% difference looking at single core of Xeon vs single core of Atom) unless generations of Xeon and Atom were many years apart. I think newer RbPis CPUs have AES instructions so they could be faster than Atoms in applications that utilise it.

[1] https://en.wikipedia.org/wiki/AES_instruction_set

Thanks man.. I know. I've an old Asus laptop/netbook with an Atom processor. Quad-core I might add. The travesty! It's slow as hell. I bought it because it had an 11 inch screen and I wanted something small. I bought it for AUD $300 back in 2015. I haven't used it more than a month. It's just part of my electronics junk collection. Now a days I'm a lot more careful before buying these things. That's why I've been holding on to the idea of buying a bunch of RPi 4s or Jetson Nanos. It's probably a better idea to get a proper GPU for my Ryzen desktop.

As a general concept, an old low-end laptop can sometimes be usefully repurposed as a low end NAS. It just needs a sufficient amount of ram (depends on the NAS software), and the ability to plug in disks via some means (USB, etc).

It wouldn't be fast, but the battery in it (unless completely dead) even provides some power-outage protection. :)

Very nice work. Super geeky. I would love running a Plan9 cluster on this setup. Any thoughts on trying it out?

I have a similar cluster at home that I mostly use as a media center! One change I made over your setup is using an external NAS for storage. I'm also using k3s for making the whole setup completely portable.

When I need to access my movie library within a deployment, I just mount the folder as NFS volume. It's working quite well and performances are still decent! Maybe something to consider for Leviathan?

Hey there. Thanks for the article!

I wanted to ask you if it is worth it implementing something similar in the cloud to learn kubernetes. The hardware is prohibitively expensive on Brazil, so maybe a cluster of t2.micro on AWS running for the total amount of 750 hours could serve the learning purpose for free

I don't know how it is in Brazil but in Europe and the US offices frequently go out of business that have dozens if not hundreds tiny desktop computers in their bankruptcy auctions. They've got i5 cpu's 4-8gb of ram and 128gb SSD's and go for (much) less than a hundred bucks. Stack 3 of them and you have a kubernetes cluster that you could carry in one hand and I think outperforms the 7 raspberry Pi 3's.

Servethehome has a series of articles about the machines, though they're focusing on the slightly higher end ones.

I think it’s lease companies replacing paid off business PCs. They rent boxes to offices on multi year contracts, collect and recycle them when the term is up.

If you have a PC, then just run it on VM(s) on that. That's what AWS is giving you anyway.

Here's an example: https://kubernetes.io/blog/2019/03/28/running-kubernetes-loc...

Thanks for reading! It's probably worth implementing it in the cloud using the free credits if hardware is prohibitively expensive, at least for the purposes of learning.

If you do intend to self-host some services in the longer term, the cost of running it on the cloud will catch up with the hardware cost in no time at all.

Or you could also checkout lowendbox.com (not affiliated with them). They provide really cheap vps servers.

hey nice one will

> An alternative was to use commercial-grade network switches with 16 ports but that was obviously was out of the question (and budget).

Ebay?

Thanks for the tip. I did consider Ebay, but at that point, I figured it was more trouble than it's worth if I bought them used. Ebay doesn't exactly have the best reputation.

Ahhh. I guess it depends on what you're looking for.

Mellanox cards and related cabling is generally still pretty good (in UK and Australia anyway).

A few years ago I obtained a few 24 port HP Procurve switches (3400cl-24G), with 10GbE (CX4 connectors) on the back.

Those worked pretty well, though turned out to be slightly too noisy for use as desktop switches. ;)

Taking a quick look on US Ebay now, something like this (US$39) might have been workable:

https://www.ebay.com/itm/HP-J9145A-ProCurve-2910al-24G-Ether...

I don't know that seller at all, but ex-enterprise HP Procurve gear should generally be pretty safe. From reputable vendors anyway. :)

For Singapore, I'm not sure. Might have needed to get something from the US shipped over. This one seems like it'd be ~US$80 including shipping, which was probably too much:

https://www.ebay.com/itm/HP-ProCurve-Switch-1800-24G-switch-...

To help with the noise try replacing the fans with high quality ones. I threw some nocturnas into an old managed dell switch and it quieted down immensely.

Just be aware that sometimes they change the pins around from the standard pinout, I blew out one of the fans before discovering this. Fan still works, just lost the rpm monitoring :(

Thanks, very much agree.

I took a naive approach to it before having any clue about electronics, just replacing one of the (3) internal fans with a quieter one of the same voltage. Mostly to see if that would work.

While it functioned, the "fan warning" light was then permanently on (with the replacement fan in).

If I still had the gear and were to do it again, I'd investigate properly and figure out a real solution. :)

Server fans are not cheap, they’re smaller and faster. Look at current and CFM rating. They might look insane from DIY market figures.

I have to admire someone who spends their first paycheck on a pi cluster:

> The total damage my wallet sustained from Kraken was S$626.63 or in $449.73 USD, which was virtually all I had left for the month after subtracting expenses and education loan repayments.

This is the part of the article that stood out to me most.

I definitely don't have the confidence to spend my entire paycheck on a just-for-fun project, especially considering how many projects I planned on starting but never did. Kudos on the author for being this confident and actually going through with everything.

All power to him.

Nobody got successful by playing it safe.

This is cash for joy!

I think one could easily calculate ROI in incoming recruiter messages after reaching first page of HN with personal blog, if nothing else.

What's the actual use case for rpi clusters like this? If the goal is to learn docker or k8s using a realistic setup involving multiple machines, can't it be achieved by using multiple VMs on one physical desktop instead of using the cluster? The 7-node cluster has only 7GB RAM in total, which is not too much to ask on a desktop.

I don't know what people do with rpi clusters and I'm honestly puzzled why people bother building them. If you get a beefy expensive server, I can understand -- you care about performance. If you get a single rpi to serve files & media in a home LAN it makes sense too -- low footprint, low power and low cost. But I don't know what a rpi cluster excels at. Is it just a toy? Or a learning tool to make computer science & networking concepts concrete and tangible? Or just to get a computer system on a shoestring budget, much cheaper than even a single desktop or laptop?

BTW, computer hardware in Singapore is overpriced. $140 for a switch in Singapore vs $73 for a similarly-capable switch on Amazon. https://www.amazon.com/TP-Link-Ethernet-Unmanaged-Rackmount-...

The initial point was to build it just because, there was no real reason behind building it. But at this point, I've found quite a lot of use cases for it, specifically hosting web services for family and friends. In the process though, I've learnt to work under extreme resource/budget constraints and that alone has been invaluable imo.

What storage do you use for the web services, given that SD cards are unreliable? Do you have one disk per node or let them share one disk over the network or something like that?

I’ve been running a 4 node rpi cluster at home for about a year and a half now. Originally it was purely for the fun and interest of standing-up a tangible (kubernetes) cluster (like others have said, VMs just aren’t the same thing...). Now I use it all the time to run things on my home network I don’t want to pollute my real computers with. e.g. IoT services like MQTT and a heap of one-off weekend experiments etc. I also use it as a kind of reverse proxy from the outside internet to whichever computer I need to hit internally - just deploy a new nginx pod and simple reverse proxy config, and traefik and some other microservices I have running will go ahead and create lets encrypt certs and create my dns records for me. Oh and I also have some arduino/hardware sensors hooked up to the GPIO pins on different nodes as well. Taint a node to say it has X sensor attached, and k8s runs the pods that need that sensor on the correct node... need to move things around? Just change the node taints then scale down/up the pods and everything is running again. Although that’s probably not a normal use case (I do a fair bit of electronics hacking in my spare time).

I built a cluster similar to this a while back. I am using Rock64 boards instead of Raspberry Pis. Each has 4 gigs RAM, 4 cores. It also support USB3 and I am using a USB3->SATA adapters to attach an external SSD to each.

I could've done this with VMs (and did before.) It is more of a science project to learn kubernetes, etc.

I'm operating an 8xPi4/8GB cluster here, the 8GB Pi4s make this a bit more interesting, but my use case is this: I use Wireguard to backhaul an AWS EIP address, and use k8s to manage instances of various small services (gitlab, websites, wordpress+web application firewall, etc), simulate AWS Lambda, and just generally have fun. But to your point: No, it certainly isn't something I undertook with pure practicality in mind.

I think Jeff Geerling answers this question succinctly in this video https://youtu.be/kgVz4-SEhbE?t=307 (starting at the 5min mark).

TL;DW - It's fun and you gain the added skills of managing bare metal with physical IO limitations.

The use case is tinkering. I think it’s pretty sweet to be able to build a super low cost cluster like this. Play with docker, k8s etc.

if you only want to play with software, VMs are a lot easier though (and you got to learn the basics of the VMs)

Generally, the point of a cluster like this is to also gain insight into things that involve physical components as well.

It's a different type of desired end result / learning than a purely software approach. :)

This could be very handy for running integration tests of cluster-management software!

> Or a learning tool to make computer science & networking concepts concrete and tangible?

That seems like a perfectly good use case to me?

You see, there is a big emotional difference in unplugging a network cable of a node, and see your cluster keep going and re-routing traffic compared to clicking on a "shut down VM" button.

These small emotional things end up making you more interested, and keeps you going in projects. It's the little things! After all this is just for playing around, if you wanted any performance, a single machine would give you more.

Reminds me of moot "revealing" the "datacenter" that hosted 4chan: https://imgur.com/sBLRuZY

I'm putting the air-quotes because I still sincerely hope that it was a joke

https://news.ycombinator.com/item?id=22767439

> I'm putting the air-quotes because I still sincerely hope that it was a joke

Hey if it worked, I'd consider that impressive, not concerning!

Heh, it can get worse.

>Wiichan it's hosted on a real Wii and it's served throught a shitty Mexican ISP.

https://no.wiichan.uwu.mx/

And photos of the lovely server

https://no.wiichan.uwu.mx/meta/res/51.html

How did that load so fast??

Because that seems to be the mirror. The mexican url is https://wiichan.uwu.mx/meta/res/51.html (proof : https://iplookup.flagfox.net/?ip=201.128.144.213&host=wiicha...) It's really cool tho

The host backing it is a ramnode vps in NL?

Oh, my bad, I thought it was hosted on the Wii.

I'm more impressed he got trips, that's pretty amusing about the USB being as fast as an SSD. Technically SSD's are supposed to be based off SD card tech if I'm not mistaken, issue with a USB being based off the same tech is the USB speeds themselves vs a SSD can go significantly faster due to its connector (SATA or otherwise). Although I can see USB C being 'fast enough' instead.

SSDs indeed use more or less the same type of flash memory as USB sticks and SD cards, but the fundamental difference lies in how the controllers interface with that memory. While modern high-end USB memory sticks and SD cards are able to keep a sustained single write operation going at pretty high speeds, each write operation must be finalized with an update to the media's block allocation map, which is the point where the media experiences a form of throttling. Consequently this results in multiple small write operations - each of which need the same finalization procedure - bringing the performance of the stick/card to a complete crawl, with even read operations being blocked.

>I'm more impressed he got trips

There's probably admin tools for that.

I wondered about this, but sometimes people say it cause the previous post they see is 999 so they post it, sometimes you get lucky as heck haha. I doubt they'd screw with the admin tools just to do it.

On big boards there's so many posts per hour that it's impossible to really predict this way, it's very much luck.

I remember people used to run scripts to be sure to get the big gets (6+ last digits identical) but I suppose such methods don't work anymore with captcha and other anti spam measures.

Pass users don't have CAPTCHA though.

As someone not that familiar with 4chan, what are trips?

When the post number ends with 3 of the same number. In this case 000. Theres also dubs and quads.

So I was toying with the idea of something similar. I'm a programmer and not a sysadmin, but I've dabbled with Docker and had a specific use-case.

But so far I've not come further than this: how to deal with the host OS?

Like, I don't want to log into N computers and manually run "sudo apt update" etc. I want them to run the same OS but with slightly different configurations.

Network booting via PXE seems to be the best alternative, but not sure how to prepare the images for distribution.

Anyone got any pointers on how this is best done?

edit: I should mention I just found http://cobbler.github.io/ but haven't had time to investigate it. A viable choice?

I have a series that explains how I use Ansible to provision and manage a pi cluster: https://www.jeffgeerling.com/blog/2020/raspberry-pi-cluster-...

And in addition, I also run another cluster that has all its code on GitHub and runs this site/wiki: https://www.pidramble.com/

I am partial to the iPXE boot idea, you may find https://netboot.xyz useful.

As for how to prepare the images, you need a ramfs image with the root filesystem as a cpio file and a kernel as a vmlinuz file. You then serve these in a basic HTTP file server that your iPXE config points to.

Generating the vmlinuz file is pretty easy given a kernel. The cpio file is trickier. netboot.xyz recommends just grabbing a standard distribution. Otherwise this reduces to the problem of creating your own.

It seems to be primarily for initially provisioning servers, and doesn't really seem to be targeted at their upkeep. There is Rundeck but it's far too resource heavy to be useful for rPi.

Shameless plug: After a lot of research in this, I came to the conclusion that I'd just have to make it myself, hence Covey (https://github.com/chabad360/covey) a project that's written in Go, and currently in an early alpha stage of development, but it works. More details are in the readme, feel free to email me (its on my GH profile) if you want more details than the readme provides.

EDIT: seems like your looking for something that's primarily for provisioning anyways. Whoops, I seem to have missed the middle paragraph...

EDIT 2: PXE booting may not be the best option for a relatively low ram machine like the Pi (unless you have the storage somewhere else, but that's its own can of worms), even the minimal image required to run k3s would be at least 400mb (I think I'm underestimating) there are ways to deal with that (using a compressed squashfs). But none will beat running directly on the SD card. I'd suggest following the sibling comment, and using overlayfs on top of the main fs to ensure that the image doesn't get tarnished.

If the concern is initial provisioning I handle that by using Ubuntu server for my RPi cluster. Ubuntu server for the RPi supports using cloud-init for initial configuration of the server.

In my case I only have 2 different configurations, controller node and worker node. When deploying a new worker node I use a single digit line count shell script to produce the cloud-init files for that node from my template. Basically the only changes between worker nodes are the host name and static IP assigned. I then copy the resulting files to the FAT partition of the boot sdcard.

Someday I might replace this with network booting and something like ansible but for now this is an easy solution and handles 99% of my particular needs.

If you plan to boot from sd cards (most common configuration), 95 percent of work will typically be the image for the first machine. You can then typically just flash the same image to however many pi sd cards you want. Beyond that, shell script/ansible/whatever can get the job done at boot time for whatever per node setup you might need - there’s tons of ways to accomplish this.

The "Self Hosted" list on awesome-baremetal would probably be interesting to look over for this:

https://github.com/alexellis/awesome-baremetal#self-hosted

I used Ansible for the cluster in the article, PXE boot seems great but the learning curve is steep enough for me to put it off for the moment.

For the common base image, I just build a normal Debian image, customize it (chroot) and flash it directly. For customization, maintenance and operations you want some sort of CM (Ansible, Salt, Chef or Puppet). There is also Packer, but it's not that easy to use for bare-metal images for some reason.

You could also do something like userdata where you have a script or playbook that runs on first boot and performs additional setup.

I've been meaning to try out PXE booting but am yet to.

If you can, I'd suggest using unified kernel images, they get rid of the "whole PXE setup", you just send one file and you've booted.

We used iTerm on OS X to open ssh sessions to each node in a cluster then broadcast identical sets of command lines to each cluster in a set of panes in a team I worked on a couple years ago.

Full disclosure: I'm a customer of RackN and have used it to provision several thousand physical machines and counting.

You can get a free license for RackN's digital rebar and use their edgelab.digital [1] project to pxe boot and build a four node raspberry pi k3s cluster in a few minutes.

[1] https://github.com/digitalrebar/edgelab

Last time i checked ipxe was still not supported on pi boards that I think have a very convoluted boot process. Maybe someone with more time and passion than me can eventually figure it out =)

Ansible works great on PIs

Yeah, if you're concern is managing a lot of machines, Salt, Puppet, Ansible, or Chef are the right tool; this isn't really a Docker thing. The PXE boot idea is interesting, and you can even do that diskless.

Thank you all for your input! I was certain there was some solution to this, as it's hardly a unique problem.

Hello! I did it like this: https://www.instagram.com/p/CC76bkADL-c/ It costed about 3k euro for me.

And this one special for k8s: https://www.instagram.com/p/CAqtrRBpNRH/

Is the design files available for the rack mounted version (in particular the clip/holder parts).

Been quite keen to build something similar (albeit less nodes) for a little while. I find the homebrew blade style cluster amusing for some reason.

"I started noticing the performance bottlenecks started when I tried to run Wordpress on it, where a single page load of a newly installed Wordpress blog takes about 10 seconds!"

10 seconds is probably not bad for an uncached wordpress page.

10 seconds is terrible, a WordPress page should render in under 2 seconds.

A slightly cheaper option would be to get a bunch of Pi Zero W devices. They're $10 + SDcards. They would rely on WiFi which might be acceptable if the use case allows having a few dozen radios chattering next to each other. If not, then a $5 Pi Zero + $5-10 ethernet adapter might work.

Since the goal isn't to achieve maximum performance, the slow CPU & RAM (comparable to a 20th century desktop) might not be an issue. And the lower price would allow more devices so you could test more complex configurations and server management options.

Zeros are borderline unusable for actual compute due to having a single core and only half a gig or RAM. They make for great beefy microcontrollers and single-purpose boards, but are not the kind of thing where you want to run various network services.

(I have a cluster running Pi 2s, which have 4 cores apiece, but are still too short on RAM)

I actually have had a cluster of Raspberry Pi 1Bs before this current cluster. Though the specs are very much similar to a Zero W, the performance wasn't great.

I agree that it would cost much less, but I would also get much less utility out of it. At this point, the cluster is doing quite a lot, even the site itself is hosted on it.

My point here is that having room for future utility at not much additional cost, would be best.

Zero/Zero W at $5/$10 are promotional price, for first unit per person only. For second unit and after you have to order case kits or other high(positive) margin options.

How timely... just today I received my CloverPi which I backed on Kickstarter (no other affiliation), a “back panel to enable 4 RaspberryPis to be clustered”.

Finally I can earn my “My other computer is a Beowulf Cluster” bumper sticker from what feels like aeons ago (courtesy of the now-defunct ThinkGeek.com).

> now-defunct ThinkGeek.com

Noooo!! They had the best toys.

Unfortunately their shipping to Canada made everything double the price for me...

Am I understanding that he is storing web data on the SD cards of the Pis? Is that normal? SD storage is notoriously unreliable, especially in write-heavy applications.

People who complain about SD card "reliability" is just writing to it way too often.

Longevity of NAND flash is measured in TBW(total bytes written, in terabytes) but they have 4MB or so minimum size for write access, so people easily end up writing effectively gigabytes over weeks or months thinking it's just kilobytes averaged and use up all its life in just few months to years.

You don't need "better quality SD cards", you stop issuing writes. As for why Raspbian isn't built that way so you don't have to be Linux storage experts just to run it correctly, I don't know.

That's very interesting and explains a number of things. 4MB writes seems pretty high though as a minimum write block size, I've googled but can't find details among the sea of pages trying to sell me SD cards, won't you by chance have a reference? Thanks!

Google search for "microsd erase block size"[0]

From a forum post in raspberrypi.org[1]:

> My 16 GB SanDisk Extreme Pro 45 MB/s UHS 1 card reports an erase block size of 4 MB.

> My 8 GB Transcend SDHC Class 6 150x card reports an erase block size of 4 MB.

> My 2 GB Transcend SD 150x card reports an erase block size of 8 kB.

From another post from StackExchange.com[2]:

> the data cannot be overwritten without being erased first and an erase block is the smallest unit that a NAND flash storage can erase. The erase block size is typically between 128 kB and 2 MB.

According to a Micron Technical Note from 2006[3]:

> The NAND Flash device discussed in this technical note is based on a 2Gb asynchronous SLC device and its parameters (unless otherwise noted).

> The NAND Flash array is grouped into a series of blocks, which are the smallest erasable entities in a NAND Flash device.

> A NAND Flash block is 128KB. Erasing a block sets all bits to 1 (and all bytes to FFh). Programming is necessary to change erased bits from 1 to 0. The smallest entity that can be programmed is a byte.

(Note: 2 gigabits = 256 MB)

In a more modern TLC 3D NAND chip from 2016[4]:

> Block Size 27,888K bytes

or 27.888MB

---

[0]: https://www.google.com/search?q=microsd+erase+block+size

[1]: https://www.raspberrypi.org/forums/viewtopic.php?t=11258

[2]: https://raspberrypi.stackexchange.com/questions/32884/

[3]: https://www.micron.com/-/media/client/global/documents/produ...

[4]: https://www.micron.com/-/media/client/global/documents/produ...

Thank you, very useful information. Makes complete sense for a media type initially designed to power cameras and audio and video recorders.

"Endurance" microSD cards, normally sold for use in dashcams, are much more reliable in the Pi.

However I switched to netbooting my Pis, no local storage on the Pi at all.

This does require you to have a netboot server already, of course.

Indeed, though at the point of building this cluster, "Endurance" microSD cards were not in the market yet, hence I didn't go for it.

It’s also tricky to find an SD card with good performance. I’ve been using a spare m.2 SSD drive inside of a USB enclosure with my Pi and the IO performance has been great.

Brand SD cards are quite reliable. If you configure your SW to avoid sources of needless constant write churn (eg. like postgresql stats collector), and use good power supply, they will last years.

You're absolutely right. I've running these SD cards for about 2 years now. I use log2ram to stream logs to a ramdisk and then sync that at regular intervals to disk, to extend the life of the SD cards.

Nope, the site has its persistent storage on an SSD shared via NFS across the cluster.

"This build is for you if you:"

ahem, for 600 dollars I get a fine computer which can host all the kubernets and docker I ever want in a nice formfactor...

I mean, it's cool and all, but effectively utterly useless except for the blinking lights.

Depends what you're going for. I did a pi zero cluster a couple years back and it was an absolutely fantastic learning experience for me. There was something about it being tangible that helped keep me interested and force me to actually get it running. Also, the memory limitations, for example, made for a cool problem in setting up Hadoop on it.

It can totally be worth it to do something suboptimal, if it leads to a better learning path :)

Very cool work! I wonder the same, for a 'cluster' I'm expecting performance and large memory for example for scientific computing. Other than that I couldn't think of any other use cases. If anyone could enlighten me? Cheers

I made a cluster of 9 pi's for a work project, not for the computing power but for the segmentation. I had 8 pieces of equipment that needed to be monitored while undergoing various physical tests. A simple app was developed that would run on each piece being tested and communicate over the ethernet port each millisecond back to a pi, which then forwarded the on/off status to a central pc connected to the 9th pi acting as a wifi access point. This computer served as the monitor/display and logger of the system so we could see at a glance if everything was green and reporting or not.

It was important that each piece be physically isolated so that we knew the whole path was operational and nothing was short circuiting via something else's network connection. You could have accomplished something similar with VLANs and some routes, but this allowed for easier troubleshooting and to explain to the powers that be how things were guaranteed isolated because they could see it. Plus it looked cool.

They built a multi-thousand pi cluster to test the SMP and configuration stuff before running it on a super computer.

Virtualising thousands of CPUs on a single machine is still not trivial.

Never mind simulating real world network issues.

https://www.bitscope.com/blog/FM/?p=GF19A

yes, if you want to optimize a worst-case MPI-cluster, then a Pi (4) might be optimal for you (because sadly, 4 measly ARM cores with 100MBit/s is a some magnitudes removed from 100 cores and 100GBit/s Infiniband). But then you can also use a stack of old desktops, which is cheaper and you can just throw in a standard image and everything (including CUDA and MKL) can work.

> But then you can also use a stack of old desktops, which is cheaper

If electricity and cooling is free.

And cost effective redundancy.

There's a new carrier board with integrated switching for Pi clustering: https://turingpi.com/

I wonder if there is something to be said for not using Ethernet in something like this? Perhaps it is possible to make a kind of IP over GPIO interface, or attach them all directly to a USB bus? It just seems to me like RasPi -> USB -> Ethernet (for less than 1m) -> USB -> RasPi has a layer or two more than necessary (and I believe the on-board ethernets on RasPi are USB interfaces to the chip too). Could be an interesting project...

It would be educational.

I'd say the most interesting way to approach this at the moment would be to use RasPi 4s and link them via the PCI express bus.

None of this stuff is more than educational, though. I admire this build because it's almost a rite of passage for nerds to build a cluster, but when you're all done and the cool wears off you've basically got a pile of obsolete hardware for which you need to find a home.

People wanting to torture themselves with k8s at home ... I still dont understand why.

I actually have a k3s cluster running my blog/a few services. It just works, really, and so far it has been zero maintenance, zero downtime, apart from the large upfront investment (one weekend). Most importantly it was a great learning experience.

I'd want to - or rather, the idea of managing a series of applications in a distributed fashion is interesting to me.

HOWEVER, i like to write distributed software that i want "uninformed" users to run. Aka not run on nice, automated k8 deployments - but on dirty, random computers.

My current idea is to run this software on all my personal PCs, but also a couple RaspPis over wifi in random locations.

Slow, poor connectivity machines sounds like a decent and predictably bad user environment.

Imagine that the jobs you are interested in want to see some k8s experience. Either you teach it to yourself at home or you justify implementing it at your current job (resume driven development basically).

I have played around and learned the basics of docker etc without never ever wanting to run my home stuff on it. Learning and experimenting is not the same as using it as the primary tool to run your 'production' services at home.

If you want to get more performance out of a cluster like this you can look at my distributed HTTP app. server + JSON database solution: https://github.com/tinspin/rupy

It's actually usable for live services on RPi 2 and on RPi 4 it beats Intel on float ops./joules at 100% utilization.

And if you use multicast UDP you can make these behave cooperatively to scale horizontally beyond what's possible with a vertical solution at a fraction of the purchase/maintenance cost.

I think we are peaking at 2 Gflops/watt for CPU, so if you need silence (passive cooling) and small size this is as good as it will get, probably ever: http://talk.binarytask.com/task?id=8015986770003767235

I'm building a RPi 4 cluster with these heatsinks.

Neat setup. An alternative would perhaps be a Pine64 Clusterboard with 7 SOPINE modules: https://store.pine64.org/product/clusterboard-with-7-sopine-...

I cannot recommend doing business with pine64, their customer service is nonexistent. If you cancel your order, or need to get ahold of someone @ sales@pine64.org for any exception, in my experience you may as well shout at the nearest wall.

- Someone out $165 spent on a cancelled pinephone order, no phone and no refund, complete radio silence.

> - Someone out $165 spent on a cancelled pinephone order, no phone and no refund, complete radio silence.

Hopefully you've asked your bank/creditcard to reverse the charge?

Up until hours ago my pending PayPal claim seemed lost in a different form of silent purgatory, but the refund appears to have finally been authorized as of tonight.

I ordered the phone on 5/12, cancelled the order and filed a PayPal "item not received" case on 6/26, escalated the case to a formal PayPal claim on 7/16 (under pressure from PayPal's "close or escalate" automated ultimatum), and as of now it appears the claim finally got refunded on 8/12.

Not a single email response has ever been received from numerous attempts to contact sales@pine64.org throughout this total waste of time and frustration. They ignored requests for updates before I cancelled, seemingly ignored the cancellation, filed an "item shipped" update in the PayPal case on 7/8 despite my cancelling the order on 6/26.

YMMV

That's very unfortunate. I own 3 Pine64 products myself, my experience with their customer support has been great so far.

I've been a bit curious about this and other multi-board CoM carrier boards like Turing Pi[0]. Have you used it and what's your experience?

Like the author mentions, pre-4 Raspberry Pis gives sad throughput due to the shared bus; Rpi 4 would be much more suitable. There's also Odroid C4, Rockpi 4, Nanopi M4V2.

[0]: https://www.jeffgeerling.com/blog/2020/raspberry-pi-cluster-...

No practical experience with any of these "cluster boards" yet, I'm afraid.

Be careful of that. Those Clusterboards seem like a nifty idea, but looking through their forum they seem to have a tonne of problems:. :(

http://forum.pine64.org/forumdisplay.php?fid=91

That being said, I haven't personally used one. (But might get around to it one day out of curiosity.)

> The maximum number of ports on a consumer-grade network switch was 8, enough for 7 nodes and a cable to the router.

Are there laws in Singapore that delineate between consumer and business grade networking equipment based on port numbers or something?

Here in Yankey Land you can pickup a consumer grade 16-port unmanaged gigabit switch for $50 at the local Wally World. The going rate on switch ports is $3-4 per port.

It looks like it was more a question of price in their market. In the post he mentions a 16 port switch but it's 10x the price of an 8 port. I bet it's a question of there being fewer people in the Singapore market looking for an >8 port switch on the low end of the market.

The switches he references in his post are from two entirely different product verticals. One is from the unmanaged vertical, the other is from the managed PoE vertical. Not only is he ignoring that a 16 port switch exists in the unmanaged vertical, he's ignored an entire vertical of managed switches without PoE.

TP-Link makes a $40 unmanaged 16-port gigabit switch and a $80 managed 16-port gigabit switch.

He's created a False dilemma/dichotomy by pretending that other options don't exist. This is like saying "I'd like a Toyota automobile larger than the Yaris but I can't afford the Lexus LS".

It's entirely possible that those options weren't readily available in the local store he's using. I'm not familiar with Singapore in particular but a lot of Asian countries don't have the huge electronics box stores that are common in the US and Europe. I've seen quite a few videos where the best electronic source the person had was what would be a small sized store in a US mall (20 feet wide, one isle with a counter running down one side for example). Online shopping does exist but shipping times and cost can also be pretty large.

> I'm not familiar with Singapore

Singapore is a very small (~250 sq mile) country, so it would take less than 30 minutes to drive across.

It's very wealthy.

It has, almost dead center in the middle, an electronics mall called Sim Lim Tower that's like 10-20 stories tall.

They most certainly have options.

And you forgot the superb food and it's variety.

I hope you're not referring to durian. :)

I was just pointing out characteristics that would lend themselves toward sourcing a network switch (e.g. accessible, wealthy, giant freaking electronics malls).

Seems down, here's a google cache of it https://webcache.googleusercontent.com/search?q=cache:lIePhG...

Thanks, I wonder if the site was hosted on the Pi Cluster?

Yes the site is indeed hosted on the cluster.

Seems to be down again now. Guess it'll be spotty until traffic settles down.

Maybe the traffic will provide an interesting exercise in optimisation?

eg move from wordpress to static content or something, maybe add some CDN pieces, then whatever else turns out to be needed?

It's back up now.

Nice! I tried this out a while back as well, though I did nothing nearly as snazzy.

Southampton had instructions here, I believe it was: http://www.southampton.ac.uk/~sjc/raspberrypi/

(Maybe still useful if you want to do without docker?)

Personally I would not use pi's for this. Massive waste of money, the experience is worth it but other than that..

What would you use then?

600$ buys you a considerable amount of time in a cloud vendor. For example you can have a 1vCPU instance in Hetzner for 3€/month, which means 600$ can buy you about a year's worth of 20 instances running 24/7.

If you don't need 99.9999 availability of your cluster and are only interested in firing up the cluster from time to time then those same 600$ can buy you a few years of cluster time

The 3€/month is a single thread (not even a core), and it's not even dedicated. The lowest dedicated instance is 2 vCPU (1 core) for 23€/month with 8GB or RAM. That's 276€/year.

8 of those instances is equivalent to an 8 core (16 thread) chip with 64GB of memory. Using, e.g., Supermicro AMD EPYC 3251 + parts, you could build an entire server for less than 1/2 that price. You could have twice the compute for the same price the first year. Servers easily last 3 years minimum, so Hetzner has a 6x markup at the low end.

Admittedly, building your own server doesn't include transit, but transit is cheap. Assuming you have a fast home connection (a big if for many people), the hard part is access to static IP addresses. There are dedicated services that lease IPs over a VPN tunnel, or you can setup your own VPN into, e.g., Hetzner or Vultr using a handful of the cheapest instances. (Not sure about Hetzner, but Vultr let's you lease and attach 2 additional IPs to the same instance for $2/month/IPv4 address.)

Now, the above setup is probably far too much trouble for most people, and it has many downsides[1]. But the point is that if we're just talking about CPU time, not even Hetzner or some other cheap VPS provider can come close to the cost of running your own equipment.

[1] Power and cooling are two downsides, but the 3251 has a TDP of 50W. Granted, it doesn't have the highest clock speeds, but there's a reason cloud providers don't normally advertise Ghz. Anyhow, rather than build 2x 3251's you could build a single 16 core rig using a higher-clocked and newer EPYC or Xeon for about the same money (definitely at or under 2200€ all-in).

> 8 of those instances is equivalent to an 8 core (16 thread) chip with 64GB of memory. Using, e.g., Supermicro AMD EPYC 3251 + parts, you could build an entire server for less than 1/2 that price.

Your choice of comparisons suggest you completely missed the whole point. If the author wanted a desktop I'm sure he would buy one for 600$ instead of spending that money to put together an underperforming cluster that's entirely unusable as a desktop.

If instead you are interested in gaining experience managing a cluster that runs distributed systems then yes you would do something like putting together a raspberry pi cluster.

Yet, if you seek to have the same experience (or arguably far better) without spendig a hefty sum as capex and having access to more computational resources for a negligible opex then cloud providers such as Hetzner are clearly a far better option.

That one non dedicated 3€/month instance is still faster than two pee boards combined.

Maybe used micro desktops?

https://www.servethehome.com/introducing-project-tinyminimic...

I used 3 dells 9020m 3x quad core i5, 3x 16GB of ram, 2x128+1x256 ssd.

All in that was around 450 eur over a year ago, all bought in EU.

All three running linux with no load consume around 30W.

450eur total, or 450 a piece?

I think the author mentioned only spending $449.73USD (i can't verify because the sites down), which seems cheaper than what you mentioned.

I'm also struggling to find exactly what you're talking about. I'm seeing 9020's, but no M, do they actively sell it still?

I'm not nitpicking your comment - i too am interested in throwing a couple Pis in a closet for some distributed systems projects i'm working on. I expect them to be slow, low bandwidth, but simple over wifi/etc. I don't want to pay more than i need for closet computers, though, so i'm interested. :)

~450 eur in total, I bought all of them used. But most of the price came from maxed out ram and quad core cpus.

For something cheaper, I can find in my country lenovo tiny m72, for something like 40 eur. No SSD/Ram, you can configure it as you wish (and still will be cheaper then RPi, even after adding like 2GB of ddr3 - it's something like 4EUR per stick, there are 2 slots in each machine).

That is with Intel Core i3-2120T, so would expect much more performance then even RPi4

Edit: I see now that the same seller has them for ~72eur with 120GB of ssd and 4GB of ram if someone wants something ready to use.

Wow, its wonderful.

Until Raspberry Pi has native hardware crypto instructions (AES) it’s useless for real work.

I think that there are a lot of use cases of Pis for which online cloud providers might be a better option.

Where does it become a better idea to just rent a few instances in Azure/EC2/Google Cloud?

Reply on news.ycombinator.com