Turing Pi: Kubernetes Cluster on Your Desk | Hacker News Reader

Turing Pi: Kubernetes Cluster on Your Desk

turingpi.com

130 pointsjcamou59 comments

While Raspberry Pi's are awesome, and the power consumption is nothing to scoff at (when considering a cluster), you can accomplish this same thing for a lot less, and have quite a lot more compute power, by purchasing a used server or even something like an AMD 3600X...

A single 3600X will grossly outperform this cluster (and cost less) with less headaches (you don't have N physical machines) by using KVM to deploy a few virtual machines and using Kubernetes to orchestrate and allocate within those VMs. You'll also have a lot less latency between nodes running in VMs on the same physical host.

Another thing that unfortunately sucks about Raspberry Pis (less with Pi 4, but still mostly applies) is really shitty I/O performance...

I spent a large amount of time over the past summer and fall trying out various ideas to have a "cluster" at home that was both practical and useful. While, the PIs were nice, they never really amounted to much more than a demo. Latency and I/O become real problems for a lot of useful interconnected services and applications.

Honestly, if Ryzen 3000 hadn't come out, for cheaper cluster builds (~300-400) I still think Pis would be a solid choice but... Ryzen 3000 is just so fucking fast with a lot of cores, it's truly hard to beat.

Addendum: to touch on used servers, yes your power bill will go way up, no joke, but for some applications like large storage arrays-- it's hands down the cheapest/easiest route. Search by case, not by processor, it sounds weird but the case is likely the most valuable part of the old server (like ones with 20+ SAS2 slots for $500) or PCI-E slots that GPUs can fit into.

you mean using a ryzen with VMs (cluster on one ryzen) I assume? bc cost of ryzan as a node would be pretty pricey compared to Pi 4s.

Yes, a single ryzen node running multiple VMs.

So, you don't see a difference between clustering on physically separated nodes and VM?

Not in this case, no. The physically separate nodes are so low power and still share significant points of failure that it doesn't matter.

I see a difference, but it's not in favor of PIs because they're physically separated.

I think it's far more realistic to choose a hypervisor (KVM, raise the roof) install it and configure it, run N virtual machines on a single machine, provisioning and slicing the hardware, setting up and managing the network between them, having disks run in a raid so the VMs don't starve for I/O and data is replicated, and then running kubernetes (or whatever personal hell you want) within your "virtual cluster." That's the whole "running a cluster" part all of these Pi Kubernetes things miss because they're running on consumer grade hardware.

Your hosting provider is surely not using thousands of small under-powered machines running bare metal installs of your applications.

Want to find out how resilient your cluster is? Just fucking kill one of the VMs. Boom. Instant feedback. Now try that with a PI, you'll have to unplug it, and then wait five minutes for it to boot. It becomes tedious fast.

Clustering and distributed systems are extremely complex and difficult... just having physical machines doesn't really even begin to scratch that problems you'll face, just my two cents.

The other part of 'on your desk' is hearing damage.

Server hardware vendors have traditionally not given two shits about their servers being north of 90 decibels, and I'm pretty sure I've witnessed a few that were pushing 100.

That Raspberry Pi is probably going to absorb more noise than it makes.

Is there any reason you need a kubernetes cluster literally on your desk?

Isn't the whole point of using these layers of abstraction over hardware that you pay someone else to manage it?

The only times I can imagine needing hardware on my desk are when latency is super important or when I constantly need to manipulate the hardware (change hardware, play in bios, etc). In either case, I would not use k8s to run my software.

It's fun to play around with and learn on. If you can build a cluster from scratch with Pi hardware, you get a lot of knowledge of how things work under the hood for a real cluster.

Because at the end of the day you are a mammal and pretending you don't have visceral reactions to sensory input is impoverishing your options, and honestly, robbing you of motivational tools.

Information radiators make no sense at all if you view them from a purely objective standpoint. Wouldn't it be faster to just open the web page on your computer? They only make sense because of the way humans interact with the world (and I suspect in particular, the doorway effect).

The last demo I saw for Pi clustering, the guy fiddled with the blink rate and color of an LED on the motherboards. Sounds boring, as a demo it summed up a whole lot of crap into a simple visual.

> Isn't the whole point of using these layers of abstraction over hardware that you pay someone else to manage it?

That person is also needed in the future, and that person needs to start learning somewhere. A good start is messing around with it on a few Raspberry Pis

You could also go down the path of buying one or more Dell Optiplex SFF or Micros (9020s or better). 8 threads and 32GB of RAM (and better) per box and not a lot bigger than Pi in a case.

I managed a few HP C7000 blade systems once in an inadequate environment. They have jet like fans and at the time a firmware bug that meant the fans had two levels: 79% and 100%. Compared to after the firmware was fixed it stayed around 30% at full load and was bearable at least

Oh, man...reminds me of the IBM BladeCenters. Maybe they had 2 fan settings...90% and 100% of a 737 at takeoff. They had an extra cost "office kit" that included baffles and other mods to (hahahahaha) quiet them down enough for an office environment. Maybe knocked it down from a 737 to a Gulfstream G4. Nice kit if buried in a datacenter someplace.

As a project, I like the idea of having the nodes be separate, because endless layers of virtual things makes it harder to grop what's truly going on, IMO.

Also, gotta factor in electricity hosts.

Yes, but this looks super cool on your desk

I can't agree more.

I have had the same setup for over 10 years, a PC in the garage running Ubuntu with lots of ram. I just upgrade it every so often with last-gen CPU and motherboard and swap-in the RAID controller.

I script all of my dev environments, if I need a K3S cluster it's literally 2 minutes (Bash, Python and some Ansible).

Want a set of RHEL machines? No problem, it's another script and 30 seconds.

For training you can't beat it. If I have to get out the "big guns" I have an old Dell tower server with 128GB of memory that I got cheap and has 12/24 threads. Sure it's a power-hog but I use it surgically.

Pis are great and you get a nice, limited blast-radius but the SD cards go wrong, they are slow at single-thread and they don't really have enough memory (Pi4 4GB being an exception). You can't beat x64 for compatibility either.

There's something amusingly cyclical about a blade server architecture for Kubernetes. The tech comes out of a whole movement towards combining commodity machines using clever software instead of buying specialist hardware, but then adds the specialist hardware back in.

Some deeper integration between Kubernetes and the hardware (acceleration/offload ASICs maybe), branding of k8s + this hardware as a unified product, and this would literally just be a mainframe. Which is not a terrible idea! Maybe Kubernetes is the mainframe operating system of the future.

Think of it as a way of using mixed hardware: mainframe hardware in conjunction with commodity hardware. It's extremely useful in that context.

Some mumblings have been heard accusing us all of trying to reimplement mainframes, badly.

For a long time I have been watching the ebb and flow between peer to peer and client server and it’s gotten quite a bit fuzzy lately. I suppose if you treat cloud providers as a large amorphous server, it sort of still fits the mold.

There is also the https://www.pine64.org/clusterboard/ for $99 which takes these modules https://www.pine64.org/sopine/ at $29 a piece, which are quad-core ARM Cortex A53 with 2GB LPDDR3. This is their wiki: https://wiki.pine64.org/index.php/PINE_A64-LTS/SOPine

Do you happen to know how long these have been around? I build a little nanopi cluster using a combination of standoffs, cases, and a little light tinkering (aka filing), about 18 months ago and I couldn't seem to find anything like this.

That looks really interesting :), but it seems like they have an pretty-much-blocker bug with the individual modules not turning on again when rebooted:

https://forum.pine64.org/showthread.php?tid=5849

75W 5v bricks can't be that easy to come by were as 12 or even 19v we're usually tripping over.

I'd use either this https://www.mini-box.com/picoPSU-80 and repurpose a spare 12V brick for it, or get something for powering LED strips. There are gazillions of them rated 5V 15A at about 30 bucks.

Note the $189 price tag doesn't include the Raspberry Pi compute modules, which are about $30-40 each.

It's a neat form factor but you could just buy some regular Raspberry Pis and an Ethernet switch.

I thought it was $189 for the whole kit and caboodle, but as you said its not.

$189 is a little expensive for what you get.

That's without eMMC though. Having a bunch if normal Pi's running SD cards would end up in tears at some point.

Isn't eMMC on each RPi board? No?

The compute modules come with onboard eMMC

If you're looking for a cheaper alternative then https://clusterhat.com/ is worth taking a look. I have one sitting on my desk (4 Pi Zero nodes and Raspberry Pi 2 controller).

Very cool, Thanks for the link. Are you also running k8s on it?

Hey guys, I am a co-founder of Turing Pi. I see a lot of comments around Raspberry Pi, performance and VM. I just want to shed a light on some things here. Turing Pi is not about performance, it's about cluster architecture. If you look at Turing Pi as a homelab project, then yes, you can get more performance with some used cheap servers. Of course, if you get approval from your homies to occupy a closet. You can even run apps in containers with Kubernetes orchestration using VM and so on.

The main idea behind Turing Pi is to deliver compute to the edge. If we look at cases where some compute will run low latency, highly available and internet independent apps to automate processes, and often in a hard to reach places, then the classic servers, not a solution. Turing Pi is an early version of edge computers with cloud-native architecture. Why it's important? Because if you are a business with some services running in the cloud and you want your edge computing organically to coexist with your cloud stack, then edge clusters could be a great choice. The speed to innovate and deploy your code into production to both cloud and the edge environment could be a critical component.

The existing Turing Pi model more oriented at forward-thinking developers who want to learn and push cloud-native to the edge. Why Raspberry Pi computers? They are not the most powerful computers, but they definitely can lower the entry point for developers by offering a huge and well-documented software ecosystem.

Looking at the specs, it seems almost dishonest to promote this for kubernetes.

> The nodes interconnected with the onboard 1 Gbps switch. However, each node is limited with 100 Mbps USB speed.

Not only that but Compute Module 3+ are limited to 1GB RAM, is it really expected someone could run a realistic workload? How stable is the control plane node with such limited resources?

It seems like picking up 3 raspi4s (4GB RAM each) and powering via PoE would be a must better result.

I agree, using the USB bus for inter node communication seems like a poor design choice. Anyone got any insight on why they wouldnt use the much faster ethernet connection?

My understanding is that the ethernet module on the raspberry pi is a usb device, not a pci device.

The SoC used in the Pi 3 does not have an Ethernet MAC. It's always provided via USB. The CM3 does not have a USB Ethernet device onboard, it's provided by the carrier board.

I randomly came across a similar offering and thought about buying for fun: https://www.mininodes.com/product/5-node-raspberry-pi-3-com-...

Any idea how the specs compare?

This is still based on the compute modules, so you're going to have the same tradeoffs.

These are interesting ideas, but until the compute modules start having more on board RAM, you'd be much better off working with a few RPi 4's. They will be faster (1Gbps ethernet, more RAM), and cheaper since you only need a gigabit switch to connect them together, not a custom carrier board.

If the compute modules start to get more powerful, then having to deal with only one power supply and ethernet uplink would be nice. It's a very appealing idea. But, you're going to almost always have a better experience with a bunch of standard RPis.

This is what I did. Throw PoE hats on the pis and with a PoE switch it's a real clean setup.

3D printed trays for rockmount fiber cassette blanks and set it in a network rack.

If you insist on using Pis (IMHO you're quickly at the point where a (potentially used) NUC or small office PC is the better choice), why would you go for extra expense and effort for PoE? Just connect a 5V PSU to the power pins? I guess it gives you individual power switching without any DIY, but other than that?

Simply for not having to deal with all the usb-c adaptors and cables.

It's also pretty cool being able to turn the pis on or off by cycling the switch port from your network control plane (Unifi in my case)

You could even do it dynamically via the api for a "bare-metal autoscaling" type workflow.

> Simply for not having to deal with all the usb-c adaptors and cables.

You can get one large 5V PSU and go directly to the pins on the Pi. No need to bother with individual USB adaptors...

I built a cluster of 6 pi 3b+ duct taped to a USB hub for power and a switch and router a few years ago for the explicit purpose of experimenting with clustering technologies, including kubernetes. I wasn't running it for long periods of time but it was surprisingly stable. But to your point, 1GB RAM doesn't get you very far with many of the popular distributed systems these days.

I'd hope people wouldn't be using a setup like this for any other reason than learning k8s or tinkering/fun.

edit-before-actually-posting (sorry I'm a bad person for typing a reply before clicking the link): Wait, they're selling these things? Ok, then I'm stumped. What would you do with 8 rpi's that you couldn't do with one?

I'm holding out to see if a CM 4 with 4GB ram comes out before I commit to something like this.

Someone had one of these 'backplane' style boards a few years ago, but they only used it for flashing Pis for distribution. I don't think they ever made it commercially available.

This looks fairly similar.

And can we all just pause for a moment and look at that heat sink on the ethernet controller? Holy cats, what's goin' on there?

This is neat, but I'm really more interested to hear about potential use cases. I'm guessing this is mostly useful for ARM workloads? Maybe some situations with low power requirements?

Personally, for my multi-node test clusters, I just run VMs on cheap x86 hardware.

What's the practical point of running containers on VM?

This sounds like every talk given by Bryan Cantrill regarding public clouds running containers.

To expand on my personal use-case, I don't want or need a whole stack of physical servers sitting around just for a test k8s environment. There are some things that you can only really test properly in a real multi-node environment rather than single-node solutions like minikube: failovers, shared storage, some networking particularities, etc.

But there are reasons to run containers on VMs in production too. Hypervisors and container orchestration tools solve very different problems. Depending on what problems you are trying to solve, it might be useful to leverage both.

Can you please elaborate a little bit more on the reason to run containers on VMs in production?

Lets say you have a complicated distributed application that needs to run on multiple cloud providers, and maybe it needs to run on-prem sometimes.

You could write your application multiple times, and publish multiple images for the same application across different infrastructures.

Or, just build your application once to run in containers, and run those containers on k8s everywhere.

Here's a reason if your production platform is Kubernetes:

---

The default pod limit per node 110. This is by design and considered a reasonable upper limit for the kubelet to reliably monitor and manage everything without falling over into NotReady/PLEG status.

If your node has a ton of cpu and memory, then 110 pods will not come close to utilizing all the metal. You can go down the path of tuning and increasing the pod limit, but this is risky and often triggers other issues on components that are designed for more sane defaults.

It also means that if your node goes NotReady (not a hardware failure), it's now a much bigger deal because you have fewer total nodes and many more pods to re-schedule at once.

This is solved by splitting up these massive nodes into smaller nodes via virtualization.

It's also nice having an api-driven layer to manage and upgrade the vms versus shoehorning a bare-metal solution. I would argue it also encourages immutable infrastructure by making it much more accessible.

There are bare-metal solutions but it's often more complicated and slower than launching/destroying vms.

Not the person you replied to, but we do this.

The main reason being on premise deployments. Our workload is primarily machine learning models, as such they are well suited to docker containers due to good support and being able to quickly build images, deploy and tear down.

Additionally, whilst the task would be well suited to a Kubernetes deployment which is ideal in the cloud, on prem we're likely running on a single box running multiple VM's. Hence we use containers within the VM.

It isn't the perfect solution and definitely has it's downsides, but it is good enough and better than the alternatives when considering everything possible.

for this use case, i would rather use something like SimpleNUC[0] .

It's basically just NUC machines with rack mount. It's a lot more powerful than Rpi, quite power-efficient (when compare to actual server rack) and dead silent.

More importantly, this setup can handle some actual workload

[0]: https://simplynuc.com/server-shelf-solution/

That takes it from $300 to $3000.

this turing pi cost $189; 7x rasp pi 3 cost $210 and we need to buy SD card or eMMC, power plug or usb power hub. the total cost is not cheap either, when compare with used server rack.

Finding and running docker containers on ARM is unfortunately a pain.

Reply on news.ycombinator.com