Hacker News Reader

Comment by th3h4mm3r | Hacker News Reader

What could you do with a similar setup? I mean: what's the purpose and/or the advantages compared to a "classical" solution ?

View on HN

Purpose: Learn linux clustering/diskless nodes.

Advantages: Cheaper than conventional servers, limited physical form factor suitable for constrained environments such as apartments, luggage, etc. Low power consumption, suitable for mobile/off-grid/renewable.

Commercial advantage: Run highly cost and power efficient data centers, call centers, etc. Eliminate redundant state. Remove manual configuration rollout. Infrastructure as code. Full CI/CD with kernel-level control, per node, for any OS. Basically becoming a lost art in this day of "outsource to cloud". With SDN knowledge, confidently build your own cloud provider.

You could rent a couple of 1USD/month VPS, or if really on a budget rent a bunch of containers with ipv6 for just a few cents

You absolutely could. But you wouldn't learn how to build a system that recovers from yanking a power or network cable, wouldn't own it, and wouldn't be able to run it on your terms. You'd need a credit card with an American-run payments network to operate, with a declared identity and a bank account, for which a prerequisite is a few months of stable living and income in a particular jurisdiction, itself something large portions of the world do not have access to. Further you'd need to be happy with the fact none of your data was in your country, under your physical control, and never would be. But feel free to ignore the rest of the world and the fact that all this rental crap runs on open source software, just throw in the towel and be a rent-payer: a strategy that has its perks and efficiencies, for sure. But don't discourage those who seek to learn more.

https://ipv6onlyhosting.com/ is the first result, and accepts payment by bank transfer.

Many other small VM hosts accept payment by Bitcoin.

> You'd need a credit card with an American-run payments network to operate, with a declared identity and a bank account, for which a prerequisite is a few months of stable living and income in a particular jurisdiction,

None of this is true.

Try being from a random South Asian or African village. Tell me how you go buying cloud services with your nonexistant banking infrastructure and personal credit line.

The physical location of a person isn't generally the key barrier to being unbanked; instead it's poverty.

"The most common one was having too little money to use an account. Two-thirds cited this as a reason for not having a financial institution account, and roughly a fifth cited it as the sole reason. Cost and distance were each cited by about a quarter of those responding to the question, and a similar share said they do not have an account because a family member already has one. Lack of documentation and distrust in the financial system were both cited by roughly a fifth of adults without a financial institution account, and religious concerns by 6%."

https://www.cashmatters.org/blog/globally-17-billion-people-...

(And to state the obvious: poverty is also a barrier to owning physical infrastructure)

No, try buying cloud services while living in a remote village where clothing is non-existent.

This sounds like some kind of fantasy; how many people without stable living conditions or without a bank account that can be processed by the US are concerned with...cluster computing?

"What about all the homeless Iranian Ferrari owners!" doesn't seem like much of a rallying cry.

Surely if there is even 1 unbanked person in existence that is interested in cluster computing, then this is a worthwhile endeavour.

No, absolutely not.

Or maybe to Don Quixote, but he was famously delusional.

Is there any number of unbanked people showing interest in cluster computing that would change your opinion?

I "show an interest" in space, but I'm not entitled/deluded enough to expect someone to deliver a shuttle to me, because that would be a waste of time/materials/energy/money.

It's the same here.

If you had spoken to people in the 1960s and told them how many transistors we would have on a chip in 2020, they probably would have called you delusional.

Also you are comparing a space shuttle to a few raspberry pis, hardly apples with apples.

Someone needs to invent a way to drop things off your desk onto a power switch via ssh.

Where does one find VPSs this cheap?

A few years ago I could vouch for LowEndBox: https://lowendbox.com/. I've not frequented the site in a very long time though, so can longer be sure.

RamNode has some starting at $15/year on their OpenVZ platform. Have been happy enough with that for several years now, although it's quite limited as to what it can do in terms of performance. But as another commentor mentioned, check out LowEndBox. It's where I discovered this.

https://www.serverhunter.com/

From my experience the cheapest reliable ones in terms of storage, cpu, ram performance come at about 3$/month (+-50 cent). I can recommend Hetzner.

One issue I have with going the VPS route is that what I'm charged is often not what I'm expecting. I will use estimation calculators and figure my bill will be $25/month (which I get for a couple months) and then suddenly get a $120 bill. This has happened more than once and has had me loose all confidence in these services for hobby projects. I'd rather just buy a Pi cluster at this point.

Your agreement was $25/month, so if they start to charge you $120 you pay them $25 and then cancel the service. This is very annoying but do happen dealing with unscrupulous businesses. Although if they have 500% margin eg. you pay $5 for something you can get for $1 elsewhere they will care more about keeping you as a customer. Compared to for example a cloud bill, renting a VPS is both cheaper and more predictive. If you are however paying $25/month you should start looking at dedicated servers, where all resources are preserved for you.

Sounds like AWS or one of the big guys. The smaller providers usually have an all in price for a VPS with a fixed amount of rosources, no surprises. Lightsail is similar but kind of sucks.

Yeah, it's AWS. In the future I'll be looking at the smaller providers.

Any links to an ok $1 VPS that would work? I tried a few a while back and always have an issue with RAM.

What's the best way to go about clusters since I think you need 1gb RAM per node and lowest I've seen is $5.

Thanks

> Learn linux clustering/diskless nodes.

Honestly, could be as valid with a setup that just involves running a bunch of VMs.

> Low power consumption

Yeah, well, a typical laptop should draw around 50W, which is what, like 0.05$ a day?

> Commercial advantage: Run highly cost and power efficient data centers

Hu, no, in terms of power efficiency per unit of work, you will not beat specialized hardware.

> Infrastructure as code. Full CI/CD with kernel-level control, per node, for any OS. With SDN knowledge, confidently build your own cloud provider.

Sure, also, blockchain, 5G, serverless, microwaves, rust, wasm

Lets come back to earth for a second. There is objectively no purpose for this, except it's fun and nerdy. Do we really have to find excuses?

To be clear, the commercial utility refers to the skills, not the hardware.

> > Learn linux clustering/diskless nodes.

> Honestly, could be as valid with a setup that just involves running a bunch of VMs.

Slightly less valid. You don't have the same IO latency between nodes using VMs unless you start messing around with artificial attempts at inducing latency on each node individually.

It's much easier to accurately simulate real world networking by not bothering to simulate and just using real world networking.

There's a huge difference between latency on a wire vs latency within a hypervisor, and you won't know what you've papered over through lower latency via VM until you move to real hardware.

Thanks for this, the other comments had left me, a layman, thinking this was actually useful instead of just for fun.

> Yeah, well, a typical laptop should draw around 50W, which is what, like 0.05$ a day?

At least where I live that would be 1.2kWh and about 13 cents per day, around $4 per month, or around $50 per year if running 24x7.

Pacific Northwest has I believe some of the cheaper power in the U.S. at 10-11 cents per kilowatt hour.

If you want to learn clustering with physical hardware, and you don't have the power and size constraints, you're better off buying some off-lease business desktops. These i3 and i5 systems with 4 to 8GB of RAM can be had for less than $100 each.

When you're done with the cluster, you can likely sell them for nearly what you paid for them.

i think you will lose in the end this way if you actually factor in power costs.

Yup, 6x old i5 thinkpads will require more space, more power and will likely be more expensive than 6x Pi.

I'd take 6 old ThinkPad over 6 new Pi any day. :)

I host a lot out of my house on a T430S with a cracked screen.

I know there are thinkpad hoarders with like 10 or 20+ old thinkpads. Seems to be a nice hobby, but you will rarely find one who builds a thinkpad cluster.

In the case of Raspberry Pi it is pretty normal nowadays to connect a bunch of them into a cluster and have some fun. In fact, it has become so popular, that you u can find stackable cluster cases and other Pi cluster hardware on amazon.

Not when you factor in performance per watt.

http://web.eece.maine.edu/~vweaver/group/green_machines.html

I’d love to see the M1 on this list.

Is the power consumption really that good? I mean comparing to just getting a single 5950x or something would this actually win in terms of performance per watt?

I bought a Turing Pi 1 . It's a 7 node mini-cluster, not very expensive (I think about £300 fully populated), that fits into an MiniITX-style case with a single small power supply and single ethernet exposed to the world. You can easily pick the thing up with one hand. It's certainly not powerful, each node is a very modest ARM processor with "only" a gigabyte of RAM and 8 GB of local storage.

The alternatives are much more powerful, but far more expensive and not very attractive in other ways. Blade server? Huge, noisy, power-hungry and I wouldn't get much change from many thousands of pounds. Rack of NUCs? Powerful of course but far more expensive, large, separate power bricks, ethernet switch and lots of wiring required. VMs running on a desktop? Still more expensive, larger, and not the same tangible experience as having a real cluster. VMs on a laptop? Laptops at the same price point would have too little RAM to run lots of VMs well.

So these Raspberry Pi clusters fit a niche which is not well served by other solutions.

Its cool and fun and teaches you a skill at a low price point. There is essentially no functional reason to do this over a single high power server unless for some reason you need a bunch of ARM servers for something.

All fourteen Pis, even v4s (which are significantly hungrier), would be lower power (as in electrical draw) than even a feeble (as in compute power) traditional machine.

If you can distribute your workload (or it's not intensive anyway), i.e. having no single machine with a faster processor, then I think it's a good choice, cheaper to run.

Edit: I'm happy to be corrected if down-voters would like to reply... A Pi 4 draws 6-7W at full load. A former desktop PC turned server likely idles over 100W.

They're different beasts, so I'm not saying 'a Pi is better than a PC', I'm saying 'if your load could also be handled by a (bunch of) Pi(s), then all else being equal that might be better than the PC'. Of course all else is never equal, ymmv, you may already have the old PC, you may need very many Pis, so even if pennies cheaper to run it might take years and years to break even on the Pi acquisition cost.

Did not downvote but desktops idle way below 100w.

Idle as in powered on, not sleeping, just 'no' workload? I stand corrected if so, I didn't realise that was the case.

I also like the idea of having an easy homogeneous setup, each Pi having one disk, vs. my alternative (old PC parts) being something like '4 SATA ports on this one, and 6 on that one, but one's broken, less one on each for the boot disk' - I quite like the idea of having a bunch of Pis, and one image that I apply to all of them, which can include details like /dev/sda being its only data disk. That's not really a functional advantage, just appealing for keeping it simple to come back to and remember/understand how I set it up when I need to tweak something (and really easy to add another one increasing available cpu/ram/disk).

This high end system draws ~35W idle: Intel Core i9-10900K @ 5,0 GHz, AMD Radeon RX 6900 XT (16 GB), Asus ROG Maximus XII Hero Wi-Fi, 2× 16 GB G Skill Trident Z Royal 3 600 MHz 16-16-16-36, Samsung 970 Evo M.2 1 TB, Samsung 860 Evo 1 TB, Seasonic Prime Ultra Titanium, 1 000 W.

Picture in Swedish, from graphics card review comparing average power draw for the entire system for various GPUs, idle and while benchmarking in Metro Exodus, numbers rounded to nearest multiple of 5: https://cdn.sweclockers.com/artikel/diagram/22452?key=2c6f21...

If the direct picture link doesn't work, it's under "Effektmätning" here: https://www.sweclockers.com/test/30908-amd-radeon-rx-6900-xt...

Exactly. Take a look at the "ENERGY STAR ® Power and Performance Data Sheet" for Dell "office" desktops. Many are around 10 watts for idle.

It's not that difficult to have a reasonably modern desktop PC idling more in the 10-20W range. For 100W you need to do something seriously wrong, or add extras that would blow up the Pi usage too (e.g. a home server full of hard disks will of course use more power, but so would Pis with the same number of drives)

Don't forget that Pis also draw quite some power while idle. The Pi 4 draws around 3.4 Watts [1], or around 2.7 Watts since they are running headless in a server setup you'd probably disable HDMI [2]. Times fourteen that's 37.8 Watts just sitting there. An idling headless Desktop PC will be below that for sure.

[1] https://magpi.raspberrypi.org/articles/raspberry-pi-specs-be...

[2] https://www.pidramble.com/wiki/benchmarks/power-consumption

A good functional reason is when you need to have high redundancy for low-intensity jobs while keeping power consumption ridiculously low. As long as you don't use SD cards, Pis can run for years without stability issues.

lol? what if you don't need a heap of compute power but require a lot of redundancy? A single server is not good for that...Also not every site has internet so yes having small redundant easily replaceable nodes on-site does have plenty of use cases.

Y'all negative folks against this seem to have very restricted use cases in mind. Not all redundant rack servers live in a 36U case in a air-conditioned internet connected server farm. There are plenty of use cases outside of this.

It’s fun to hack together. Not everything needs a clear business case with strong ROI.

This. I think the Hacker News mindset has substantially shifted in the last few years from the "Hacker" mentality of doing things because they are fun and knowledgeable, and more towards Enterprise News..

Not everything needs a business case, some things (most things) should be done just because you want to see if you can.

If someone really needs to think in items of ROI and that everything needs a business case, then this ideology of doing things for fun and testing limits is the only way to really innovate.

Clustering SBC's is a great learning experience, and makes for a great budget friendlier home lab setup.

I have a friend that uses similar setups to tech kubernetess but previously he also used it as example for all sorts of distributed computing (when Hadoop was still a cool thing to do)

Could be used for testing hardware+firmware. Example connect 14 hardware revisions of a $SOME_HARDWARE to each RPi and test firmware changes.

You can neatly mount your Pis in your rack where you have the rest of your gear supported by network switches, power delivery, cabling, cooling, good internet connectivity connectivity (if housed in a machine room or colo) etc.

If you need a cheap dedicated (for privacy) server to run a mail server and perhaps a private cloud with Nextcloud, this is ideal.

Nextcloud would be much cheaper to run on a x86 board.

Can you expand on why?

My guess would be Nextcloud put a lot of load on both CPU and disk (for thumbnail generation, file encryption, etc.) making Raspberry Pi which lack performance in both front not very suitable for the task.

I tried running Nextcloud on Raspberry Pi 4 (4GB) a while ago, and hitting performance issue on SD card very fast (initial DB import take very long). I tried to solve this by moving all Nextcloud data into iSCSI disk but RAM & CPU became a problem once I started adding more files. After a week or so, I tore everything down and went back to a VPS instance.

It might works better with Raspberry Pi cluster on Kubernetes, though. (This very Raspberry Pi is now happily running as a DVB streaming server.)

Thanks. I keep meaning to try it out but booting from an SSD instead. I was hoping anything intensive (e.g. thumbnail generation) would be a one-time thing so not noticeable in daily usage. Perhaps I will re-think, I have a ThinkCentre M75q I could use but that seems overkill.

I find you can get a much better experience with a cheap celeron NUC than a RPi because everything from single thread performance to IO speeds favor x86 right now.

Try using a SSD connected via USB 3.0 on the RPi 4.

Which x86 board do you have in mind?

The RPi4 with a USB SSD should be fast enough and use little power.

Reply on news.ycombinator.com