The cheapest way is to pick up an old ThinkStation (or other tower), load it up with 128GB (or more) of ram and install ESXI on it. That's a perfectly good baseline, and you can run about 30 4gb linux VMs on it.
Ideally you'd have a bit less than 1 core per VM, just so it's a bit slow. Lots of people assume your nodes are quick, but in real life they may not be. And really, most of the time your machines won't be doing squat.
You might want to have SSDs in there too, because ESXI doesn't have RAID capability (or at least mine didn't). I don't think you can get a cloud device that uses spinning disks anymore, and you wouldn't use it in real life anyway.
A 2tb drive is cheap these days, or just slap all those old small SSDs in there. Everyone has a bunch of those small SSDs left over, and they're perfect.
1. Write any stateful program
2. Now look at every single LOC and imagine what happens to the system if the service crashes before executing the next LOC. Then modify the system to deal with those scenarios.Yeah, people miss this. If your app interacts with another app - bam distributed.
By designing twitter in a 45 min interview.
It's written in Go, so it'll help if you are familiar with Go. But the code is not difficult to understand even if you don't.
2. A bunch of communicating local VMs (easier with a beefier machine like my current desktop).
3. Mininet (there are other options) to simulate a network environment, can fully control the topology very easily. Lighter weight than (2), more control for simulating different network effects than (1) alone.
"Singularity" systems are an abstraction afforded to us by the grace of the hardware we run them on. If you start pushing their performance hard enough, however, you inevitably get distributed behavior.
This is also a good potential career reason to try to make software which is as performnt as possible - you'll get all the tasty edge cases and complexity war stories to talk about.
Use a small k8's distro (kind, minikube, k3s) and build something that talks amongst itself and is resilient.
If you want to be "optimal, neat, or best practiced" read a book, and get stuck in tutorial hell. If you actually want to learn how to do something, literally go do it. Nobody has ever built anything of value (whether that is financial, intellectual, or emotional) by leetcoding.
The problems they're trying to solve are related to really large distributed systems that fail a lot, and their design decisions are basically a "this is how we worked around that problem."
You can also look for the LISA archives (https://www.usenix.org/publications/loginonline/thirty-five-...). System administrators were the first people that had to deal with large distributed systems at scale, and university system administrators led the charge.
You might want to hunt down the comp.sys.admin archives (I can't remember the newsgroup anymore).
Most of the ideas and issues behind distributed computing are obvious if you think about it. Many of the actual implementation and mitigation of those are not obvious, though.
And there's also the client side of distributed computing, which I don't think is discussed as much.
As an example, exponential backoff is one of the go-to techniques for clients when the servers are under load. Unfortunately that doesn't really work IRL, because instead of spreading the load you get waves of load coming back over and over. Likewise on the server side you have problems with peak load.
https://www.amazon.com/Designing-Data-Intensive-Applications...
Maybe some Internet of Things applications would provide a good avenue for some distributed systems exploration?
Some people are just up front about it - I've read a lot, and practiced the best I can, but am looking for some real world experience to marry that too.
I have my current thing set up to create VMs from a downloaded cloud-VM image with minimal updates (add my ssh pubkey and install Python), and then use Ansible for everything further.