Stripped-down Kubernetes on the Raspberry Pi
blog.alexellis.io
blog.alexellis.io
One might argue that USB 2.0 interface of RPi doesn't add much value when a USB SSD is connected to it, but benchmarks show that it adds at-least double the performance[2].
[1]:https://www.raspberrypi.org/documentation/hardware/raspberry...
[2]:https://jamesachambers.com/raspberry-pi-storage-benchmarks/
At first I thought the recommendation to use an SSD with a Raspberry was crazy, but the benchmarks do indeed show an improvement. And, as that article points out, since the SSD's performance isn't important (hitting the USB 2.0 limit) you can grab the cheapest, decent SSD which ... comes in at just $20!
Though I still like SD cards for the form factor. There's the newer A1 class SD cards with better minimum IOPS, which I'd be totally fine with. But reliability is still such a big issue with SD cards. I had one die a few days ago; always annoying.
Reg the cheap SSD, yes also the benchmarks show the cheapest SSD (Crucial BX 500) on the top. I'm not certain, why this is so; I think it is because people put the cheapest SSD due to RPi limitation but then again Samsung SSD in those benchmark score less which scored high in actual SSD benchmarks for PC.
I think the conditions in the RPi doesn't bring out the best performance from Samsung's controller but does so with Crucial/JMicron. SD cards are supposed to be much more reliable than SSD, it is our improper application that's causing it to fail especially in the OC RPi.
https://github.com/rcarmo/azure-k3s-cluster
...as well on my own ARM cluster, with a private registry:
https://taoofmac.com/space/blog/2019/05/18/2034
I find it refreshingly straightforward for personal and testing setups (and more practical than microk8s for me right now), and am waiting for rio to hit a couple of stable milestones:
https://github.com/rancher/rio
(I try openFaaS now and then, but after contributing a deployment template early in the beginning, I lost my enthusiasm for it - it also ran the gateway and the admin UI on the same process, which I considered a design flaw).
Also, Rio sounds like "host your own Google App Engine" — am I right?
One of my coworkers tried to use microk8s instead of minikube and we debugged it extensively for a couple days, but ended up baffled. We had to setup some rules to forward localhost:5000 into the cluster for docker push; instead we got a random nginx instance that we couldn't figure out where it was running. Even after uninstalling microk8s, we still had a ton of random iptables rules and localhost:5000 was nginx... It was weird.
Minikube works great, however. You will still need some infrastructure to push to its docker container registry in order to run locally-developed code. Out of the box, you can persuade your local machine to use minikube's docker for building, but it runs in a VM and unless you use non-default minikube provisioning settings, it doesn't have access to all the host machine's cores, which is kind of slow. I ended up making minikube's container registry a NodePort so that every node (all 1 of them) can get at localhost:5000 to pull things. I then added some iptables rules to make localhost:5000 port-forward to $MINIKUBE_IP:5000 so that "docker push localhost:5000/my-container" works. It's kind of a disaster.
I also had to write an HTTP proxy that produces a proxy.pac that says "direct *.kube.local at $MINIKUBE_IP" so that you can visit stuff in your k8s cluster in a web browser and test your ingress controller's routing.
After those two things, I quite like it.
I still don't think minikube is a good platform for developing microservices, though. The build/deploy times are too long (and things like ksync don't work reliably, even if you generate a docker container that can reliably hot-reload your app, which kind of involves a lot of setup). I once again wrote something that takes a service description and a list of its dependent services, allocates internal and external ports, puts them in environment variables, starts Envoy for incoming and service-to-service traffic, and then runs the apps wired up to receive requests from Envoy and make requests to other services through Envoy. It took a while but now that I have it, it's great. I can work on a copy of our entire stack locally, it starts up in seconds, and is basically identical to production minus the k8s machinery.
I am still surprised I had to solve all these problems myself, but now that they're solved, I'm very happy.
What I have is as follows. Every app in our repository is in its own directory. Every app gets a config file that says how to run each binary that the app is composed of (we use grpc-web, so there's usually a webpack-dev-server frontend and a go backend). Each binary names what ports it wants, and what the Envoy route table would look like to get traffic from the main server to those ports. The directory config also declares dependencies on other directories.
We then find free ports for each port declared in a config file, allocating one for the service to listen on (only Envoy will talk to it on this port), and one for other services to use to talk to that service. The service listening addresses become environment variables named like $PORTNAME_PORT, only bound for that app. The Envoy listener becomes $APPNAME_PORTNAME_ADDRESS, for other services to use.
Once Envoy has started up, we then start up each app. The order they start in doesn't matter anymore, because any gRPC clients the apps create can just start talking to Envoy without caring whether or not the other apps are ready yet. And, because each app can contribute routes to a global route table, you can visit the whole thing in your browser and every request goes to the right backend.
I used Envoy instead of just pointing the apps at each other directly with FailFast turned off because I needed the ability to send / to a webpack frontend and /api/ through a grpc-web to grpc transcoder, and would have used Envoy for that anyway. This strategy makes it feel like you're just running a big monolith, while getting all the things that you'd expect with microservices; retries via Envoy, statistics for every edge on the service mesh, etc. And it's fast, unlike rebuilding all your containers and pushing to minikube.
It kind of solves the same problems as docker-compose, but without using Docker.
I don't know if that means I'm responsible for writing Dockerfiles, or if I can change out for things like Buildpacks.io buildpacks v3? But I do think that means the system can scale to zero replicas when the traffic diminishes to the point where we haven't seen any requests for something like 10 minutes, and it's fully in doubt whether or when I'll see another request I need to serve.
I have been wondering about Rio but so far not enough to break down and try it.
https://blog.alexellis.io/build-your-own-bare-metal-arm-clus...
This list might help. https://www.hackerboards.com/home.php
I have a few Raspberry's floating around the house doing odds and ends, and balena seems like a nice option for reducing their management needs. But I really want them to be able to get the latest security/etc updates without me having to manually update each of them from time to time.
The main consideration for a feature like this is that sometimes containers have dependencies to interfaces exposed by the operating system which are not always stable. This is especially true for IoT usecases because containers will typically interface with some device connected to the system.
Tangential to this, we're working on an extended support release schedule (a la firefox) for balenaOS. I could see us building an automated OS update mechanism on top of that. We'll definitely think about it, thanks a lot for your feedback :)
Btw balena is a great service, good job!
(Nothing wrong with docker, its just that some nodes are already running lxc)
It's faster and much more light weight than minikube, since you can quite easily set it up on your machine and avoid a VM.
"Install this software on your server to make it serverless!"
Took awhile for the cringing to go away on that one.
Serverless is marketing term that loosely describes a shared tenancy environment where you don’t manage the host.
There’s no formal definition for “serverless”; it literally is just a marketing term that loosely describes shared tendency environment and The only reason Heroku isn’t “serverless” is because they don’t market themselves as that.
So it’s a pointless exercise nitpicking anyone’s definition since there isn’t an actual formal definition. Eg AWS uses the term serverless to describe other services that isn’t lambda.
The whole term is just made up marketing bullshit for shared tendency.
In the 70s we used to call it time sharing. But I doubt many people these days will remember that term.