Hacker News Reader

Comment by linkmotif | Hacker News Reader

My biggest problem with local k8s dev is building containers. I’m on macOS. Does this solution address that?

Is the problem with building containers that you don't have a Dockerfile and you don't want to spend time making one?

Draft does that pretty well with Draftpacks, in my experience if there is a Draftpack for what you're using then the Dockerfile that draft spits out will get you pretty close to a workable container image with basically no effort, or sometimes even all the way there.

I'm trying to understand what you mean that building containers is a problem. Is it the procedural act of actually building and rebuilding the container that is the problem? (Draft handles that too...)

Edit: According to the Tilt tutorial, yes, tilt will build your containers. Eg: https://docs.tilt.build/first_config.html

I’m all aboard Docker, but I don’t use Linux natively, so I’d need to run a VM inside macOS, or have another box to build containers. Even if I did, I use GCR in production, so I’d need to template my container images in helm to use GCR only in production versus whatever I use locally for development. And what would that even be? I haven’t explored minikube to know what it does for local dev container registry.

Anyway, thanks for the link. Missed that.

EDIT: Looks like tilt is using GCR in those examples. I’m not trying to traffic container images off of google cloud for local development. That requires (fast) internet just for local development, and outgoing traffic really adds up quickly in terms of cost.

PSS: never heard of Draft. Checking it out now. Not using azure though, maybe that’s not relevant.

Azure team open source products can almost all be used without Azure cloud.[1] I do use Azure but not for everything, you can definitely use Draft without Azure, it's Kube-native software so you can use it anywhere you have Kubernetes. (Like Tilt, from what I understand about it so far... Tilt and Draft look to be very similar tools, with different approaches.)

I helped someone on Twitter with this question last week. I have a (* single-node) Kube somewhere and I want to build images in it, and not have to pull them from a repository. The parlance in Kubernetes for this is, an imagePullPolicy of Never. (you actually do need a repository if you don't have a single-node Kubernetes, full-stop.)

Docker does actually run a VM with Linux in it, doesn't it? The modern version of Docker for MacOS also offers a Kubernetes checkbox, so if you're already using Docker, you needn't have another VM just for running Kubernetes.

This is certainly a problem that could use more clarity and for users, maybe a nice howto blog post for clarification, but it's absolutely possible to do this without a registry. I think Tilt mentioning "gcr.io" in the image tag is not necessarily a signal that you should actually push your image to the gcr.io registry. It's therefore possible to build and execute your containers while never pushing or pulling. If that's what you wanted to do, then that's exactly what you may do by setting your imagePullPolicy to Never.

(I have no idea how this works with Tilt, but I'd assume it's possible to use a similar approach since it's also on Kubernetes.)

The tweet in question: https://twitter.com/yebyen/status/1080534315157635072

[1]: the exception is things like aks-engine. You can only use aks-engine with Azure as I understand it, because Azure implements their own API for creating virtual machines, and nobody else implements that particular API. But I can't say how unique the Azure API actually is, or if it's also open source, could you build your own Azure in a rack, from all open-source components? That would be pretty cool, if you could...

If you tried something like docker-machine on Mac a while back and were disappointed with abysmal performance, it's worth another look. They've replaced VirtualBox with HyperKit, which uses the Apple-supported Hypervisor.framework and works much, much better.

Local kubernetes solutions (like docker-for-mac and minkube) usually let you push your image directly to the local docker daemon. No need to upload to a remote registry only to re-download it. A lot of tools in this space are pretty good about detecting this optimization, including Tilt.

(disclaimer: I work on Tilt)

Sounds like you should try skaffold, it has a workflow for local builds, with fast hotloading to auto-rebuild your image. It also redeploys automatically too. If you're using Minikube it'll share the docker daemon so that you don't have to push anything anywhere.

I find Docker for Mac to be completely fine, and Minikube can be a resource hog, but still workable if you make your resource allocations configurable (i.e. don't request 1G for each pod in your local env, even if you need to do so in prod). There is an xhyve driver for Minikube that should reduce the footprint by not requiring you to run virtualbox.

Try working with Elixer and Phoenix and you will start to appreciate how builds can be a nightmare.

Not trying to be obstinate, but I'm a Rails dev and probably won't do that

There is a Draftpack for elixir: https://github.com/technosophos/draft-elixir/tree/master/pac...

And here's an Alpine-based dockerfile: https://github.com/bitwalker/alpine-elixir-phoenix

What kind of problems would I expect to run into?

Is your issue with building containers and getting them into the minikube VM?

If so, you can use "eval $(minikube docker-env)" to make your shell's docker commands talk to the docker daemon in the VM.

Running a VM is a total drag. I only have 16GB RAM. Then even if I run a VM, the docker registry is not the same as my production docker registry (GCR). I guess I can account for this with a variable in my helm templates, but then it’s just more complexity.

Would you recommend minikube even with 16GB RAM? How much do you allot to the VM?

I'm a long-time minikube fan, I like the isolation (totally worth whatever minor VM memory overhead), reliability, ease-of-use; it feels like an appliance in a good way, which is something I traditionally associate with VMs. minikube contributed strongly to me personally getting excited about k8s in 2016.

The local docker registry is an asset because it gives me flexible when offline.

Thank you for your report! I will look into this.

I have no problems building containers on macOS. Can you elaborate?

Building containers requires Linux, so I need to run a VM or build on another box/the cloud.

I built a container a few minutes ago on macOS, not sure why you believe this to be the case.

Docker containers/lxc require Linux. They are a Linux technology.

This is technically true but in practice Docker for Mac hides this from you and does not require a prohibitive amount of memory. 99% of the developers at my company have 16gb MBPs and we all build multiple Docker containers daily.

Thanks. I didn’t know the visualization system changed. I will check this out again.

As weird as it sounds, Microsoft had implemented the full namespaces stack in Server 2016 (unavailable in Win10 as the base for native containers is Server where kernel differs a bit), which Docker supports (and k8s in alpha/early beta).

Docker switched from LXC to containerd in 2016.

That is very weird. Wow. Thanks for letting me know. I guess it's not a Linux technology then.

If you are running Docker for Mac, it should work out of the box :

Docker for Mac uses HyperKit instead of Virtual Box. Hyperkit is a lightweight macOS virtualization solution built on top of Hypervisor.framework in macOS 10.10 Yosemite and higher.

From https://docs.docker.com/docker-for-mac/docker-toolbox/

I build Linux target images on either my Mac or a handy EC2 instance for deployment to AWS using their ECR as a repository. No difference.

This is somewhat outdated.

Docker for Mac uses a relatively lightweight and largely-invisible virtualization layer now.

Thanks. I didn’t know the visualization system changed. I will check this out again.

Reply on news.ycombinator.com