Virtualizing development environments in 2023
hocus.dev
hocus.dev
I don't really understand the point of a "dev box" that's hosted in a DC and shared, at all - at least not the way they're painting it here.
Hardware capabilities for even consumer level laptops and desktops have progressed much faster than average network connections.
Having testing/preview/branch named environments in a DC? Sure. But this line:
> They should be able to run any software they want inside their own workspaces without impacting each other.
What does that even mean?
Is this about someone working on a feature branch that uses some new dependency that needs to be installed?
That's 100% the sort of thing your local development environment is for, until you're ready to push it to your hosted test/whatever environment.
> But when it's running in a VM, the VM gobbles up as much memory as it can and, by itself, shows no inclination to give it back.
Someone, anyone, tell me which hypervisor doesn't enforce a memory limit on VMs?
https://code.visualstudio.com/docs/remote/ssh-tutorial
See also:
https://code.visualstudio.com/docs/devcontainers/containers
People pay for this.
Virtualization means images means standardization means everyone is using the IDE and tools that IT and "Enterprise" decides.
It has it's pros and cons. But one of those cons is that it can make for an intolerable work environment. At least for me. I've gotten to the point where I now ask about dev environment during the interview process. If they mandate Mac or Visual Studio I'm not interested (not trying to start a flame war, those are my personal preferences). I know that certain organizations need their control for auditing, security and remote wipe reasons, and that is understandable. I'll just go work elsewhere. It's fine.
I used to work at Uber and what we ended up doing with devpod (https://www.uber.com/blog/devpod-improving-developer-product...) was to enable the popular IDEs to connect to these remote environments - all the dotfiles etc etc were persisted so it literally felt like the local IDE, just way faster. Admittedly, it costs a bunch of money to build internally, but there's a path to having people be happy with dev environments.
(we collected data on what IDEs to prioritize based on surveys)
surveys were anonymized
You couldn’t drag and drop the whole folder, else you’d overwrite the numerous configuration files that were all over the place.
I really don’t miss it.
Dev boxes don't have to mean standardizing on One True Editor (although everyone knows that's vim), as long as you can get to the source, typically via ssh, and edit it, whatever you want to use is just fine.
What a world that would be.
/s
My point really, was if your local dev environment requires you to jump through a bunch of hoops doing manual setup to get it working you're doing it wrong.
As much as I dislike the experience of remote dev environments in general, there are certainly times where it may be worth the pain.
I think that’s your problem right there
I do like local development better, every time it is feasible, but there's not much point in doing so if it can't reasonably reproduce a production system.
No. Some prefer image based provisioning, but it doesn't mean you can't have your 'own' machine.
VMWare player? Besides it not being OSS, I didn't notice any performance benefit.
The bottleneck (for my web development needs), is around the spottiness of shared folders and JS tooling that needs to keep a watch for file changes.
So you might have a windows dev using hyperv, another usimg vmware workstation, a mac user with parallels, a linux user with lxc and another with kvm.
Even HashiCorp know vbox is scraping the bottom of the barrel for hypervisors:
> if you are using Vagrant for any real work, VMware providers are recommended since they're well supported and generally more stable and performant than VirtualBox.
I'm sorry but you just lost me completely. If you're not taking a laptop with you, what's the fucking point of having a laptop?
But if you're not happy with that scenario - I don't know maybe your work will be drastically affected if you have two different but identical VMs - tb3/4 external ssds are perfect for storing VMs on, so that you can move between machines at the drop of a hat. I've been doing this for about 5 years as a safety valve to my desktop having a fault: I can just unplug the drive and plug it into the laptop or a new machine and the VMs are all there exactly as I left them.
the VM on an external SSD is a better solution, but then it's still something you have to carry with you even though it's more compact than a laptop...
Devbox will give you the same project environment (packages, env-vars) on your work and home laptop. It leverages nix, and uses your native file-system avoiding the overhead and complexity of using Docker.
Not to mention the popular virtualization platforms like kvm, vmware, and hyper-v all support ballooning[1][2] where you can define a base amount of memory and a max, and when the guest garbage collects its heap within the vm the memory is reclaimed by the hypervisor.
This also allows overcommitting to better utilize the resources that would otherwise site mostly idle, and trust me every cloud provider overcommits their hypervisors, I used to work for a major one but anyone can check the cpu steal time in top and see how thrashed their host is
1. https://www.linux-kvm.org/page/Projects/auto-ballooning
2. https://pve.proxmox.com/wiki/Dynamic_Memory_Management#Ballo...
They don’t mean the memory usage is allowed to grow indefinitely.
They mean, if your laptop has 32 GB RAM, and if your OS and apps consume 20 GB, it’s using 20 GB RAM.
If a VM has 32 GB of vRAM allocated, and the OS is aggressively caching, that VM will consume 32 GB of RAM on the physical host, even if only 20 GB of vRAM is being used within that guest VM.
Of course, every good hypervisor will have a way to prevent this from happening, so the article does seem disingenuous on this point.
If everyone is working in docker in VS Code it’s not a huge jump to have them develop in remote VMs. Now your devs can get setup instantaneously and can use thin and light MacBook Airs instead of heavy MacBook Pros.
Whatever way you're using Nix for developer environments, you can reuse most of that work to define a package suitable for remote deployment.
someone needs to bring a heroku-like experience but for cloud-native development
Memory and CPU cores are cheap. Getting the UX, resource allocation/sizing/instance type, and general pattern for multi-tenant dev boxes seems more fruitful.
Would love to see any data supporting this prioritization, and what the answer is for Workstation / HEDT-type environments given the industry's current focus on AI.
"Dev environment" depends on what kind of work you are doing. What a web developer needs is worlds apart from what a driver developer needs in terms of capability, resourcing, isolation, and rebuilds. Trying to offer a single environment for anyone who builds anything using a computer, is analogous to a single environment for anyone who uses a screwdriver. An electrician is using the same tool, sure, but they're doing fundamentally different work from a kitchen installer or a plumber and have vastly different needs. The single environment with all the capabilities that any computer developer needs is called a "computer."
IMO this is the genius of VSCode remotes. They built a core capability of running the bulk of the IDE "somewhere else" with just the display layer on your local machine. Then they can allow the developer to choose if container(s), VM(s), and local or remote are right for their case, without having to learn new tooling. They can even let you choose if "owning a computer" is required, or if a web browser is sufficient.
I honestly can't find a use for it. My personal machines are ancient quad core setups, but I have a beefy server at my disposal.
But the benefit of faster build times isn't worth the hassle. Between the suboptimal performance of the remote interface, syncing files back to my local machine, and all of the little idiosyncrasies of setting up the remote environment, it's still faster to develop locally. I just go grab a glass of water when I start my compile.
This might make more sense if you're working on some C++ monstrosity with millions of lines of code that takes hours to build on a mainframe, but if you're in the region of minutes per build, there's just no benefit.
Or I guess if all you have is a thin client, this might be better than nothing. But then again, you can get an IDE and compiler for most languages on an android tablet. If you have a machine with a processor, it can compile your code.
I am often involved with multiple projects at the same time, but if those projects are set up well (and the toolset is decent), that does not cause any issues. In practice that means that developers always have to take into account that the project itself includes references to the specific versions of the tools that the project needs. And that those tools are -as-much-as-possible- installed in the context of those projects instead of in the context of the whole machine.
A concrete example in node world is to use "nvm" and to always install tools locally in the context of a project instead of globally. In dotnet world, use global.json to specify the version of the tools you want to work with, install dotnet tools locally in the context of a project instead of globally.
I have worked on projects where this is a problem. For example, projects that depend on hardcoded paths. Or projects that depend on hardcoded connection strings for a database. Or projects that do not use something like nvm to specify the version of node/npm they are compatible with.
Each of those things is an issue with the setup of the project and is something that you will bump against when you try to setup a CI/CD pipeline for the project. Although, I agree that it can be useful to use docker containers with a correct version of the tools to create builds in a build pipeline.
The reason that I prefer to develop natively is that it is faster, snappier and more enjoyable. I also believe that it is easier to diagnose issues and investigate performance. Aside from that, renting a virtual machine with the same performance as my development machine, would be a lot more expensive, and I would still have to deal with the lag. I am aware of recent changes that allow you to for example run a JetBrains IDE locally (similar to what VSCode already had for a long time?) that is connected with a remote headless version of the IDE. I do not have any experience with that kind of setup yet, but I assume that would remove most of the lag.
I used to use WSL 2 or a VM, but on a Linux host LXD is a really nice workflow. I can create a fully isolated Linux instance with all my dev tooling on it (optionally script/automate the install of all my tools), and with nested containers enabled I can even run _Docker in that LXD container_ so that when I type `docker ps` on that instance, and `docker ps` on my host, they each have their own set of containers running. For example, I have an ElasticSearch and Redis instance running in the dev box but Syncthing on the host.
Then I can use snapshots to back up the dev environment, or blow it away completely once it gathers too much cruft all without affecting my host. It's really great.
Pair it with VS Code remote SSH and you have a very feature rich setup with little effort.
I think a Dockerfile as a recipe for an environment is pretty elegant.
I've used LXC with proxmox and managing what's in a container is kind of like being a sysadmin.
This is vagrants' real superpower.
Docker packages applications, and should only really have one app inside it. LXC is similar to a vm in it presents itself as a stand-alone machine but uses the host kernel, it’s not like a vm in it’s not as isolated as sharing the host kernel.
So with LXC I agree, managing it is like being a sysadmin as that’s what it’s designed to be.
I use Docker for things that are stateless, maybe throw away, or just test an app quickly. I use LXC for things I want to run multiple services inside, more statefull, typically where people plumb a bunch of Docker images together in I’ll use LXC. The advantage in Proxmox is I tell Proxmos to backup my LXC nightly as it’s treated similar to a VM.
For making LXC feel less like needing to be a sysadmin, you can use Nix to build your LXC images and import in to Proxmox. Your LXC container becomes declarative and not to dissimilar to using a Dockerfile, it’s a far more powerful Dockerfile. What I’ve done is create a bare minimal NixOS LXC with some basic config and use that as a template then edit ‘/etc/nixos/configuration.nix’ inside the LXC on first boot. However as it’s just nixos you can build push the config remotely, use NixOps etc.
It’s a really good workflow using NixOS with LXC however it took me a while to get it as the docs are a bit thin and an old+new version of docs with the new version skipping things mentioned in the old you need to do, I.e change the tty to /dev/console to get a shell inside proxmox console.
I have started to run more and more software inside a VM for better security isolation, but the loss of performance is pretty discouraging. For things that are probably fine, I might be willing to trade some theoretical security benefits.
What has been your favorite local dev environment?
I personally docker-compose everything w/ .env overrides and I don't really feel like I've ever needed anything more than this.
I still wait for properly usable remote work environment that trumps local in performance and convenience.... I feel like the VS Code approach with server-client IDE architecture and the client running as a web application is going in the right direction but not quite there yet.., Especially if you can leverage lots of RAM and GPU power in your workflow and often work from different locations, don't want to carry your laptop around but still want to use the same power, environment & configuration on any machine you work from....
One of the best practices when using docker-compose to develop is actually to mount your code base as a volume. This lets you modify the code locally without requiring a complete rebuild of the container each time code is modified.
You might have to no-hup a service or run it not as pid1 (init container or supervisord) so it can restart without killing the container but that depends on the work being done.
I'm trying to find a good example to show you but I'm not finding much. If you have a medium account this might be decent, I don't have one but it starts off going through the process https://medium.com/swlh/tutorial-how-to-use-docker-volumes-t...
With that said I use literally the exact same containers that are pushed to production.
Does anyone know if there's a technical limitation to nested virt on EC2 or if it's just so people use .metal? I know Azure recently (1-2 years?) started offering it.
Containers have their place, they can be great, but I've tried the whole "containerize everything" approach, and it's no less time consuming than not containers, and now you have another Thing You Have To Learn (TM).
This isn't to say that virtualization is a bad thing for a development environment. It can be a very good thing, actually, but you don't need containers. A development environment, in my opinion, should be well organized, easy for a new developer to pick up, but also scrappy and adaptable to new and interesting situations. Containers, at least in their current conception, are antithetical to this. They're best kept to deployments or running things in a scenario where you know for sure you're not going to tinker with anything. I don't think I've worked on a software project where, at some point or another, I didn't need to reach under the hood in some unconventional way to investigate an issue or simulate some circumstance. Yes, you can do those things with containers, but now you have to deal with that complexity, which can get in the way by merely being confusing and indirect.
Having given up on using containers for development, when I want to use virtualization, I just spin up minimalist instances of Debian under Qemu (using the UTM GUI). This way, I can run a Thing (TM) in isolation, easily open up a shell, or SSH into it from the host shell, or start a GUI from within it if I need to, or install however many services I want within it and open up ports to the host, or have shared folders, etc. etc. etc.
But what if I need multiple of a Thing (TM) running at the same time? In that case, I just clone the VM. No Dockerfiles, no docker-compose.yml, no "volumes", no orchestration, no images failing to build because reasons, no qcow2 files blowing up with Docker For Mac (though that situation has improved), no pruning of orphaned containers/images, no worrying about whether your container should only be running a single process/service, and so on.
The only drawback to this is that, if the host is Linux, then it's indeed kinda dumb to be running other Linux kernels in VMs. But hey, it's 2023 and the difference in performance may not be meaningful depending on what you're doing. I know that, on an M1 Mac, I can run graphical applications at monitor resolution with incredible speed (perhaps moreso with Parallels instead of Qemu), so I'd struggle to care about any performance drawback if my host was Linux on the same hardware.
I'm sure that, for some, containers in development will be totally worthwhile. But if they don't seem worthwhile to you, dear reader, then that's probably because they wouldn't be.