Hacker News Reader

Comment by hacb | Hacker News Reader

Fun to read this comment, as I'm part of the team that deployed the exact same measure as for your wife: we now shutdown dev environments each day at 8PM. We decided to do so because over the weeks, we saw that a lot of devs "forgot" about their dev envs and they were up & running for 6+ months for absolutely nothing. On the other side, we are constantly requested to reduce infra costs, and dev environments (even unused) cost a lot.

In the end, I'm glad we deployed this.

View on HN

Couldn't this be done by giving the devs some transparency into how it's killing their team's budget and reminders, instead, so that they'd manage this themselves? Because this sounds super frustrating to deal with.

We kind of already have. We built a CLI that allows them to create/update/delete/list all their dev envs, so they have all the tooling needed. Also, we are still a small/medium sized company, so we often had the chance to talk with them to expose this issue, but in the end it never really worked.

> we are still a small/medium sized company

Sorry this is somewhat off-topic, but _why_ do you have virtual development machines?

We currently have ~40 microservices running (for the same number of devs), and having the full stack on a laptop is not possible. Especially with 80% of the devs having MacBook and Docker in a VM.

Also, this allow us (the infra folks) to chose how devs will work (partially), so it's easier for us to debug when they have issues because we do not have to deal with custom Makefiles etc.

Is it really needed to have 40 microservices for 40 devs? What I mean is that you generally scale out to so many services when you need each to have variable scalability, so each part of the system can scale up/down as necessary.

If you don't need this, all these services could be written as larger modules, with proper separation of concerns but easy to be deployed as a single piece. Collect some stuff that are similar in domain and create a well designed monolith. Easier for operations, easier for devs to maintain. Dependencies upgrades aren't a nightmare, or you need very good tooling to not create toil (which generally a small operation won't have).

Just throwing this in the wind because it sounds pretty heavy for a 40-body shop to be carrying as many services, at least I don't see the benefits unless for the aforementioned scalability advantage.

One example can be to target many different OSes/CPU architecture / GPUs architecture.

A weekly digest over email, or a dashboard on a real screen if you all work out or the same office, indicating costs and what's contributing to them. That will do wonders.

That will also add a lot of overhead/maintenance cost that we can't currently handle.

How much does it cost to have a single environment running overnight?

Not shutting down my machine saves me 2-5 minutes probably. Which seems like it might be worth a few pennies, but I hate the waste and environmental costs of wasted energy.

What we really need is a program that shuts off a devs environment, but then can boot it back up 5 minutes before they start working and somehow open all the applications that were running back to the exact state they were in before shutting down..

Initially, we were using kube-downscaler to do this. It stops a namespace at a given time and restart it in the morning. However, a lot of namespaces (= dev env in out case) were restarted for nothing, so we created our own tool that stops them at 8PM, and restart them when the dev runs a command with our internal CLI.

Not only would they be forgotten about, but without forced downtime/reset, they start slipping into production use. The dev asks someone to “try it on my server”, then the word gets out about this other server that works better and has more features. Before you know it 2 years have gone by and this “dev” server is a critical system the business needs to function.

In our case, those dev envs are actually feature envs: they are a copy of staging with only the microservice they are working on that differs. So it is impossible for those env to end up as prod stuff for us. However, each FE is an almost exact copy of staging, and each dev has 1-2 running FE, so it requires a lot of compute and stopping them has a real impact

Reply on news.ycombinator.com