Instant dev environments for cloud-native apps | Hacker News Reader

Instant dev environments for cloud-native apps

okteto.com

91 pointspchico8350 comments

“developers shouldn’t install Kubernetes locally as this requires hardware resources and deeper knowledge about this technology”

Totally disagree with that, developers should be sympathetic to the environment there code runs in, definitely spend the 5 mins to install kube through docker desktop and be done.. shorter feedback loop then too having it run properly local.

Ramiro here (co-founder of Okteto)

I agree that "developers should be sympathetic to the environment there code runs in". But that's about where the code runs in production, not where the developer runs it while coding. That's why we are pushing towards using remote clusters, so that you are developing on an environment that's as close to prod as possible.

In my opinion, Minikube, Docker for Desktop, Kind and others present two big challenges: First, it's not trivial to run them locally. They require extra memory, cpu and battery, they have issues starting/stopping (I don't use my local cluster that often, and I've spent way more than 5 minutes just starting/stopping/troubleshooting it). And, at the end, they have different components (load balancers, disk, network) and configuration from a remote kubernetes clusters, creating more dev-prod skew.

Second, the feedback loop is not that fast. You still need to either mount volumes (which is slow if you handle multiple files, like node's node_modules folder) or end up building containers to validate your changes. Sure, you don't have to push and pull them, but what if you didn't have to even build them?

Yeah I guess that might work for some workflows, we try and keep local as close to prod as possible, and how we build local artifacts that we test as close to how we build prod artifacts. Giving us a fighting chance of being able to repro prod bugs for example, if a team was making a different set of trade offs that would work.

Thinking about it more, we have had the problem of docker desktop using a bunch of memory and slowing productivity, so shipping that giant vm elsewhere would solve for that, can I spin up 16gb+ namespace/env in okteto?

If your kubernetes cluster can support a 16GB pod, it will work with okteto.

That said, the pattern we recommend to follow is to split it into smaller dev envs. That makes it easier to manage, and allows you to have dedicated dev envs for different parts of your app (e.g. your API dev env and your frontend dev env).

Ah I see, I assumed it was a okteto managed cluster. I should rtfm more =)

Splitting parts of an env is against our principles tho, put everything together and test against real endpoints where possible, there is no api dev env or web or app, there is just a dev env.

There's two parts to Okteto. The open source project (https://github.com/okteto/okteto), which launches development environments and works with any kubernetes distribution.

And Okteto Cloud (https://okteto.com) our development platform for Kubernetes Applications, that integrates everything you need to build cloud native apps (development environments, a build service, a private registry, SSL endpoints, etc...)

"When I am coding I’m focused on just one thing: delivering value to my users as fast as possible."

Sorry, this rubbed me the wrong way. This is almost a non-statement. It sounded condescending, but that may just be me misunderstanding your tone.

The trick is in how you define that value and how you deliver it quickly. You will probably get a lot more mileage out of building an understanding of what your users need than out of having a "blazing fast inner-loop".

Your users are developers and I'm not convinced you understand what we need. You don't talk about how we can handle service dependencies - How to keep a development database up-to-date and useful for each developer? Should there be one dev database per namespace or a single dev database that the whole team uses? How should third-party APIs be integrated into the development environment? And on and on.

Just running `kubectl create namespace` and rsyncing my code directory into a dev container in that namespace (?) doesn't even get me even close to a proper dev environment. It doesn't solve any REAL problems I have setting up a Cloud Native dev environment.

Agreed, this is a topic on my mind, we have a need for better cloud based dev environments so this article's titled appealed to me, but it wasn't answering any questions or aligning with common issues.

We would love to talk more about the issues you have. If you want to schedule a call just drop us an email.

Well first off, we don't touch Kubernetes, so probably not a product we're looking at either way.

Those are really good questions, we have similar requests from our users. We don't have general answers yet, but let me explain a few solutions we have in place. Data: data is hard. There are teams that workout the problem with fixtures and a database per developer. We also have a PoC for cloning namespaces and its data using Velero (this way, you can clone staging on your own developer namespace). We also have teams that have a dev database for all the frontend developers... Service dependencies: we push to define your dependencies using one or more Helm charts. This way, developers have a one-click experience to deploy the full stack on their namespaces. Once the app is running, developers use the Okteto CLI to put on dev mode any service and start synching their local code changes. Some parts of the stack might be shared by all the developers in a common namespace. Third-party APIs: this is also a wide spectrum, but our experience says it is easier to integrate them at the cluster level that in every developer workstation.

Iteration speed is sometimes an effective substitute for deep product insight. If you are a Steve Jobs you can go off and spend a few years building something in secret because you know people are going to love it when you show it to them. For everyone else, being able to ship stuff quickly offers a chance to maximize the number of spaghetti strands they can throw at the wall until they find one that sticks.

Github codespaces is a step forward to eliminate local setup and have replicable dev environments running in containers. But for true cloud-native applications, you would need a kubernetes namespace to run other services, databases or access the kubernetes API. This blog post explains our vision of cloud-native development at www.okteto.com

What's the performance like with compiled languages? I'm using rust and compiling in containers always seems slow.

I haven't tried rust, but I usually work with Go. The performance really depends on the resources assigned to your container. Okteto Cloud gives you 4 CPUs and 8Gi RAM. Also, it is important to persist dependencies and cache information using volumes to take advantage of incremental builds.

For whatever it's worth, I think your service (A) looks great! and (B) looks a lot like Heroku in a lot of ways.

One thing Heroku learned was that people kind of abused their free service. It used to work like this: You could create a free web service, and with no activity, it would "go to sleep" after (I think) 18 minutes of inactivity. Then it would take about 30 seconds to start up... fine for personal projects.

The abuse cut in when people set up (say) a raspberry pi to sit on their home network, and ping the service every five minutes, so it never went idle.

So they initially set up some punishment-like thing where your app had to be idle for some percentage of the day.

Finally, they made a MUCH better fix in that they changed the way the free tier works. You get (roughly) 730 hours of uptime a month, use as you see fit -- you can have one app running constantly, two running half time, lots if you only hit them occasionally.

So I can see someone setting up a Fred Developer account, and then setting up their raspi to do some simple deploy step every five minutes, to change something inconsequential about the app.

Also, what would 100G of disk cost? 1T? Thanks!

Thanks for your feedback. We changed our free tier a few weeks ago to avoid abuse, but we will probably improve over it in the following months. Our focus is on development, not on hosting production apps, so I hope we can do more drastically things than heroku.

Great work! I agree that developing in a Kubernetes namespace makes a ton of sense. This is also the direction we want to take the GitLab Web IDE. Kubernetes is the interface to the cloud, both for development, testing, and production.

Yes, the possibilities with this are very exciting. We have already integrated Gitlab runners in our Application Catalog, and we are working on a UX to deploy a branch and put it on dev mode on a single command.

Awesome. Please let me know if there is anything else we can do.

give jenkins x devpods a look[1]

[1]: https://jenkins-x.io/docs/reference/devpods/

It reminds me a bit of the monolith -> microservice -> monolith dance. Now monolith -> cloud-monolith -> monolith. :D Just kidding, but not just kidding.

Don't you tell me I'm living in "the dark ages" mister, I'm fully serverless. It's you who's grasping for a world soon to be gone.

I just find it to be patronizing, and it made me not want to read further. We have developers using Emacs and Lisp, we have Python devs, we have people using browser IDEs all under the same roof. There are any number of capabilities in the former that are missing in the latter, not least of which is "more complete control", something none of these products offer me in a convincing way.

Sorry if you find it patronizing, I tried to describe the problem in a general way to make it interesting for the general reader, independently of how we solve the problem.

Okteto, in particular, does not impose any IDE. It works with online IDEs, vscode, vscode remote, IntelliJ, IntelliJ remote... it is one of our core values. What we move to the remote cluster is the development environment runtime.

How do you develop your functions? Using Kubernetes as a development platform also applies to serverless, we have articles with Cloud Run and OpenFaaS: https://www.openfaas.com/blog/painless-serverless-developmen...

The idea is to have replicable and production-like dev environments, no matter your deployment tool

I develop in the cloud!

Serverless Framework deploys are now less than three seconds. It's almost exactly like `run watch` now!

I deploy to a dev AWS account, then a staging AWS account, and then promote to prod out of github.

Check it out: https://github.com/serverless-components/

That really depends on the size of your app. You also lose the ability to run debuggers. Not to mention how hard is to run serverless java applications, which is the dominant enterprise programming language.

That's not entirely true anymore. Several third party services offer debugging experiences.

Are there issues running java using serverless? I've worked with a number of enterprises using lambda and java. Seems to be fine ¯\_(ツ)_/¯ YMMV

Lambda layers, uploading caching and more make app size a non-issue.

Don't take me wrong, I am a big fun of serverless. We use it in production too. But I also think it does not cover a large percentage of the deployment spectrum, at least for a few years. And serverless can also benefit from tools like Okteto

Docker setup that allows me to test my functions locally - I have a wrapper that simulates the handoff between AWS ALB and AWS Lambda.

When ready, I merge into master, push, and they deploy to AWS through CI - I have simple Python scripts that manage Lambda Layer creation, environment variable configuration, etc.

Don't really need much more than this.

Right, that works. I think that setup does not scale well for large teams or if you are not able to run all your services locally due to resource constraints

...and then you remember that your app depends on half a dozen vendor webhooks...

why? we offer vanilla kubernetes namespaces. There are no hidden dependencies

I believe they mean their application relies on third-party integrations that would be burdened by such a fast loop

I mean its harder to route webhooks to twenty random dev machines than to a couple of DTAP servers.

There are challenges on this model for sure. A big advantage is that, by routing to the dev environments, you can have the dev interact directly with the webhook when they code, instead of having to wait for a shared environment to test integrations. This is pretty useful, specially if your application depends a lot on webhooks for it's main functioning.

This is an area where I'm particularly interested in. We are constantly experimenting in ways to better integrated it into our idea of remote development environments.

> you can have the dev interact directly with the webhook when they code

But how? A webhook needs to call a single endpoint (with a dns/ip pair). How can you route incoming webhooks from a 3rd party vendor to every dev machine running the service locally?

Weird people downvoted my previous comment on this, it's a genuine problem with this "DTAP on your machine" setup, isn't it?

There's a couple of approaches we've tried:

1. Register each dev env with the provider. Dev Envs can have predictable URLs (e.g in okteto cloud it's the name + namespace + domain), so you can directly register it. This works well with self-service webhooks like Github.

2. Have a proxy that routes to the right dev env. This works if there's a key you can use for the routing, like a user ID or the subscription.

3. Have a proxy that duplicates traffic and send it to all the dev envs.

I'm curious wow do you solve this on the DTAP env. Are you registering a single endpoint with the webhook? Then how does it reach the separate dev envs? (or do you have a single, shared dev env?)

Are kubernetes namespaces this well isolated? I wasn't aware it was at the point where you could put untrusted tenants next to each other in the same cluster.

You can achieve good isolation with a combination of RBAC, network policies, OPA rules and runtimes like gvisor. We also monitor suspicious activity with Falco. For deeper security, we offer Okteto Teams, which runs on a dedicated cluster, and Okteto Enterprise, which runs on your own cluster.

I deal with a lot of "works on my machine" bugs, but the reason is we can't go cloud native. Or cloud.

I'm really interesting in a cross platform, rootless and low overhead containerization solution for native desktop apps. I haven't seen one that fits my use cases yet.

Would something like Okteto Enterprise fit your needs? https://okteto.com/docs/enterprise

Unless I'm reading something wrong that has nothing to do with what I'm talking about.

I'm going to check out your product on the weekend. It looks pretty neat. But it would be cool to hear a bit more about how you achieve some of your stated claims. Particularly working with debuggers. The kubernetes/minikube setup at my current role seems downright hostile to debugging -

* We need to maintain separate profiles for running apps in debug mode, to expose the java debug port to the host and supply the right JVM options

* Latency - even to a VM running on the same computer when compared to running a local process - means a significantly worse debugging experience

* Don't leave your debugger paused for too long, or your healthchecks will fail and your app will get rebooted from under you

* And forget about trying to debug something that happens on application boot

For debuggers, we wrote a few guides like this one: https://okteto.com/blog/how-to-develop-node-apps-in-kubernet....

> We need to maintain separate profiles for running apps in debug mode, to expose the java debug port to the host and supply the right JVM options

Yes, you should. This is something you can do in the 'okteto.yml' file. You can specify different environment variables, different commands and even a different image (e.g. an image with your dev tools and debugger, which you don't need in prod).

> Latency

This can be a challenge, for sure, specially if you have a bad network connection. In our experience, debuggers and incremental file sync don't require too much bandwidth.

> Don't leave your debugger paused for too long

This has happened a lot to me. In the latest version we introduced a switch to disable healthchecks during dev time so avoid this.

> And forget about trying to debug something that happens on application boot

This is the trickiest definitely. I've had luck by pausing the debugger on attach (on go and node you can do this), but it's not the most optimal.

I didn't get around to checking out your product yet, but from your comment and the guide you linked it doesn't sound like you've actually solved any of these pain points in a meaningful way.

It would be REALLY COOL to have a one- (or even few-) click IDE integration to attach-debugger-to-instance-deployed-in-Okteto. If you had that, I would already be pitching your product to our head of engineering and asking our head of security to expedite due diligence.

edit: put another way, the pain points you DO currently solve for are ones I don't currently have, because I'm already mostly avoiding them by not running my dev instances in Kubernetes unless I really need to.

Step out of the dark ages by burning piles of cash to generate light.

One of the advantages of using Kubernetes vs a remote VM, which is an approach followed by companies like Stripe, Slack, Eriksson and much more... is that Kubernetes is very efficient allocating resources. Idle dev environments don't consume resources, they can be scaled to zero and restarted in a few seconds, and the same infra is shared by your entire team

Reply on news.ycombinator.com