ContainerSSH: Launch containers on demand | Hacker News Reader

ContainerSSH: Launch containers on demand

github.com

237 pointsgaocegege66 comments

Nice! Will give this a try. My current setup is a vscode docker container with every dependency and plugin I need for development. It runs on a powerful server.

It is SO nice to come back where you left from multiple machines. Also it is very nice to have a 10 gig connection. It works incredible well. With Scala and the metals plug-ins it is very close to a full blown intellij experience.

Your setup sounds a lot like gitpod.io

Yes but everything "self-made". Been working like this for years now.

Also have arch + i3 + openrdp dev containers which work awesome. Even with multi-monitor support. The only thing missing is gpu acceleration. Then it would be perfect.

sounds intriguing. is there a blog post with your setup? can this be (easily) used between several developers? each needs their own server? can the server spin on-demand?

> arch + i3 + openrdp dev containers

Any chance you've published the build files somewhere? This sounds fantastic

Try xrdp and xorgxrdp (with --enable_glamor) for gpu acceleration of x-windows in containers. I have successfully tried it on integrated intel and radeon GPUs with opensource drivers.

https://github.com/neutrinolabs/xrdp/issues/1029

It seems (from the material on the site) that the motivating use case for this isn't development, but rather running computer labs for students.

If did understand it correctly, it can be used to auto deploy a container on an ssh host, start the cli and once you exit, it will clean everything up. This would be very usable for developing certain things.

You did understand it correctly. ;)

For sure, it could be used that way. But, a lot of people were comparing this unfavourably to how they currently manage their dev workflows. My point was that they weren't really the target market.

I think this is the workflow most people should be moving to: immutable development.

If your dev environment's state mutates and deviates from the rest of a team's (or away from production), the deviation introduces bugs. This tool lets you "start fresh" with the correct environment every time, but (I imagine) also lets you update that shared environment by pushing a new container.

In this way you ensure everybody is developing off exactly the same thing, and can easily update that shared environment, but nobody clobbers anybody else's WIP, and you can test new changes in your own container, even persisting your own configuration using persistent volumes.

Sadly, this does not support port forwarding, which is the final feature necessary to do remote development (run your web app remotely, see it in your browser locally).

If you build from the libcontainerssh sources on GitHub it does. We just haven't released it yet. :)

Is there some docs on how to use it after I've compiled it? I'd like to try it out... Evaluating solutions like these right now, could be really useful for us

For VS Code you will need to build your own binary because it needs port forwarding which isn't released yet.

Edit: also, ContainerSSH tears down the container when you disconnect, so that may not exactly be what you are looking for.

Hey folks, one of the authors here. The website contains a lot more information than the GitHub page: https://containerssh.io

If you have any questions, I'd be happy to answer.

Sorry I did not ask you to post the project to HN. I just found it interesting.

Oh, no worries, thank you for posting it, we are all very excited it made it on the front page. :) I just wanted to point to the website as it contains a lot more info than the GitHub repository. :)

From a quick glance it's unclear if there are any ready-made auth servers for this? Eg, ldap, sso for o365 or Google?

Currently, there is only the test server that's available. In the next version we will support direct oAuth2 / OIDC integration. It's a little ways out, but testable with Google at the moment. Here's an early demo from when we started working on it: https://youtu.be/SGHee9cV_rA

Slightly related: Is there a similar ssh auth system for regular openssh? One where no client software is needed and to login you use a browser to authorize the login based on Google/GitHub identity?

Yes, there are several PAM integrations, some more rudimentary than others. The most developed seems to be this one [1], but I haven't tested it. However, it only supports the device flow, so it's limited to 50 logins per hour on GitHub.

[1] https://github.com/slaclab/pam_oauth2_device

Love the concept of the explainer video. But am I the only one that didn't understand what this does? I had to watch the video twice and really read the readme to finally get it.

Am I understanding that this is a fancy way of doing: - SSH to server - Create docker container (or Kubernetes) - docker exec -it container /bin/bash But in 1 command & auto cleanup?

Pretty neat. Not sure I have a use for it though. Would've loved to see more use case in the docs (along with making the video explain the project better since it's a great concept!)

In a nutshell, yes. However, the main use case is when you don't want to give people access to the container environment, instead you want to drop them in a restricted shell. Think jump host. :)

there is no use for this, or most of the stuff around docker/k8s that's posted here. It's all just a fad. And people keep mindlessly pumping out more code.

you may downvote my comment, but you will never juice out anything useful from most new age "technologies".

Why are you even in this industry if you hate it so much?

who says im in "this" industry?

Usually to be a regular on HN you're in the startup/web world in some fashion.

I guess there's people out there who like to complain about stuff they don't use?

Awesome! I wonder if I can use this to replace jupyterhub for my small data science team. The jhub kubernetes architecture is (roughly) -

Proxy —> hub —> spawner based on selected image —> container with /user persistent PV

But the image needs to be spawnable by jhub (i/e it needs to have some variant of jupyter-server-proxy installed), and currently that limits you to JupyterLab, RStudio and openrefine.

If you can port-forward with containerssh you could bypass the whole jhub proxy / hub / spawner rigmarole and simply

Select image -> containerssh w/port forward -> start the desired dev environment from inside the container -> get to work.

Then you’re not limited to jhub-compatible environments, and you don’t have to manage the complexity of jhub.

Yes you can and you wouldn't be the only person doing this. However, port forwarding is only supported in the in-development version, so you have to build your own binary for that.

Recently, I found another tool built for machine learning environment: https://github.com/tensorchord/envd. It integrates with buildkit and can share the config like Python functions cross the team. This can reduce a lot complexity for data scientists.

For the “jhub but for any container that speaks HTTP” use case, you might find our Spawner project interesting: https://github.com/drifting-in-space/spawner

We don’t have support for volumes yet, but I’m open to ideas.

For those that prefer a local environment that's disposable but still supports most graphical apps as well, there's Fedora's Toolbox for pet containers.

https://docs.fedoraproject.org/en-US/fedora-silverblue/toolb...

This is really cool and interesting, appreciate the post.

Sounds a lot like VS Code devcontainers https://code.visualstudio.com/docs/remote/containers

Not exactly, VS Code devcontainers work by directly interfacing with the container environment. If you don't trust the connecting party with direct access, you can use ContainerSSH. One use case would be a shared development environment.

I'm struggling to understand why this is useful. How is this useful from doing a docker exec, or kubectl exec?

It was born from a need in the webhosting sector. I wanted to give users shell access so they can do things like git pull directly to their website. Without containers it's difficult to constrain a user. The first SSH server I build along these lines involved the components what make containers today: chroots, cgroups, etc. (That was more than 10 years ago.)

Over time, more use cases have developed, most of them around the need for jump host or lab access. There has also been some research done into SSH attack patterns using this as a honeypot, which will hopefully be published soon. See: https://containerssh.io/usecases/lab/

It is difficult? It is something most web hosting did way before container existed. You just create the user on the system (well, probably have the user in LDAP) and give the user the permission to access only its files and possibly execute only a number of trusted programs (or not execute programs at all, and dive only SFTP/SCP/git access and not a shell).

Containers may give some false sense of security tough, a lot of people doesn't understand really that escaping a container is not such a difficult thing, since it pretty new stuff and bugs are being discovered, also the container may be configured badly, while the UNIX permission mechanism is around since forever and it's pretty solid (not that in the past there weren't bug that bypassed it, but the same bugs may as well be used in a container anyway).

Most webhosting services barely managed to offer encrypted FTP/SFTP before containers existed, most went with plain text FTP. When I worked in the sector back 2011 we had a customer with a hacked/stolen FTP password every other week.

As far as features are concerned, yes, you can make a git server you can push to, but allowing users to get a full shell and pull from their git server is a whole other comfort level. More modern alternatives would include on-server development with VS Code, which we aim to support in the next version with port forwarding supported.

As far as container security is concerned, if properly configured, these are still a sight more secure than trying to give users shell access without them. The UNIX permission mechanism is woefully inadequate for keeping users from messing with each other's stuff. This is obviously not a problem if you don't want to provide a shell service to users, but some services, like the LX Plus service at CERN mentioned in the other thread [1] is specifically that: a shell service for users to access.

One of the problems containers (or more accurately, network namespaces) solve are the language or development servers users may start for their development needs. These often don't contain any extra security beyond binding to 127.0.0.1. However, on a shared server this is obviously not enough.

The other problem with using purely UNIX permissions to isolate users in the webhosting sector is users messing up their permissions. Back in the day this was also a constant problem, so nowadays all webhosters run the webserver / PHP-FPM instance with the same user ID as the user uses to upload their code, often having multiple websites for the same user using the same user ID. This lends itself to cross-contamination between sites if one is breached. If the sites run on a different user ID, however, it becomes more difficult for a user to move data between them.

[1] https://news.ycombinator.com/item?id=32790856

Indeed, I let anonymous people run arbitrary code on my system... though I haven't bothered to make it available over SSH.

https://rkeene.dev/js-repl/?arg=bash

It creates a secure environment on every connection, though it doesn't use cgroups, just chroot, resource limits, and other boundary protection mechanisms, etc.

In one sentence: Everybody already knows how to use SSH, only the infra department knows how to use kubectl.

Don’t underestimate how much pushback and struggle even such a small tool addition can add to developers with already too many other things on their mind.

This enables the whole dev org to use ephemeral workflow from k8s, without having to understanding k8s.

SSH also comes with a lot of tooling integration that is not as readily available as for k8s. Tunneling through anything, IDE integration or sshfs to name a few.

Granularly locking down a k8s cluster can also be a challenge. It’s easier to just block your k8s api server from anyone who is not admin.

Otherwise in principle you are right, kubectl should on paper be equally or even more capable.

I would rather have a Nix devShell.

Is there a good solution for devshell with associated services? The app itself is fine that way, but what about the database, message queue, seed data, etc.?

I usually write a script that idempotently spins these things up locally, and use the shellHook to call that script. That way whenever I cd into the dir the first time, direnv loads the nix-shell, which starts the db, message broker, etc. I also have a shutdown script that I run when I manually want to shut things down. Works quite well.

For more advanced use cases, such as k8s with tons of services, you're better off using something like tilt.dev.

You can install any services in your nix dev shell environment. Run them however you desire--a bash script that backgrounds their tasks, systemd user services, supervisord, or any other process manager.

edit: A really simple and effective process manager is foreman and its procfile format. I like the golang version of it (simple single binary with zero dependencies): https://github.com/ddollar/forego or https://github.com/mattn/goreman

So I'm aware you can script anything you want. I meant a more integrated version. Now I'm tempted to write one myself where you provide the (for example) mysql you want to run and it automatically inserts the right scripts...

I've been using https://github.com/F1bonacc1/process-compose with great success.

It's a userspace process orchestrator/scheduler that works across all relevant platforms, supporting daemon processes and k8s style readiness/health checks.

In combination with nix flakes, it quickly reduced my projects docker-compose usage for easy-to-configure services.

This gave huge performance benefits for the M1 Mac folks on my team especially for CPU intensive processes thanks to native binaries.

For maximal ease of use, the remaining docker-compose containers are started/stopped as a process-compose task. Quite meta :)

I would, too, but the barrier to entry is much higher.

This has been discussed ad-nauseum, but "getting started" with nix (or home-manager, or devshell) is an experience worse than trying to do anything with Linux in 2005.

Cobbling together a bunch of blogposts and docs which kind of, sort of, might let you piece together a config which works after enough iterations and unreadable backtraces is far more difficult than it needs to be, and that's before you get into cases like "I want Python linters in my environment, so I need Python, but I also need access to some modules from Python on the system, which aren't installable (let's say `python-apt`, which cannot be installed from pypi), and..."

For probably the 500th time someone has said it on HN this week, nix will never be really usable or gain wider adoption until someone puts together solid documentation, or even a solid "getting started -- intermediate" guide which can bridge between "let me set up a trivial env" and "I'm running nixos and it's nix all the way down"

I like the idea. With this, I can do something like - a Golang IDE container for go projects, a Javascript IDE container for frontend projects and so on.

If this works with Docker, it can work in AWS ECS with a bit of wrapping, right?

Yes, your config server can pass the Docker connection parameters directly, so you can scale it. I believe you could scale AWS ECS up and down as needed and then expose the Docker socket with TLS to ContainerSSH.

Would this work with Fargate? I don't think I've ever seen any Docker (or alike) API there that can spawn new tasks.

I wonder how hard it is to make a custom backend...

No, Fargate has its own mechanism for this as far as I know. However, if this is a usecase for you please open an issue and we'll look into writing a backend for Fargate.

One of the plans after the current release is creating a pluggable backend system. This is currently not possible and backends are, unfortunately, fairly hard to write.

The one minute video doesn’t load for me

It should link to this YouTube video: https://www.youtube.com/watch?v=Cs9OrnPi2IM

Yeah, but I get this error:

https://ibb.co/TYjY8Y0

I just went through the YouTube settings, nothing there indicates anything that we can do about it. I also checked in several browsers and devices to no avail.

I'm afraid, this may be something where YouTube isn't working as intended. (For example, I had problems with YouTube in Firefox on Linux for quite a while and needed to disable tracking protection entirely to get it working despite paying for Premium.)

Thanks for the trouble. I’ll try again later

I uploaded the video to Google Drive, maybe this works for you: https://drive.google.com/file/d/1tIf86ba68Bc1RbWkVgrXlXeIFMD...

The regular YouTube link started working, go figure. Thanks anyway

Thanks for the feedback, I'll remove it from the drive then.

Or just use SystemD with nspawn.

You can, but it be comes quite difficult to secure and manage very quickly. Nikos talks about the SSH use case at CERN in this video [1] and mentions that they tried quite a few things before ending up with ContainerSSH.

[1] https://youtu.be/d6aCBiInMbg?t=525

Thanks for this! This explains the project better than the GitHub site

Yeah, we've gone through a few iterations with the README on GH, but it's really hard to explain the project in the length available there without going into only a very specific use case.

Reply on news.ycombinator.com