Cockpit: Web-based graphical interface for servers
cockpit-project.org
cockpit-project.org
Sure, clickops is no way to run a server - but neither, if we’re honest, is ssh.
For a working machine, server state should be reproducible from scratch. Install an OS, add software, apply configuration, leave well alone. If you’re going in with ssh or cockpit you’re just going to screw something up.
So the only reason you should be working on a server directly is because you’re doing something exploratory. And in that case gui vs command line isn’t as clearcut as people want to make it. GUIs emphasize discoverability and visibility which can be helpful in that experimental phase when you’re trying to figure out how to get something set up right.
server state should be reproducible from scratch
Why? I'm not necessarily disagreeing, but too often are these kinds of statements thrown about without any qualification, as if they are self-evident truths. But they're not -- there are engineering trade-offs behind any choice, and it's no different here. So, in order to guide this discussion away from dogmatic platitudes: why should server state be reproducible from scratch? What does "from scratch" mean? Why is clickops no way to run a server?
Install an OS, add software, apply configuration
Do you think this captures "server state" completely? Software patch levels are not part of server state? What about application data? User data?
So here's my counterstatement: for any working machine, I can reproduce the server state exactly by performing a restore from backup. Backup/restore is perfectly compatible with clickops, and it's faster and more reliable than reinstalling an OS, adding software and applying configuration -- even when the software and configuration are scripted. And if your server stores non-volatile data, as is often the case in clickops environments, you will need to have a backup system anyway to restore the user data after deploying a new server.
It's because different people think in different levels of abstraction. One admin might be thinking about a handful of servers and another an entire fleet of VMs. The way you manage each is very different. Clickops can work well for a small number of servers and a full orchestration setup can be over engineering.
But your real issue is that blanket statements never work in such scenarios. However, I think it's pretty well established that reproducible server state is a best-practice. How you get there is up to you.
But as an argument against backup/restore -- you can't use backup/restore to generate new servers from an existing template without some kind of extra scripting (if for no other reason to avoid address/naming conflicts). And if you're already scripting that...
Changes done are only captured at snapshot intervals and are no coherent and atomic, so you can easily miss changes that are crucial but capture destructive changes in between deltas. Worse are flaws that are introduced but not observed for a long time and are now hopelessly intermixed with other changes. Reproducible build systems allow you to use a revision control system to manage change and cherry pick changesets to resolve intermixed flaws, and even if they’re deeply intermixed you can resolve in an offline server until it’s healthy to rebuild your online server.
The issue with reproducible build systems isn’t they aren’t superior to backup and restore in every way. It’s the interfaces we provide today are overly complex compared to the simple interface of “backup and restore,” which despite its promised interface always works in the backup part but often fails in the restore. These ideas of hermetic server builds are relatively new and the tooling hasn’t matured.
I would say actually click ops is an ideal way to solve that issue. Click ops that serializes resiliently to a metadata store that drives the build and is revision controlled solves that usability issue. If the metadata store is text configs and can be modified directly without breaking the user interfaces would be necessary to deal with the tedium for complex changes in a UI, while providing a nice rendering of state for simple exploratory changes. Backup and restore would be only necessary for stateful changes, but since the stateful changes aren’t at the OS layer, you won’t end up with a bricked server.
But regarding the topic at hand, I don't think being able to manage these things with a graphical interface is necessarily a bad thing. It's basically user-space iDRAC/IPMI.
I'll spend less time just setting them up by hand.
The company will survive a few hours of downtime.
And, while with the analogy of pets, when you are on holiday, allow your neighbors to look after your pets?
The reason that (some) people don't do this is the cost/benefit analysis looks kind of weird. You'll spend a lot of time mucking around in puppet/chef/ansible/whatever for a single snowflake server, and it would be a lot faster to just go edit that config file directly.
In reality, proper backups and shell history can get you pretty far if you ever find you need to replicate a snowflake.
Also Ansible is incredibly useful at my work and there's a very large overlap. Which is obviously the main motivation.
I'm curious if you have a specific tool or tools in mind. I've been using Ansible in my home lab, particularly for configuring Raspberry Pis. The OS install part (only?) works because it involves a bitwise copy of the image to the boot media (and some optional configuration.)
When I say ‘working server’ though, I typically mean one that is doing a job - providing a critical business service.
A ‘home lab’ of raspberry pis is a different beast.
I presume you only run NixOS then?
A good server process is idle when nothing is happening, and should be using miniscule real memory that should be easy to swap out. If the server in question uses significant memory for your use-case, you also don't want it starting on demand and triggering sporadic memory pressure.
It does make it easier to avoid blocking on service start in early boot though, which is a common cause of poor boot performance.
One is boot performance. Another is zero cost for a rarely used tool, which may be particularly important on a VPS or a small computer like a Raspberry Pi where you don't want to add costs for something that may only rarely be needed.
I think a nice benefit for an administrative tool is the ability to update it, and reload the updated version. You don't need the tool to have its own "re-exec myself" code that's rarely used, and that could fail at an inconvenient time.
The reason why inetd didn't stick is because it's a pain to use -- it's separated from SysV init, so it needs to be very intentionally set up. Plus there was the inetd/xinetd disagreement.
Tying in init, inetd and monit into a single system that can do all those things IMO made things much nicer.
Zero cost is only true for unused services. For rarely used services, it's a rarely occuring full cost that might come by surprise at a bad time.
> I think a nice benefit for an administrative tool is the ability to update it, and reload the updated version.
This is only a benefit if the systemd socket unit is co figured to operate in inetd-mode (Accept=yes), where systemd spawns a new process for every accepted connection, which is quite inefficient resource-wise.
"Normal" systemd socket activation just starts the service and hands over the socket. The service runs indefinitely afterwards as if it was a normal service, and needs to be manually restarted or hot-reloaded after upgrade or configuration change.
> The reason why inetd didn't stick is because it's a pain to use -- it's separated from SysV init, so it needs to be very intentionally set up.
Being separated has a lot of benefits - easy nesting, easy reuse in minimal containers, etc. The integrated model works best for monolithic servers.
1. Start the daemon on boot and have it running all the time, like some undereducated neanderthal.
2. Configure your system to run a daemon which monitors a port/socket and starts up only when there is traffic, like a civilized person.
I believe which one of these to use is highly dependent on your resources, usage, and deployment model. For services that are fast and cheap to start but are rarely used, #1 makes more sense. If you have a server or VM which only does one thing (very much the norm, these days), then running just keeping that service running all the time is easier and better for performance.
FTP, SMTP were all stateful, so living under inetd worked OK. One process per overall session rather than individual messages within a session.
Obviously, inetd could have been hammered on to basically consume the pre-forking model then dominant in something like Apache, caching server processes, etc.
But it wasn't. Then databases became the other dominant server process, and they didn't run behind inetd either.
Apache + CGI was the "inetd" of the web age.
https://discourse.ubuntu.com/t/sshd-now-uses-socket-based-ac...
"On upgrades from Ubuntu 22.04 LTS, users who had configured Port settings or a ListenAddress setting in /etc/ssh/sshd_config will find these settings migrated to /etc/systemd/system/ssh.socket.d/addresses.conf."
It's like Canonical is doing 1960's quality acid.
At least the garbage can be disabled:
"it is still possible to revert to the previous non-socket-activated behavior"
With having to remove snapd then mark it to not be installed and in the next Ubuntu having to fix ssh back to the current behavior, it might be easier to migrate my servers back to Debian, or look for a solid non-systemd OS.
There is no reason every single application should manage network socket acquisition on its own - I'm not very fond of the times everyone and their mother wrote whacky shell scripts to start and stop their services, either. But somehow those seem to be the "good old times" you guys miss.
I.. uhh.. Yeah.
If your systems are more pets than cattle, then I think I too would prefer an always-running ssh daemon. If your workflow is only to ssh into machines during bootstrap, however, then having sshd run only during initial bootstrap and then shut itself off does seem like a nice way to free up a small amount of resources without stopping or disabling the daemon post-bootstrap.
SSH should probably be running 24/7 on any server(to keep those resources allocated for maintenance access), but if it's my workstation with a monitor - then it's a non-issue.
If I could offer a little advice: The systemd man pages are useful as a reference, but are terrible to learn from. Part of this is because there are parts of systemd that everyone uses, and there are parts that almost nobody uses and it's hard to guess which these are at first. Also, the man pages are dry and long and quite often fail to describe things in a way that would make any sense whatsoever to someone who isn't already intimately familiar with systemd.
Most of my systemd learning came from random blog articles and of course the excellent Arch wiki.
I'm happy to run it (aka: have it installed) on all my little raspberry pi's, because sometimes I'm not at a terminal when I want to scope them out, and/or if I'm at "just a web browser", being able to "natively ssh into them" via a web server (and then run `curl ...etc...` from a "real" command prompt) is super helpful!
Do you essentially mean that systemd socket activation is used basically only if/when the Cockpit web app end-user/client sends a REST/GQL/etc/? request for logs, for example?
systemd passes the socket on to the application so I don't think it has any reference to it anymore, so it wouldn't be able to know when the socket closes.
The next big thing will be a web server where you don't need to use the command line to deploy your project, just sync your workspace folder and it will automatically execute the file matching the URL.
I have watched startups fold for not pushing product development further into UI/UX with off the shelf backends. At one company I worked at I showed how our backend (completely custom container orchestrator) could be replaced in a weekend with AWS Lambda and ECS. But our UI/UX and workflow tools would take much, much longer. Yet we continued to waste money and time on "building a new raft based cluster". In the mean time I was handed "add batch processing" and we already used Go so I just used Nomad under the hood and moved on.
I like working on teams that ship features not JUST tech for tech's sake.
https://git-scm.com/book/en/v2/Git-Internals-Plumbing-and-Po...
2021, 128 comments: https://news.ycombinator.com/item?id=26197510
2018, 149 comments: https://news.ycombinator.com/item?id=16445612
Cockpit is okay but it's basically Red Hat's equivalent to the Windows Server Manager tool, and I have no doubt it was directly inspired by Server Manager. It's development and improvement over the years has been painfully slow.
Nobody who is comfortable with an ssh session uses Cockpit, except maybe to create new VMs, and even then all of these comments comparing it to Proxmox are just whack because it doesn't have a quarter of the features the Proxmox UI offers. The utility for managing VMs is a recent development and even then I still prefer the Virtual Machine Manager tool because I don't want to deal with the latency increase and other limitations of working through a browser.
But anyway, there's a ton of things you can't do with Cockpit, and never will be able to do. It's for people who want to point and click, can't do a bash for/while loop, don't understand pipe chaining commands, and don't like using vim.
Like happyweasel said, it's basically webmin for Red Hat.
It's kinda cool, but it's so old now and development has been so slow and it's been so over-hyped that I don't pay attention to it at all and I've never used it except what was required to get certified.
"Instagram filters are for people who don't know how to work with Photoshop layers, don't understand basic color blending operations, and who just want to swipe."
I mean, yes.
But I may have a problem with people who can't do a bash for/while loop or understand pipe chaining commands be responsible for administrating my company servers.
I don't see how the comparison between adminstrating servers and sharing pictures on social media is a useful comparison.
I suspect most people here used the HN web interface to post their comments, even though constructing an HTPP request to send to HN using curl to post your comment wouldn’t be significantly harder. The fact that they used the point and click HM Web UI doesn’t mean they’re incapable of constructing such a request where it actually is needed.
I don't think like that at all. Of course someone can know bash for/while loop and still choose to use a point-and-click tool to administrate their servers. Nothing wrong with that.
Not sure where in my message did I say that if someone chooses to point and click, then they don't know what a bash for/while loop is.
What I said in my comment was that if someone doesn't know what a bash for/while loop is, then I may be uncomfortable adminstrating my servers.
My point was to show that comparing adminstrating servers and sharing pictures on social media is not a useful comparison.
Reminds me of a colleague I met once who was happily boasting he's a 'real' HTML developer because he writes everything in Microsoft notepad. I took a look at his work and cried inside - broken tags, bad formatting and of course he didn't escape all his characters correctly.
Being able to type some characters on a keyboard does not make anyone superior to a user that uses a point+click interface.
There isn’t a causal relationship, but the two are heavily correlated. Getting deep into the details of most server software means doing so via CLI, because the people who built the software intended it to be used there, since servers are usually headless.
Even if you run a DE, chances are high you’ll still end up editing some config files in a text editor, and running commands in a terminal emulator.
It’s a bit like [n]vi[m] - learning its esoteric commands doesn’t make you good at coding or ops, but there’s a decent chance you wouldn’t bother to spend the time learning it unless you needed the speed increases it grants you.
Anyway, what has that got to do with my point though that sharing pictures on social media is not a meaningful comparison to make with something like administrating servers?
idk what kind of legacy orgs people in this thread work for where there are Linux sysadmins working for their companies which directly manage server configs and security settings. "Administering my servers" is an interesting phrase in 2023, because there often aren't any servers to directly administer (even a VPS is hard to find). The last few companies I worked for all used some kind of virtualized infrastructure and usually through some kind of declarative interface (Docker, Kubernetes, or some terraform-style tool). Certainly, the people who OWN the servers have sysadmins managing these things, but such things are an abstraction these days, where many organizations don't have to deal with CLI configuration and bash, and just leave infrastructure to devops or even the developers themselves
I don't know if you consider Amazon/AWS "legacy org". When I worked there I didn't think it was a legacy org. Yet they needed skilled Linux sysadmins. Sure they are called by fancy names like infrastructure engineer, production engineer, etc. but the work they did used core Linux kernel skills, scripting skills and programming skills, just to name a few of the skills.
A few years later I worked for another cloud provider and it was no different. I don't understand why you think only legacy orgs care about good system-administration skills.
Which unfortunately is a problem in itself. A lot of core knowledge is lost and many people running infrastructure don't know how to read actual logs and debug outside of what the GUI shows. Just like you mention it's not even VPS these days it's a Dockerfile pushed to some cloud.
Copy/paste a Dockerfile, edit some yaml for the CI and claim you know DevOps. It's a shame really. I don't mind a nice minimal GUI when it makes sense, but it's important to understand the pieces below it.
Plenty of k8s users are accessing their services and logs through CLI. It's just that the relevant logs are from the application and not whatever the underlying infrastructure is. It makes it way easier to have amazon deal with the underlying server and just let me focus on the actual application. Saves labor costs (in theory) by not having to hire an IT guy, and you can put more faith in the security practices of the cloud providers than in your own organization.
Your criticism reads like old school developers complaining about new devs learning Javascript without learning C or Assembly. Technology progresses, the set of baseline skills required to do your job changes. Few software engineers know anything about hardware or electrical engineering, but that used to be a requirement many decades ago.
> Copy/paste a Dockerfile, edit some yaml for the CI and claim you know DevOps. It's a shame really. I don't mind a nice minimal GUI when it makes sense, but it's important to understand the pieces below it.
What you've described is a way to make it dead simple for engineers to develop code without having to interface with a human being (a sysadmin) in between. It makes deployments consistent and easy. When you add CI/CD into the mix, you don't even need to run the command anymore, you just merge the master and swap your staging and production instances. Amazon can hire the sysadmins, the client can hire engineers.
Valid point, but I'm a javascript dev tired of helping my peers figure out their tools when they're not even interested in discussing lower level topics on a normal day. I wish I had some grumpy old-timers around me to learn from.
> It makes deployments consistent and easy.
Consistent on the platform chosen at start, if you run into egress costs you might already be locked in to that platform and migrating AWS container format for every service might take time for those who don't understand what it does exactly. One simple mistake can be really costly if you forget to set limits. It's dangerous to put too much power in the hands of people who don't understand the possible consequences.
Don't get me wrong, I like modern software dev and day to day tasks should be easy. But too many people are lazy and uninterested in the details behind their craft. "Just install half the universe, why bother reinventing the wheel" is too common. You're not a DevOps unless you can setup and manage the workers on a CI on bare metal, you shouldn't always go that route in production though, but the knowledge is important.
It’d be great if JS devs could learn JS, but I’m not holding my breath.
Your comments read like someone who doesn’t believe you need to understand how the abstractions work, which tends to end in failure (or extremely high cloud bills).
Abstractions leak. They’re great, but you still need to know how they work.
You could have godlike Photoshop skills and it will take orders of magnitude more effort to get results. With SSH and shell scripts you'll likely be faster than the web UI once you're skilled enough. And it's easy to automate.
> It's for people who want to point and click, can't do a bash for/while loop, don't understand pipe chaining commands, and don't like using vim.
Lol! Are you ready to deploy an ssh-capable terminal emulator at all times? What's wrong with making simple tasks simple?
I run multiple Raspberry Pi cameras (with nicer camera modules) to watch the pets if the family travels. The RTSP camera streams run in a systemd unit on their boxes. I have some healthchecks to make sure packets are being streamed as other systemd units. Each camera gets its own private IP on a ZeroTier network I manage. Since copilot is only run on demand, it's a no brainer to have around for administration.
Sometimes one of the cameras just starts streaming out blank frames. I'd much rather manage this through a copilot web interface on my phone when I'm on vacation than find a keyboard to use SSH with and restart the camera stream unit. I mean sure, I could write a healthcheck which checks whether blank frames are being emitted, but it's just so much easier to restart it via copilot than it is to write that healthcheck and it only ever happens a few times a year. Shrug.
What terminal emulator isn't ssh-capable? Where would you not be able to open a terminal emulator? I am so confused.
> What's wrong with making simple tasks simple?
The limited tasks exposed by cockpit are also simple (or depending on the individual, simpler) in a terminal, but if you want a point-and-click UI for just a few things, go ahead.
That cockpit is very limited and seemingly has no future does not mean you can't like what it does now. Just might be worth considering if there are better-supported alternatives.
A mobile device. I don't use terminal emulators without access to some non-touch keyboard so I want a simple interface on my mobile device.
> That cockpit is very limited and seemingly has no future does not mean you can't like what it does now. Just might be worth considering if there are better-supported alternatives.
Sure then compare cockpit with other webmin-esque tools, not a terminal emulator. These are different interfaces, much like I don't compare a voice interface with a mouse-oriented one.
Whether you want to use a terminal emulator on your phone doesn't change whether it's available.
I wouldn't want to use a clumsy web app with dangerous buttons that put me a single mis-touch on a poorly thought out touch target away from messing up/rebooting/powering down a server from my phone either, but that doesn't change that cockpit is available.
> Sure then compare cockpit with other webmin-esque tools, not a terminal emulator.
You compare it to the options available, which includes both terminals and webmin equivalents.
I don't know if there's much of a space left for cockpit or webmin equivalents though. There's hypervisors with dedicated UIs like Proxmox and oVirt, but that's not the same.
I consider these big pluses, I use Cockpit on Debian on my servers that run VMs rather than something like Proxmox, because 1. it's much less invasive, 2. the machines tend to run other things too, like docker containers.
Have been using it for this since ~2019.
The stats views are useful too, but I wouldn't install it for that on it's own.
edit: and honestly, there's not another good (maintained!) option that fits the niche of 'let me create libvirt VMs from a web browser on a single machine without taking over my whole system'.
Genuine question.
What would that be? I'm always on the lookout for better tools.
Seems pretty cool to me, "meet your users where they are" and all that.
I actually wonder what other options for this sort of web based management panel there are out there, maybe more DEB oriented ones.
Even being in a leadership position and basically competing within Red Hat against this, I found no answer to your question.
I hate the mouse so much that I've got a script to move it off-screen (it follows my focus), and I usually live in my terminal. But trying Fedora on one of my Pis I tried cockpit since it was installed by default and I'm surprised how much I like it.
I think the lack of features is a good thing, there's not much bloat and it recommends extra packages I could like. While i love my terminal cockpit has been nice to get a quick glance. So far the only thing I'm missing is support for doas.
Every tool does not have to be able to do all the things.
Meanwhile I'm happier doing everything on command line on Linux, understanding and learning all its features has been worthwhile. But I can imagine some people just want a server set up and to get on with their day.
This sounds like the dream.
I don't really know what you'd use it for? Maybe to do minor monitoring, but it's not great to admin.
OMV: -- has Docker plugin with Compose support (no need for sep. Docker GUI like Portainer) -- SMB shares are (somehow) more reliable on Win clients -- has more beginner friendly GUI and attitude, easier to share with other users -- batteries-included features like fail2ban + Wireguard
Cockpit: -- first class citizen on EL / Fedora distros -- Podman yes, Docker no - no Compose/Quadlet support -- killer features like VM managements and Terminal -- bugs with Samba
I currently use OMV for serving files over my local network (just for myself) and running a handful of Docker containers. It works fine but I don't use 90% of it's features.
OMV takes over your system - lots of "Auto-generated and maintained by OMV, do not touch" in system configs. By comparison, with Cockpit I could tweak and set up my own stuff. With OMV, when I needed to change my network settings, I had to fight bugs in the OMV GUI, and couldn't edit the configs directly. Same thing when I was trying to set up my disks in a particular way. This is a big issue because when something breaks, none of the general (non-OMV specific) answers on the forums help because you can't actually edit the configs...
The other is jank. I ran into many, many issues with OMV. Even for installing, I had to resort to 'curl .. | sudo bash' as the officially recommended option, with no proper uninstall method.
The new bridge is written in Python and when time comes we want to rewrite our webserver into some modern.
And I also find it really interesting to see whether a product is programmed using one clear stack or a mixture.
I know some languages and ecosystems much better than others, so I have an idea how well I could support it up-front if needed. Others have different deployment styles, ranging all the way from "just copy this one binary somewhere" to "first install this language interpreter with a fricken curlpipe, then this language-specific package manager, then these hundreds of dependencies, then our app if you're still awake. But don't forget you'll still need an application server..."
The widespread use of Docker has made the last even less common, but I still run into docker containers just don't work, and I don't know the tech stack, and learning a whole tech stack just to troubleshoot someone else's broken code in order to try it out is not my most favoritest use of time.
"X exists so why would anyone ever build Y". Why not? Competition is good, think about it for a minute and I'm sure you'll figure out how.
People are allowed to create their own X even if thousands of options already exist.
Second: webmin also has a patchy security history. I don't know enough about cockpit to say it's any better but it would certainly be enough of an issue to review all of the options.
Why beyond being different? Proudly announcing "but I don't use systemd" seems less like a humblebrag and more like an old man "get off my lawn"
They may have more nuanced answers.
"A number" being the operative word here. It's a small number, and for a pretty good reason.
I get there is an important reason to have choice, but bragging about how you don't use systemd is just meme-y and comes from a place of (misplaced) elitism usually.
I think you're projecting. No one bragged about such a thing, they just merely mentioned their specific situation.
It's not all text-editor-wars and squabbles, as stated and even acknowledged before : there exist reasons aside from nerd elitism to choose something other than systemd.
name one.
there are a lot of decisions that go into using older/obscure/specific software aside from 'pride', and simply wishing that some software supported other stuff is by no means 'fighting' anything.
"Love it-- but since I'm not using systemd, it's a no-go. Would love to see it support more diverse systems"
That poor lack of diversity.
Dbus is a message bus for all of Linux, this is a requirement if you want to develop stable tools. A uniform message bus so you don't have to keep changing your code for every single distro and release.
sudo runuser -u cockpit-wsinstance -- /usr/libexec/cockpit-ws --port=9090 --for-tls-proxy
And of course, we'd still need to reimplement many of the existing systemd-based modules. I wonder if the creation and maintenance of these modules (given the existing size and status of the ecosystem) is less-than-trivial in comparison to something like a set of NixOS config bindings. At the end of the day, I'd be more exited about a project that puts portability and customization first than retrofitting an upstream product to whom I'm not a target user.
Note: Not understanding the concerns with systemd or dbus. This works for major distros.
systemd is undeniably powerful, but I enjoy how broad the ecosystem is. Off the top of my head, I personally like having real logfiles under `/var/log`, which I've occasionally had to examine by mounting the disk on another machine after really mucking something up :p
I used it in the past, it has plenty of modules and parse config, so you can edit them by hand too.
It was the opposite of NIH. Red Hat had a bunch of features that their customers wanted Docker to have, like custom image registries and daemonless, but docker wouldn't accept the pull requests adding them.
In this thread: people who disagree on CLI superiority going at it.
In fairness it's got a wealth of features...it's just ugly as sin.
Either way, the OP Cockpit has been unrestricted LGPL for years now so somehow you're mistaken.