Hacker News Reader

Comment by mdaniel | Hacker News Reader

I really enjoyed my time with Vault's ssh-ca (back when it had a sane license) but have now grown up and believe that any ssh access is an antipattern. For context, I'm also one of those "immutable OS or GTFO" chaps because in my experience the next thing that happens after some rando ssh-es into a machine is they launch vi or apt-get or whatever and now it's a snowflake with zero auditing of the actions taken to it

I don't mean to detract from this, because short-lived creds are always better, but for my money I hope I never have sshd running on any machine again

View on HN

> any ssh access is an antipattern.

Not generally. In one particular class of deployments allowing ssh access to root enabled accounts without auditing may be.. but this is an exceptionally narrowed definition.

> I hope I never have sshd running on any machine again

Sounds great for production and ridiculous for development and testing.

> Sounds great for production and ridiculous for development and testing.

I believe that "practice how you're going to play" to get devs into the habit of not using a crutch to treat deployments like they are their local machine. The time to anticipate failures is in the thinking time, not "throw it over the wall and we'll think later"

Your devs should not be managing deployments. Making deployable software and actually worrying about the deployment environment don't need particularly tight coordination. I'd also worry that you're overfitting your software to whatever third party deployment target you've selected.

Principle of least privilege trivially prevents updating system packages. Like if you don't want people using apt, don't give people root on your servers?

Even for immutable OSes, SSH is a great protocol for bidirectionally authenticated data / file transfer.

You can audit if you put something like hoop.dev, Tailscale, Teleport or Boundary in between the client and server.

Disclaimer: I work at Hashicorp.

But I avoid hascicorp stuff whenever I can!

How do you handle db.

Stuff I work on is write heavy so spawning dozens of app copies doesn’t make sense if I just hog the db with Erie locks.

I must resist the urge to write "users can access the DB via the APIs in front of it" :-D

But, seriously, Teleport (back before they did a licensing rug-pull) is great at that and no SSH required. I'm super positive there are a bazillion other "don't use ssh as a poor person's VPN" solutions

This led me to google "teleport license," which sounds like a search from a much more interesting world.

You might be interested in Peter F. Hamilton's Commonwealth Saga sci-fi series, then.

Among other tech, it involves the founding of a megacorp that exploits the discovery and monopolization of wormhole technology for profit, causing a rift between the two founders, who each remind me of Steve Jobs and Steve Wozniak in their cooperation and divergence.

https://en.wikipedia.org/wiki/Commonwealth_Saga

Yo, dudes, how’s it hanging?

Is this a reference to the books? It's been a while since I read them.

Its what Ozzie or Nigel say over the radio after they landed.

Ah yeah, that's a great scene! The bravado and hubris of gatecrashing an interplanetary livestream to launch your startup out of stealth is just chef's kiss.

To save others the search: https://github.com/gravitational/teleport/pull/35259 Apache to AGPLv3

But what is the alternative?

There's not one answer to your question, but here's mine: kubelet and AWS SSM (which, to the best of my knowledge will work on non-AWS infra it just needs to be provided creds). Bottlerocket <https://github.com/bottlerocket-os/bottlerocket#setup> comes batteries included with both of those things, and is cheaply provisioned with (ahem) TOML user-data <https://github.com/bottlerocket-os/bottlerocket#description-...>

In that specific case, one can also have "systemd for normal people" via its support for static Pod definitions, so one can run containerized toys on boot even without being a formal member of a kubernetes cluster

AWS SSM provides auditing of what a person might normally type via ssh, and kubelet similarly, just at a different abstraction level. For clarity, I am aware that it's possible via some sshd trickery one could get similar audit and log egress, but I haven't seen one of those in practice whereas kubelet and AWS SSM provide it out of the box

Be careful with SSM, it can provide pretty much unlimited access: https://github.com/Cyberax/gimlet

You can use it to tunnel arbitrary traffic inside your VPC.

IaC, send out logs to Splunk, health checks, slow rollouts, feature flags etc?

Allow SSH in non prod environments and reproduce issue there?

In prod you are aiming for "not broken" rather than "do whatever I want as admin".

I built a config management tool, Etcha, that uses short lived JWTs. I extended it to offer a full shell over HTTP using JWTs:

https://etcha.dev/docs/guides/shell-access/

It works well and I can "expose" servers using reverse proxies since the entire shell session is over HTTP using SSE.

I don’t understand why this is more secure than limiting SSH to local network only and doing ‘normal’ ssh hardening.

None of that is required here? Etcha can be exposed on the Internet with a smaller risk profile than SSH:

- Sane, secure defaults

- HTTP-based--no fingerprinting, requires the correct path (which can be another secret), plays nicely with reverse proxies and forwarders (no need for jump boxes)

- Rate limited by default

- Only works with PKI auth

- Clients verify/validate HTTPS certificates, no need for SSHFP records.

“All JWTs are sent with low expirations (5 seconds) to limit replability”

Do you know how many times a few packets can be replayed in 5 seconds?

Sure, but this is all happening over HTTPS (Etcha only listens on HTTPS), it's just an added form of protection/expiration.

How do you troubleshoot?

I think ssh-ing into production is a sign of not fully mature devops practices.

We are still stuck there, but we're striving to get to the place where we can turn off sshd on Prod and rely on the CI/CD pipeline to blow away and reprovision instances, and be 100% confident we can test and troubleshoot in dev and stage and by looking at off-instance logs from Prod.

How important it is to get there is something I ponder about my motivations for - it's cleary not worthwhile if your project is one or 2 prod servers perhaps running something like HA WordPress, but it's obvious that at Netflix type scale that nobody is sshing into individual instances to troubleshoot. We are a long way (a long long long long way) from Netflix scale, and are unlikely to ever get there. But somewhere between dozens and hundreds of instances is about where I reckon the work required to get close to there stars paying off.

> at Netflix type scale that nobody is sshing into individual instances to troubleshoot

Have you worked at Netflix?

I haven't, but I have worked with large scale operations, and I wouldn't hesitate to say that the ability to ssh (or other ways to run commands remotely, which are all either built on ssh or likely not as secure and well tested) is absolutely crucial to running at scale.

The more complex and non-heterogenous environments you have, the more likely you are to encounter strange flukes. Handshakes that only fail a fraction of a percent of all times and so on. Multiple products and providers interaction. Tools like tcpdump and eBPF becomes essential.

Why would you want to deploy on a mature operating system such as Linux and not use tools such as eBPF? I know the modern way is just to yolo it and restart stuff that crashes, but as a startup or small scale you have other things to worry about. When you are at scale you really want to understand your performance profile and iron out all the kinks.

Can also use stuff like Datadog NPM/APM that uses eBPF to pick up most of what you need. Its been a long time since I've needed anything else.

Yes, there are numerous other ways to run remote commands than ssh, all of them less secure. (Running commands via your monitoring system can even be a very handy back door in a pinch.)

The argument here was that remote commands was less useful at scale, not that ssh was a particularly bad way of implementing it. Which doesn't make sense. You tend to have more complex system interactions at scale, not less.

Right. The answer is having systems that are resilient to failure, and if they do fail being able to quickly replace any node, hopefully automatically, along with solid observability to give you insight into what failed and how to fix it. The process of logging into a machine to troubleshoot it in real-time while the system is on fire is so antiquated, not to mention stressful. On-call shouldn't really be a major part of our industry. Systems should be self-healing, and troubleshooting done during working hours.

Achieving this is difficult, but we have the tools to do it. The hurdles are often organizational rather than technical.

> The hurdles are often organizational rather than technical.

Yeah. And in my opinion "organizational" reasons can (and should) include "we are just not at the scale where achieving that makes sense".

If you have single digit numbers of machines, the whole solid observability/ automated node replacement/self-healing setup overhead is unlikely to pay off. Especially if the SLAs don't require 2am weekend hair-on-fire platform recovery. For a _lot_ things, you can almost completely avoid on-call incidents with straightforward redundant (over provisioned) HA architectures, no single points of failure, and sensible office hours only deployment rules (and never _ever_ deploy to Prod on a Friday afternoon).

Scrappy startups, and web/mobile platforms for anything where a few hours of downtime is not going to be an existential threat to the money flow or a big story in the tech press - probably have more important things to be doing than setting up log aggregation and request tracing. Work towards that, sure, but probably prioritise the dev productivity parts first. Get your CI/CD pipeline rock solid. Get some decent monitoring of the redundant components of your HA setup (as well as the Prod load balancer monitoring) so you know when you're degraded but not down (giving you some breathing space to troubleshoot).

And aspire to fully resilient systems and have a plan for what they might look like in the future to avoid painting yourself into a corner that makes it harder then necessary to get there one day.

But if you've got a guy spending 6 months setting up chaos monkey and chaos doctor for your WordPress site that's only getting a few thousand visits a day, you're definitely going it wrong. Five nines are expensive. If your users are gonna be "happy enough" with three nines or even two nines, you've probably got way better things to do with that budget.

> For a _lot_ things, you can almost completely avoid on-call incidents with straightforward redundant (over provisioned) HA architectures, no single points of failure, and sensible office hours only deployment rules (and never _ever_ deploy to Prod on a Friday afternoon).

For a lot of things the lack of complexity inherent in a single VPS server will mean you have better availability than any of those bizarrely complex autoscaling/recovery setups

I'm not so sure about all of that.

The thing is that all companies regardless of their scale would benefit from these good practices. Scrappy startups definitely have more important things to do than maintaining their infra, whether that involves setting up observability and automation or manually troubleshooting and deploying. Both involve resources and trade-offs, but one of them eventually leads to a reduction of required resources and stability/reliability improvements, while the other leads to a hole of technical debt that is difficult to get out of if you ever want to improve stability/reliability.

What I find more harmful is the prevailing notion that "complexity" must be avoided at smaller scales, and that somehow copying a binary to a single VPS is the correct way to deploy at this stage. You see this in the sibling comment from Aeolun here.

The reality is that doing all of this right is an inherently complex problem. There's no getting around that. It's true that at smaller scales some of these practices can be ignored, and determining which is a skill on its own. But what usually happens is that companies build their own hodgepodge solutions to these problems as they run into them, which accumulate over time, and they end up having to maintain their Rube Goldberg machines in perpetuity because of sunk costs. This means that they never achieve the benefits they would have had they just adopted good practices and tooling from the start.

I'm not saying that starting with k8s and such is always a good idea, especially if the company is not well established yet, but we have tools and services nowadays that handle these problems for us. Shunning cloud providers, containers, k8s, or any other technology out of an irrational fear of complexity is more harmful than beneficial.

If you don't know why they failed, replacing them is pointless.

> I think ssh-ing into production is a sign of not fully mature devops practices.

that's great and completely correct when you are one of the very few places in the universe where everything is fully mature and stable. the rest of us work on software. :)

A whole lot of words to say "we don't troubleshoot and just live with bugs, #yolo".

It's a good mindset to have, but I think ssh access should still be available as a last resort on prod systems, and perhaps trigger some sort of postmortem process, with steps to detect the problem without ssh in the future. There is always going to be a bug, that you cannot reproduce outside of prod, that you cannot diagnose with just a core dump, and that is a show stopper. It's one thing to ignore a minor performance degradation, but if the problem corrupts your state you cannot ignore it.

Moreover, if you are in the cloud, part of your infrastructure is not under your control, making it even harder to reproduce a problem.

I've worked with companies at Netflix's scale and they still have last-resort ssh access to their systems.

In my world, if a developer needs access to the Node upon which their app is deployed to troubleshoot, that's 100% a bug in their application. I am cognizant that being whole-hog on 12 Factor apps is a journey, but for my money get on the train because "let me just ssh in and edit this one config file" is the road to ruin when no one knows who edited what to set it to what new value. Running $(kubectl edit) allows $(kubectl rollout undo) to put it back, and also shows what was changed from what to what

Your world is very narrow and limited. Some devs also have to deal with customer provisioned HW infrastructure, with buggy interactions between HW/virtualization solutions that every 5 minutes duplicate all packets for a few seconds; with applications that interact with customer only onsite HW you only have remote access to via production deployment; with quirky virtualization like vmware stopping the vCPU on you for hundreds of ms if you load it too much which you'll not replicate locally; with things you can't predict you'll need to observe ahead of time, etc. And it does not involve editing any configs. It's just troubleshooting.

How do you debug the worker itself?

Separate from my sibling comment about AWS SSM, I also believe that if one cannot know that a Node is sick by the metrics or log egress from it, that's a deployment bug. I'm firmly in the "Cattle" camp, and am getting closer and closer to the "Reverse Uptime" camp - made easier by ASG's newfound "Instance Lifespan" setting to make it basically one-click to get onboard that train

Even as I type all these answers out, I'm super cognizant that there's not one hammer for all nails, and I am for sure guilty of yanking Nodes out of the ASG in order to figure out what the hell has gone wrong with them, but I try very very hard not to place my Nodes in a precarious situation to begin with so that such extreme troubleshooting becomes a minor severity incident and not Situation Normal

If accidentally nuking a single node while debugging causes issues you have bigger problems. Especially if you are running kubernetes any node should be able to fall off the earth at any time without issues.

I agree that you should set a maximum lifetime for a node on the order of a few weeks.

I also agree that you shouldn’t be giving randos access to production infra, but and the end of the day there needs to be some people at the company who have the keys to the kingdom because you don’t know what you don’t know and you need to be able to deal with unexpected faults or outages of the telemetry and logging systems.

I once bootstrapped an entire datacenter with tens of thousands of nodes from an SSH terminal after an abrupt power failure. It turns out infrastructure has lots of circular dependencies and we had to manually break that dependency.

Exactly this. Have heard it referred to as "break glass access". Some form of remote access, be it SSH or otherwise, in case of serious emergency.

Passive metrics/logs won't let you debug all the issues. At some point you either need a system for automatic memory dumps and submitting bpf scripts to live nodes... or you need SSH access to do that.

This "system for automatic dumps" 100 percent uses ssh under the hood. Probably with some eternal sudo administrator key.

Personal ssh access is always better (from a security standpoint) than bot tokens and keys.

There's a thousand ways to do it without SSH. It can be built into the app itself. It can be a special authenticated route to a suid script. It can be built into the current orchestration system. It can be pull-based using the a queue for system monitoring commands. It can be part of the existing monitoring agent. It can be run through AWS SSM. There's really no reason it has to be SSH.

And even got SSH you can have special keys with access authorised to only specific commands, so a service account would be better than personal in that case.

> Separate from my sibling comment about AWS SSM,

This seems like it’s conceding the point since SSM also allows you to run commands on nodes - I use it interchangeably with SSH to have Ansible manage legacy servers. Maybe what you’re trying to say is that it shouldn’t be routine and that there should be more of a review process so it’s not just a random unrestricted shell session? I think that’s less controversial, and especially when combined with some kind of “taint” mode where your access to a server triggers a rebuild after the dust has settled.

Yes, you nailed it with "it shouldn't be routine" and there for sure should be a review process. My primary concern with the audit logs actually isn't security it's lowering the cowboy of the software lifecycle

> combined with some kind of “taint” mode where your access to a server triggers a rebuild after the dust has settled.

Oh, I love that idea: thanks for bringing it to my attention. I'll for sure incorporate that into my process going forward

The first time I heard it was a very simple idea: they had a wrapper for the command which installed SSH keys on an EC2 instance which also set a delete-after tag which CloudCustodian queried.

You don't. you shoot it in the head and get a new one. If you need logging / telemetry bake it into the image.

Are you from techsupport?

Actually not every problem is solved with the "have you tried turning it off and back on again" trick.

No, what we're talking about is (to extend your very condescending tech support analogy) shipping the customer a new PC from the factory, and telling them to throw the old one away because it doesn't matter. It will only start to matter if they have 3 bad PCs in a row, at which time it becomes (a) a demonstrable failure and not just stray neutron rays (b) an incident which will carry a postmortum of how the organization could have prevented that failure for next time

I did start the whole thread by saying "and then I grew up," and not everyone is at the same place in their organizational maturity model. So, if you're happy with the process you have now, keep using it. I was unhappy, so I studied hard, incorporated supporting technology, and lobbied my heart out for change. Without maturity levels we'd all still be using telnet and tar based version control

Once you reach the next maturity level you realize that some problems and bugs are from bad design, and cannot be fixed by restarting the server.

Fixing bad design is an art, not an organizational discipline. (Sadly.)

He asks the senior developer to do it.

Reply on news.ycombinator.com