Hacker News Reader

Comment by gazby | Hacker News Reader

There's a reason guides like this are a dime a dozen - there is no way to generalize server configuration this broadly.

But as long as we're doing it anyway - the only thing that locking the root account gets you is assurance that if you ever bork the user you created in this guide (or sudo functionality as a whole) you'll have no way to recover without booting into another environment.

Perhaps one ought not take sysadmin advice from a blog post with a first sentence that reads "I’m not a sysadmin, and I don’t want to be".

View on HN

The biggest rules about securing things is don't be in security. Just do your diligence to put your hosts several layers away from public access and make all images and containers hardened with no elevated permissions. Sure vulnerabilities will still exist... if the only thing that can access the container is through a narrowed proxy you are not going get some dumb levels of attacks on your systems.

AWS allows you to ssh into your hosts from within AWS. You just manage that security. NO ONE needs public ssh access, no one needs vpn ssh access just AWS ssh access. DON'T OVER COMPLICATE THINGS!

I agree with you. I am not gonna say don't follow a system engineers advice. I say follow everyone's advice but pick out the things that seem most reasonable. If it is extra work then you're doing it wrong, simplify everything so that the time spent on resolving issues is faster. Faster resolution means faster security fixes.

I'm also a "non-sysadmin" like the OP. I'm much more casual than you are:

1. Install LTS server with a sudo admin user and complex password (yes, password, not key. I still end up using keys for backup automation because it's easier.)

2. apt install fail2ban. Don't even need to configure it: its default configuration will auto-ban any IPs trying to brute-force SSH.

3. apt install unattended-upgrades (nowadays installed by default)

I'd like to see anyone hack that.

You mention running containers with no elevated permissions. I've taken a liking to Docker because, in addition to its many benefits, it lets me separate MY processes from those of the OS. It's just so simple when I can identify the server's running services with "docker ps". I'm curious, when was the last time a Docker exploit could escape the container and modify the host? I read it happened in the early days of Docker but is that still a risk? They would need to: a) exploit a bug in a service you're relying on (eg nginx), b) use the former bug to exploit a bug in Docker, c) defeat the Linux kernel's namespace/cgroup isolation. Is that a realistic threat in 2023?

Always a threat if the a host container is on the same HV. You get all sorts of bleed bugs identified every other year. Literally the last one was a few months ago I think. None as large scale as when it first happened.

When you say "no one needs vpn ssh access" vs "AWS ssh access", what is the difference between the two ?

The consoles at big hosts typically require good 2fa to log in to the web management console, which typically can open a command line on the instance. This is a nice authN layer.

Note that it's possible to configure multi-factor authentication using e.g. one-time password (OTP) for those regular openssh logins. The setup to achieve that still seem quite involved though, so the reluctant sysadmin in me haven't got around to try it.

Multiple factors:

1FA: Password(1F) OR private key (password blank)(1F)

2FA: Private key(1F) with password(2F)

MFA: Private key(1F), w/ password(2F) AND OTP(3F)

And have to use their shitty webui?

Ssh has 2fa options if that's the real reason.

Fwiw, this guide also suggests setting up a wg connection which is no better than ssh, and probably worse in some ways.

It doesn't need to be through the web UI, it can be done through the cli.

https://docs.aws.amazon.com/systems-manager/latest/userguide...

Google Cloud has a similar gcloud compute ssh instance-name command, and I imagine there's a similar one on azure.

That's ssh?

There's massive differences of using this compared to throwing some keys on a server and opening 22. These systems use the cloud provider's proxying and authz/authn to dynamically grant access.

One could have a box with no public IP and no open ports and still use this to connect.

Cloud providers proxying?

Via ssh? With an SSH key? Over port 22?

> Via ssh?

No, through their in-house proxy tools such as Session Manager or Identity Aware Proxy or whatever Azure has.

> With an SSH key?

Not at the edge, and not an SSH key you manage. A dynamically generated one managed by the cloud provider which exists just for that session. So, not really, not like you're thinking.

> Over port 22?

For the tunnel? No.

As mentioned you then just need to lock down AWS, rather than AWS AND outside access to servers via VPN. Lessen the attack surface.

The console in AWS allows access within its system. There is no point increasing the access area to the hosts. More surface area the easier it is to be penetrated by ssh vulnerabilities. You also shift fault to AWS rather than your company and team. You did your diligence, you just have to access control and nothing more. IF AWS has a security breach that access to your systems completely on AWS and you can demand compensation.

What you want to do is avoid fault, improve tolerance, but extend liability to the provider.

> AWS allows you to ssh into your hosts from within AWS.

This is where your argument breaks down IMHO. Unless you are saying "don't expose port 22 to the world...", which is a common (small) part of security-in-depth.

> You also shift fault to AWS rather than your company and team. You did your diligence, you just have to access control and nothing more. IF AWS has a security breach that access to your systems completely on AWS and you can demand compensation.

This appears to be an instance of the "appeal to authority"[0] fallacy and is of little solace should server(s) one is responsible for become compromised.

0 - https://en.wikipedia.org/wiki/Argument_from_authority

You still have to do your diligence. You established what is supposed be a secure system yet that system failed due to provider security. AWS is far more staffed than any other company why wouldn't you shift left to AWS? Why would you hire a fleet of security engineers to do what AWS already established? You are breaking convention, reinventing the wheel and complicating an already simple system.

This isn't fallacy it is reducing a businesses cognitive load.

> You still have to do your diligence.

My point exactly.

> You established what is supposed be a secure system yet that system failed due to provider security.

This makes no sense. By your own recommendation, "provider security" is an AWS offering.

> AWS is far more staffed than any other company why wouldn't you shift left to AWS? Why would you hire a fleet of security engineers to do what AWS already established?

What does Amazon's staffing have to do with best practices when securing a server deployed on their platform? Who said anything about "a fleet of security engineers"? How does any of that relate to securing that which one constructs, and ultimately is responsible for, when using said hosting services?

> You are breaking convention, reinventing the wheel and complicating an already simple system.

Are you saying that your original statement of "You also shift fault to AWS rather than your company and team" is somehow an accepted convention?

And what wheel did I "reinvent"?

Finally, was my identification of the common practice which is moving sshd off of port 22 the complication to which you refer?

Yeaaah you're trying poke holes. Problem is that there are larger holes in a network if you're setting up and safe guarding a VPN so you can SSH. Moving ssh if even needed at all should be to the provider's secured tools.

https://aws.amazon.com/blogs/compute/new-using-amazon-ec2-in...

I really don't recommend extending responsibility for creature comforts. However you want to do this so be it.

> This makes no sense. By your own recommendation, "provider security" is an AWS offering.

Are you stating your system is infaliable? So why would you want to bear the infaliable claim and not shift it to the company providing it?

> What does Amazon's staffing have to do with best practices when securing a server deployed on their platform? Who said anything about "a fleet of security engineers"? How does any of that relate to securing that which one constructs, and ultimately is responsible for, when using said hosting services?

Tooling takes a team to support it. You think every company can afford a team to manage that tooling? And why should they? Not all businesses are tech companies but still need a digital footprint. They need to be selfconcious and choose provider practices to get the most out of them.

> Are you saying that your original statement of "You also shift fault to AWS rather than your company and team" is somehow an accepted convention? Providers own the responsibility of their technology. In terms of failure if access is correctly configured and managed, yet their technology fails they owe your businesses it is very simple.

> And what wheel did I "reinvent"?

Implementing old security practices. Why wouldn't you move to be better pratices and prevent larger holes in your network? Often companies get into this repetitious cycle of reimplementation or reinvention of existing tools and technology just to manage access especially ssh. The convention of using a cloud platform is to use a cloud platform's security access not some sketched up VPN and SSH system.

This sums it up, "The convention of using a cloud platform is to use a cloud platform".

If you rent compute space, then you trust them to responsibly use the hypervisor instead of snooping. If you trust that or not, you are all in and may as well cement over the external port 22.

10000% on the money.

> Yeaaah you're trying poke holes.

No, I am trying to remind you of the topic which was under discussion. To wit:

  The Reluctant Sysadmin's Guide to Securing a Linux Server

> Are you stating your system is infaliable?

  A straw man fallacy (sometimes written as strawman) is the
  informal fallacy of refuting an argument different from
  the one actually under discussion, while not recognizing or
  acknowledging the distinction.[0]

> Tooling takes a team to support it.

See above quote.

> You think ...

You do not know what I think nor my experiences, so please do not be so arrogant as to assume so.

>> And what wheel did I "reinvent"?

> Implementing old security practices.

Again, please refer to the *article under discussion*. In the event it remains unclear, I will restate its title:

  The Reluctant Sysadmin's Guide to Securing a Linux Server

> Why wouldn't you move to be better pratices and prevent larger holes in your network?

See previous strawman definition and link below.

> Often companies get into this repetitious cycle of reimplementation or reinvention of existing tools and technology just to manage access especially ssh. The convention of using a cloud platform is to use a cloud platform's security access not some sketched up VPN and SSH system.

Again, see previous strawman definition above and link below.

Note that the only ssh-related recommendation I proffered was:

  Unless you are saying "don't expose port 22 to
  the world...", which is a common (small) part of
  security-in-depth.

This is a well-known, albeit very small and insufficient by itself, part of helping to reduce attack vectors.

As to "sketched up VPN and SSH system", I have no idea as to what you are referencing. Perhaps this is a recollection of a previous engagement wherein decisions made remind you of a bad situation similar to, but different than, this?

HTH

0 - https://en.wikipedia.org/wiki/Straw_man

EDIT: corrected spelling from "waas" to "was"

Strawman argument sometimes is used to draw out a point. You can not confidently say your security solution is infabiable and nor should you. The article is just a good runbook of things and less a guide. But if you are working on the cloud you shouldn't go using old management methods like they belong in your network.

You would not believe how many companies are dependent on patching through users through VPNs in order to access remote hosts. I mean some have to because of no other solutions like managing their own on-prem. I kind of would be interested in AWS access management capable of being implemented within an on-prem.

McDonalds employs the most cooks. They must have the best food.

Staffing count doesn't guarantee quality.

> Perhaps one ought not take sysadmin advice from a blog post with a first sentence that reads "I’m not a sysadmin, and I don’t want to be".

That's just perfect.

That sounds reasonable, from a "passion" angle.

However, I would like to hear what someone who dislikes sysadmining but must anyway has to say. Why? A different perspective. I know some sysadmins who "left the car up on blocks," as in, they couldn't stop tweaking and fixing and so on, and they love that, and that's great.

Someone who is forced into the job, however, is communicating to me what I would expect to be the most necessary tasks. Maybe I also am in a similar place and would like to get on with the rest of my job. And in an Agile world, programmers get pushed into sysadmining ... at least, in some versions of Agile to which I have been subjected.

> However, I would like to hear what someone who dislikes sysadmining but must anyway has to say. ... Someone who is forced into the job ...

The only way a job is done consistently well is by those whom want to do that job well consistently.

HTH

EDIT: added missing "do" verb.

Too bad.

Sometimes a business can't or won't hire a real sysadmin. The people forced into the job need guides just as much.

> Too bad.

> Sometimes a business can't or won't hire a real sysadmin. The people forced into the job need guides just as much.

I was making a more general statement regarding people having to do tasks they don't want to do, even if they have the skills to do them. Which makes resources helpful to "people forced into the job" rather difficult to produce.

How can one make a guide to assist people to be successful in something they don't want to do in the first place?

> the only thing that locking the root account gets you is assurance that if you ever bork the user you created in this guide (or sudo functionality as a whole) you'll have no way to recover without booting into another environment.

That's not a unique or novel insight. For the case your system gets borked (either by yourself, your hardware or your cloud provider) you need a plan in advance:

1. How can I access the data the server has or how much of it can I afford to lose?

2. How do I get a replacement running within a time window acceptable for my usage?

The answers will be very different depending on your use case. But how you locked the root user has very little impact on them.

Booting into another environment is always one option in my plan so locking the root user doesn't frighten me.

"the only thing that locking the root account gets you is assurance that if you ever bork the user you created in this guide (or sudo functionality as a whole) you'll have no way to recover without booting into another environment."

As a dev, I say that's a good thing. I've administered my own systems for decades and helped in small startups where we had no full time admin so definitely not new to administering Linux.

As opposed to borking the root user and being equally locked out? Assuming your sudo config is a "configure it once and then leave it forever" deal - which seems common IME - I can't see any way it would be different.

(Mind, this cuts both ways - once you force only key-based SSH, I generally don't see a problem with direct root access either.)

Just don't set root:toor and you'll be alright. Keys are good, but passwords are good as well.

You shouldn’t need root, you should have another person with admin rights as a backup plan.

It is a vm so if you do something that would break sudo or all your users you should have a vm snapshot at your fingertips ready to restore from AWS interface.

Even if you are running bare metal you should setup snapshots first but nowadays no one runs bare metal web servers it is still som hyper visor with bunch of vm-s that are easy to backup restore or just delete and create fresh.

> You shouldn’t need root, you should have another person with admin rights as a backup plan.

What?

I am the only admin on about a dozen machines. I'm not outsourcing that to a friend. That trust model is much more flawed than separation of duties and permissions segmentation as part of my administration routine.

What happens if you get hit by a bus, or become otherwise incapacitated?

That's what a dead man's switch is for. All of my credentials will be shared after a specified period of time.

Your manager or company owner should have admin account then, he should not use it for anything besides disaster recovery.

If you are single person shop that is your choice, have a root account or whatever you feel like having.

That's not true. It's not obvious what user you have that could do sudo. Thus it does improve security. I advice the same in my book (Deployment from Scratch) and I suggest that for both the host system and containers. There is little cost to not primarily using root.

>> But as long as we're doing it anyway - the only thing that locking the root account gets you is assurance that if you ever bork the user you created in this guide (or sudo functionality as a whole) you'll have no way to recover without booting into another environment.

> That's not true. It's not obvious what user you have that could do sudo. Thus it does improve security. I advice the same in my book (Deployment from Scratch) and I suggest that for both the host system and containers. There is little cost to not primarily using root.

It is true. To ensure root cannot be used when ssh'ing into a server, set "PermitRootLogin" to "no" in sshd_config (as mentioned in the OP).

Locking out root entirely, as further mentioned in the OP and suggested by your comment, does nothing to increase security regarding remote penetration attacks. Furthermore, should a non-root account which has sudo privileges be compromised, an argument could be made that having a functional root account with its own password accessible only locally and not enabling sudo is a more secure approach.

Either way, having a root account which can only be used locally ensures there is a recovery workflow should one be needed, as the GP enumerates.

Reply on news.ycombinator.com