Practical Linux Hardening Guide | Hacker News Reader

Practical Linux Hardening Guide

github.com

683 points_5b62213 comments

Not a fan of this guide: No explanations, no context and worse: missing critical stuff for your typical hosted $webapp server usecase:

run services as normal user and confine them using apparmor (or selinux if you are masochichst)

the NSA will probably hack you anyway, that drive by script-kiddie that kills your company JIRA instance by DDoSing some gameserver while exploiting some old Wordpress plugin can be stopped by apparmor!

What's missing is some kind of kernel live patching - if someone is not a script-kiddie on your machine that Wordpress shell is a nice tool to elevate to root - your kernel is rotting and exploits are plenty... so consider some kind of live-patching if your are paranoid.

Also no discussion about updates, the need to restart services for reloading updated .so files (like openssl)...

You can obey that guide and be busy with life and some skiddy root exploits your box anyway...

so /rant - there must be some better guides? found those cis benchmarks (https://www.cisecurity.org) a mixed bag - is there anything better out there?

I think the CIS benchmarks are a good option, but they are very detailed. They do provide pre-rolled images though, which may be a quick solution.

I have recently completed a project applying CIS to a clients estate. Whilst CIS is great, particularly if you are a large Org having to deal with auditors and the like, it can only be regarded as the bare minimum.

> missing critical stuff for your typical hosted $webapp server usecase:

It looks like work in progress. Selinux, updates, deployment and many other topics are on the table of contents, but not present in the document. (Yet?)

It's got some context which is better than almost any other guide. Ideally they'd include some threat models applicable to each point, but that could come in the future (also you can send prs)

CIS and the various NSA STIGs are about the best actual prescriptive guides.

But threats occcur on a continuum - at a point (which is focused organised crime/nation state actors) you will lose...but there is still a lot of low hanging fruit you can win on.

I second that, encrypted /boot? Who is the adversary this is protecting you against? Many of these things looks like snake oil to me.

Encrypted boot really bit me in the ass one time, that's one afternoon spent in the dust underneath the stairs in my student housing that I won't get back.

Would've been so much easier to repair that machine if the boot partition wasn't so weird.

Yeah, getting strong 90s vibes from this doc, perhaps because most hardening manuals like this are just collections of advice from other (bad and old) hardening manuals.

That's exactly it.

Lists like this exist to give administrators and their managers (1) a false sense of security when implementing it and (2) a subconscious excuse not to actually have to think about what you actually have to defend against. The list of topics is almost entirely useless if you are running any sort of containers on top of your Linux install, just as one example.

I'm sorry to be sounding so harsh, but I sincerely detest the security-by-checklist approach. It barely protects against any real-world threats, needlessly complicates matters, and gives credence to the perception that "security slow things down" through over-engineered solutions to non-existing problems. Moreover, it's a subconscious excuse to actually stop thinking about one's actual risks. Reasoning about actual threat models may not be easy but it is what one should be doing.

There's nothing "practical" about this at all, in my view.

> I'm sorry to be sounding so harsh, but I sincerely detest the security-by-checklist approach.

Checklists are essential (or becoming so) in all professions that require a level of safety or assurance in their operations. They don't replace careful though and action, but they do supplement it be making sure there's a minimum level of items that were checked and not omitted because someone thought they didn't apply when then did, or someone just plain forgot. A checklist is a good way to start some rational thinking about what's required for your specific case (especially if it's so restrictive that you have to (and are expected to) selectively alter portions of the checklist just so your current use case functions.

Checklists are well known and save lives in some professions (such as aviation), and are being applied to others even when there's push-back (Surgeons and emergency room operations[1]) because the benefits are just so large.

I think it's fair to dislike how people and organizations adopt a checklist in lieu of careful thought about security, and to dislike poorly defined and reasoned checklists themselves, but I for one would be much happier, and feel much safer, if security checklists were much more common overall.

1: https://www.who.int/bulletin/volumes/86/7/08-010708/en/

Checklists are fine if you have a fixed problem space. Just to go by your example - aviation safety is a mostly fixed problem space you can make mitigations in. Any item on the checklist would be there to mitigate safety issues almost any aircraft would encounter (and I'd imagine there are specific checklists for e.g. passenger vs. cargo aviation).

"Linux security" is not a fixed problem space. Not at all. And that's precisely my problem with this - this checklist pretends that it's a fixed problem space, and therefore grossly misrepresents the problem.

My point is that most problem spaces can be split into fixed and non fixed portions. Aviation safety has plenty of problems that require a real person to make a call and react intelligently, which is why we still have pilots. Checklists are used to cut down on the entirely avoidable problems that might be missed some percentage of the time otherwise.

There are plenty of things in Linux security that are static solutions that can be employed almost all the time, such as not allowing direct access to root accounts, always running a local firewall, making sure remote services aren't run as root without dropping privileges, etc.

Why not submit tickets or PRs? It looks like it's still actively maintained https://github.com/trimstray/the-practical-linux-hardening-g...

Nah it looks like author just made table of contents and counts that there will be bunch of suckers that will collaborate on his repo by submitting pull requests.

Really, fire protection and locking racks with configuration of Nginx and Apache in same guide? If I am going to harden Apache, I am going to find Apache only guide.

> Maintaining a stable temperature and humidity within tight tolerances is critical to IT system reliability.

Data center HVAC engineer here. The part about tight tolerances is not entirely accurate. IT equipment can handle a reasonably wide range of temperatures. The ASHRAE guidelines [1] have a range of 18 to 27 deg C for all classes of equipment.

What is an issue is rapid changes between temperatures. For a data center with any kind of devices (eg tape storage), ASHRAE limits this rate of change to 5degC within a 15 minute period and obviously you also want to avoid any situation where condensation could occur or static build up (high or low relative humidity).

Data centers do try to keep the temp stable towards the top of the thermal envelope for economic reasons but there's always some fluctuation due to server load.

[1] ASHRAE TC9.9 Data Center Power Equipment Thermal Guidelines and Best Practices http://tc0909.ashraetcs.org/documents/ashrae_tc0909_power_wh...

Aye. The old rules are ... old rules. Nowadays we have rather a lot of temperature sensors - inlet, outlet and each subsystem has multiple sensors - they will also report max and min and current.

Rapid changes in temperature and interactions with certain ranges of humidity can be interesting.

Wow, your gear is treated so nicely. The stuff I work on has to deal with anywhere between 0C and 80C while having the crap shaken out of it and occasionally being hit with a high pressure hose.

Different worlds...

OK, high pressure hose in a data centre piques my interest 8)

I stuck a camera on a stick at the southern boundary of my property, about three feet above the stream. The stream rose four feet for a while. I moved the camera.

Now, 0 to 80C. That is just plain odd. Here (southern UK) the temp range is something like -10 to +35C. Nowhere has 0-80C apart from a weird oven.

This is minesite automation stuff, so it usually lives in a stainless steel cabinet, ambient air temperature can be over 50C, sometimes sharing the space with high power switchgear. It's not unheard of for the inside of the cabinets to be over 70C if it's mounted in direct sunlight, so most gear is rated to 80C. Then at other times in the year it'll get down to freezing.

> For secure rooms make sure that the walls go beyond the false ceiling, and below the raised floor, large vents should also be covered with bars if possible.

Ah, the 'Aliens' threat model.

Well if feel the need to have a physically secure room, then presumably you want to actually be secure.

What, are you trying to say security theatre doesn't make everything safe???

Surely if we grope everybody's balls/boobs on their way in, and have a secret list of "people with names that are vaguely similar to middle eastern sounding names" that we won't let in, that'd make up for the walls that you can just crawl under of climb over? Right?

Secure room means there's no way for anyone (big or small) to sneak into the room undetected. Hence, having false ceiling/floor and ducts need to be secured - usually the fire/aircon zones are also separated and isolated. I have seen colo providers do this when they are hosting banks/financial institutions.

This is not at all the variety of advice I expected to see in something titled "Practical Linux Hardening Guide"

Never mind if a person (or Aliens) can get in...can someone feed a firehose into the room from next door and turn it on?

Not Linux specific but:

Don’t make your security encourage legitimate users to work around it due to pointless friction

I always felt like increasing security has some cost on convenience.

It shifts convenience. It's less convenient for me to have to unlock my door when I get home, but it's more convenient to not carry all my valuables with me during the day. And I really like that tradeoff.

Good security measures are like this. Add sandboxes so you can let users do what they want. Add authentication so people know who they're talking to. Support security keys so people don't have to worry ad much about being phished. And so forth.

It does, but there’s also what I call stupid security that doesn’t really add any measurable improvement, but does decrease productivity. Users out smart these systems all the time (e.g. forced password changes where you can’t use the last 10 passwords, users just change their password 11 times so they can continue to use a password that’s been configured on their devices. It’s stupid then to force changes like that when there are much better ways like mfa)

Smart security allows users to do what they need to do efficiently and safely.

Yeah I can't stand the forced password changes where you can't use the last X passwords, or passwords that expire every 30 days. A lot of times it's security compliance entities that push this down to companies, for instance PCI, etc all require those. I think even the new NIST standards address these practices, but the compliance entities are slow and far from pragmatic.

Yep. I can tell you that currently, leading practice involves such measures. I don't agree, but CIS, which is essentially the current gold standard, says so. People who get paid to do this often don't bother griping about it, because they are being paid to harden to a standard and that standard is what it is.

This unfortunately leaves a disconnect between the people who harden (who might actually hear about issues), and the people who write. Even if the writers do hear, it won't be implemented until the next revision.

yes but the rule does enhance security.

the password rules force you to choose [heuristically] guessable passwords, therefore they must be changed every 90 days. simple!

It doesn't if users are working around it.

Right. The first rule of password security: if you have a large enough user base, the odds of a user writing down a password increase, and as passwords become sufficiently difficult to remember, the odds approach 100% at some point that _some_ people are writing down passwords. No amount of defense in depth can protect the "I have a Post-It note under my keyboard" problem, if people can get into your building.

We've handled this by mandating password manager use and pushing length requirements to absurd levels to where it truly is easier to just use the manager, which has two factor.

Yeah PCI or FedRAMP have this 10 char password requirement, which of course no one can remember a 10 char password. So companies just make the password a pattern with some variations, effectively reducing the complexity to a tenth of a random 8 char password and the people who know the pattern leave the company so it’s effectively public. So much for math.

Usability is a component of security because of human factors; if your “secure” process or system is not convenient for use, people will in practice find ways to work around it instead of using it as intended, which will defeat security.

Often, but not always.

ssh keys instead of passwords are a good example of better security and more convenience (for the most common use cases).

It'd be nice if more "security improvements" came with ways to make them convenience improvements too...

Absolutely, but I prefer not to leave 22/tcp open to the world. If I do leave it open it is only from a restricted IP set, otherwise it is behind a VPN, probably OpenVPN.

Is OpenVPN a safer attack surface compared to OpenSSH?

I doubt it.

Is OpenSSH safer when used in addition to OpenVPN?

Probably.

Sure, especially when you VPN into a sacrificial subnet and need MFA to continue elsewhere into locked down application domains. OTOH I would leave ssh listening on a non-descript high port with MFA (key and OTP) enabled. No use worrying too much about that.

That's a pretty good description of apparmor.

Also, have a way for those efficient users to alert you to their pain points (and make it easier than the workaround).

Beyond a certain point I think that's true. There's an old adage that "security is inversely proportional to convenience."

A policy adhered to is better than a policy not.

That's why for any office that has internal wifi I encourage also having a "guest" wifi network (small offices, not using anything with enterprise-level management). People are going to want to connect devices to wifi instead of LTE even if they never approach their billing limits, and if there's an available option that they're allowed to use it cuts down on attempts to use the one they're restricted from (and complaints about "I tried to connect to the wifi but I can't get to the Internet!" "Was it the internal one or the guest? Internal is locked down. Connect to the guest.")

On that note, I would like to recommend the most complete book I've encountered on the topic: "UNIX and Linux System Administration Handbook" by Nemeth et.al. Just because I wish someone had recommended it to me earlier.

I can vouch for the book also. It's the best technical book for system operators.

It's really amazing the stuff they cover succinctly and yet complete. From the "classic" unix stuff and contemporary ones, like Docker, Ansible etc.

Damn, I definitely bought that book for a Linux sysAdmin class in college, have it on my bookshelf right now.

Great book, although when I was a young teenager in the late 80's yearly 90's I lived/owned on her sun servers (anchor.cs and the like) undetected for about 7 years. Anyways, I always had great respect for her and I was sorry to hear she was lost at sea a few years ago.

Personally, I prefer the Securing Debian Manual http://www.debian.org/doc/manuals/securing-debian-howto/

Note: This is an official Debian manual, not some random third party.

This looks good. I like that they have explanations / reasoning included, rather than just "do this". If all the planned topics get filled, that will be something to recommend.

Also, first time I hear about polyinstantiated directories, so TIL.

This was one of the neatest features of the computer formerly known as AS/400: each job (batch or interactive) had its own temporary library (QTEMP). You could create objects there knowing only you could see them, and the library was destroyed the moment the job ended. Definitely will try polyinstantiated directories.

What do you mean former? There is an AS/400 in our data centre that we look after for a customer for their off site DR box, with constant DB replication. It is about four years old, the platform is rather older. Bloody noisy beast especially on boot up (IPL?) sounds like a helicopter winding up. It was switched on after racking and that was it - I'm told it is simply available.

Although the platform still exists, IBM has renamed it:

https://en.wikipedia.org/wiki/IBM_System_i

A bland attempt at humor. They've had more name changes than the artist formerly known as Prince (AS/400, eServer iSeries, System i).

Windows Sandbox will have this, the app runs in a disposable VM using copy-on-write memory.

Like the repo mentions - this is just one switch away in systemd. But even without it, you can achieve this with namespaces and mounts.

A lot of this can be eliminated (Not the physical security or install) by doing the following:

1. Run centOS or Redhat

2. install openscap-workbench

3. Use the centOS stig and choose which profile (I recommend US govt configuration base)

4. Uncheck the firewall rules (they set it to deny all incoming; change to DMZ with basic rules)

5. Click remediate and apply.

6. OpenSCAP does the work for you to harden the system

You'll have to use other security appropriate tools for appropriate servers, but you'll know which service and its ramifications.I know that MySQL has a comprehensive security script to prepare. Other tools have similar built in functions.

Also note you can download Nessus and get a 7 day free trial as well. It's not perfect since the ticket price is $2400/yr . You could also use OpenSCAP for compliance, and metasploit as a substitute for application. There's also websuites like Burp and OWASP.

But regardless you pay or not for automated testing, you need something to automatically find bad things so you can fix them.

interesting

https://github.com/OpenSCAP/openscap

also https://aur.archlinux.org/packages/openscap/

As with a lot of other enterprisey software made by RH, it should all work in theory, but it rarely works as well. I've never been able to get stuff like openscap to work reliably with satellite (which is one of those things that should probably not exist at all). I don't remember which openscap profile it was (I think C2S), but it would just hang on nfs servers with lots of data. So you can go and hack on these profiles, but it just gives the impression that none of this stuff is that widely used/production ready.

PSA: Don't do this on a production machine.

Also, after you do "remediate" a machine, be sure to allow plenty of time to find and fix everything that's now broken.

So here's something I've always wondered. If you use linux as a desktop/workstation in enterprise, how do you deploy it securely ? Other than chromeos, I don't see any other distro trying to tackle this. With all traditional distros, you'll likely need to give the user sudo on their machine to be productive (for eg., installing packages). However, all bets are off once you do that. You can restrict sudo to certain programs, but that also isn't very practical (for eg., sudo for yum is basically sudo for the whole system since you can install arbitrary packages). You can create a chromeos like system, but is there no interest in it ? There are some developments like fedora silverblue and nixos, but how is it that this has gone unsolved till now ?

"you'll likely need to give the user sudo on their machine" - no we have sysadmins and application management systems to obviate that. If a user needs an app then they ring up the helldesk and get added to a group (after authorisation), which causes the app to appear.

>If a user needs an app then they ring up the helldesk

That seems very inefficient and friction prone. We have stuff like the nix package manager that can install stuff locally without sudo. It also has other nice features from what I've read. What I'm saying is that we need general purpose distros where you don't need sudo to use them.

"That seems very inefficient and friction prone"

That's the way its done in general: User makes request, request is approved, action. I was noting that a Linux system does not get in the way of that work flow by being a Linux system. If the box belongs to a company then the company gets to decide what is on it.

If the box belongs to the end user then obviously root rights are in order. However, my wife never gets to use anything requiring root rights on her Arch Linux laptop and does not care and would not know what it is anyway. I keep her laptop functioning effectively by watching disc space and other metrics and regularly patching/updating it for her.

Interesting you bring this up. Regardless of the provenance of packages and facility for installation you seem to be hinting that there should be a method to install specific service/feature sets to a system securely and without systems access.

I designed such a system which runs with appropriate permissions to a task, is modular, and accomplishes the provisioning via 'job' files submitted to a service endpoint. A user could describe the software they want installed. The SA performs due diligence on vetting software and designs a standard install module then generates a skeleton jobfile and provides it to the requesting user to fill in necessary details and submit. Wash , rinse, repeat.

I'm sure this is possible in some places but in others it would be completely impractical. At a university, specifically in the STEM faculties, not giving users sudo/admin would create an endless workload for IT. We end up being nothing more than professional software installers.

Depends on how much one cares about security.

On my university we still had real UNIX servers, no way IT could allow for everyone installing what they felt like.

And the local PCs with Slackware/RedHat would get booted via tftp.

You run your own Software repo(s) on site and you only allow software to appear in that repo that you trust. You use applications like puppet to automate installations from your repos. LDAP can store users,groups and attributes that can be used into determining which groups get what software baselines.

Not really all that different from what goes on in the Microsoft world with WSUS/Appstores/etc. Just the implementation is different and much more customizable.

So the agent still needs to hook into something with elevated privileges to install packages, just that instead of having granular control in the package manager, it has moved to server side with an RBAC/HBAC tie-in. I agree that it solves practical issues, but it would be nice to not have privileged access at all. Also, doesn't it create friction if you don't have a package in your repos ?

I mean 'your repos' typically are a superset of what's available in your distro's repos so it's fairly rare for someone to complain that software isn't available.

You could easily make a root-privileged daemon that listens for requests to install packages from users in a particular group.

You'd need to vet the packages you install, because otherwise you can install a package which executes arbitrary code during installation.

Puppet, Chef etc

Any reasonably secure windows systems have restricted user accounts that do not allow app installs. (I'm talking ofcourse in a corporate environment) The help desk / it call for any app install approvals is common in my experience

For installing/using Linux applications without root access, use Docker containers. For security, use rootless containers. Most major graphical apps have a Dockerfile now.

that's literary redhat, Ubuntu, suse, etc...

they all have enterprise support and onprem repos

Also...stick the SSH on a non-default port. Cuts login attempts down to near zero.

For my home network the only thing that's exposed out of my router is an OpenVPN server on a non-default port.

I do similar but I personally recommend OpenVPN on 443/tcp rather than 1194/udp. It looks like SSL traffic because that is essentially what it is. Make sure you use your own locally signed CA and certs and enforce CA trust for a successful connection - to avoid any silly MitM tricks.

There is a reasonable argument against using tcp for the tunnel instead of udp due to what can happen to latency when you have a tcp inside tcp stream when one or the other or both are dealing with retransmits (bear in mind that tcp has some guarantees about delivery, whereas udp does not).

In theory you can get an exponential stand off instead of linear due to tcp (say RDP or http) in tcp (openvpn). In practice I find that it does not generally matter these days and I can generally stream BBC iPlayer when abroad with minimal fuss over OpenVPN.

My ISP blocks many ports which would generally be considered useful to host services on.

https://www.cox.com/residential/support/internet-ports-block...

It's actually very easy to detect OpenVPN on port 443 because it isn't really TLS traffic. Much better to use OpenConnect with ocserv, it opportunistically uses DTLS over UDP on 443, and if it can't do that it can use normal TLS. Much harder detect and gives you the performance boost of UDP when possible.

I have the same , but also have an IP ACL on the VPN ports / protocols (only on new connections). When I do need to Connect from random remote ip , I have a port knocking rule that adds ip for 5min to acl

I actually have the port knock action on a blank website w a unusual subdomain. So we need i it I pull up website and can then vpn in temporaryly

I've had ssh on 47999 for a long time :).

It stopped my logs from getting splattered with failed attempts.

Conveniency through obscurity.

Yep, which is far better than 'security' through obscurity.

I'm so sorry, but my OCD could not let this pass. It's "convenience" not "conveniency". Sorry.

EDITED TO ADD: I wholeheartedly agree with the sentiment though.

look up at that thing whizzing over your head; convenience doesn't rhyme with security. ;-)

Better to have it listening on a port below 1024, otherwise a non-root user could impersonate the ssh daemon.

I agree except if a non-root user I don't know about is binding to any ports from inside then I'm already hosed.

A non-root user might not be a person, it might be a compromised php script.

True but if my system has a compromised anything binding to a port then they can talk to it via whatever they want at that point.

I'm of the opinion that if an attacker gets a local account then I'm already hosed given the regularity of local privilege escalation CVE's

This would be detected by SSH, as the host key wouldn't match. (The impersonator wouldn't be able to use the same host key, as they would need the private key, and that requires root privs.)

A lot of people who don't care much for checking keys.

This is less of a hardening tip and more of a log quieting tip.

it cuts down the drive-by noise by 97%. which lets you not overlook anomalies in the remaining 3%.

we also had a bunch of scanners probing for common php scripts on the http server (we don't run php). simply returning a 404 header and disabling logging in nginx for \.php$ uris cut down a bunch of garbage (mostly from China). additionally, using ngx_http_limit_req_module reduced the effects of numerous DoS-type scanners.

so far, this + iptables for only 80/443/ssh has been a superior solution to instaling fail2ban that continually scans bloated log files and performs constant iptables banning/unbanning foo.

clean logs are essential, not just nice to haves.

Well said.

"Log noise" is a legitimate security concern all by itself. It is actually disappointing how rarely it is discussed, and how in this world of "AI everywhere" there's no great smart AI log analysers to reduce the signal:noise.

Moving the SSH port might "only" reduce noise, but that's a legitimate security goal in its own right.

That's because people can't agree on what is or isn't valuable.

For instance, if I (or my software anyway) see a bunch of sshd login attempts from some IP, and then that IP decides to try imap ... yeah, that's getting insta-blocked.

And that's hard to do if you decide to just ignore the sshd attempts.

Could additional honey pot on 22 help? Run sshd on non-standard port and honey pot on 22. Then automatically ban all the hosts that would connect to it. There could be a concern for a mistake, so then ban for limited time at first or something.

> "Log noise" is a legitimate security concern all by itself. It is actually disappointing how rarely it is discussed, and how in this world of "AI everywhere" there's no great smart AI log analysers to reduce the signal:noise.

That's because after many people learn about not using "security through obscurity" they apply the rule without thinking as if obscurity never has any benefit for anything.

What about tarpitting known spurious attacks? Possibly proxying/forwarding to a different host?

Nothing externally accessible should allow password login at all.

SSH key-based logins are so much more secure and convenient as heck once you invest the time to learn how they work.

For PCI you still need two factor. Its overkill, but we kept password logins even after having keys.

A smartcard is still fairly convenient and allows you to tick the two-factor box (card + pin)

Additionally having a non-standard port does help defend against 0day exploits before patches are available or before you are able to disable the port.

When was the last preauth remote vulnerability in OpenSSH?

nevertheless, past performance is not an indicator of future outcomes, right?

Past performance doesn't guarantee future performance. It's usually a very strong indicator.

Putting SSH on a nonstandard port is pretty silly.

> Putting SSH on a nonstandard port is pretty silly.

what's the alternative while still keeping ssh logs that aren't filled with thousands of login attempts from bots per day? afaik there isnt any solution that is both as simple and as effective as changing the port.

fail2ban will cut down on log noise signficantly, while still allowing you to operate on the standard port. It also thwarts brute-force login attempts, if you're using password authentication. (Which you shouldn't, without good reason. Use public key authentication.)

I like pam_shield a bit better than fail2ban. It's directly tied to pam, versus trolling logs, so the configuration and operation is cleaner.

https://github.com/jtniehof/pam_shield

Last commit June 2012? Does it work with modern distros?

Yes. Using it now with Ubuntu 18.04. Available via apt-get.

Not sure if it’s sad or hilarious that people recommend fail2ban as a solution to log spam (or as a solution to anything really)

Do you really think making firewalling decisions based on unstructured attacker-controlled log data is a good idea?

You control what fail2ban does with said log data and what actually gets logged. There are better ways than fail2ban of course but it's not the worst solution on the planet.

If you're using regex to parse attacker controlled files I'm not entirely sure if you're in control.

Many unexpected things can happen, as a simple example SSH can generate log entries like this

  Jan 30 17:37:04 server sshd[26695]: Invalid user root from 127.0.0.1 from 10.0.0.1

The default rules can deal with this specific example, but this is certainly a path I wouldn't want to go down myself.

Access behind VPN connection. Or deal with the logs in a better way. If SSH noise bothers you, you can either direct it to a different file, or setup a log aggregator where it's trivial to see what you want (or filter out what you don't by default)

Could you please expand on your point ? I understand that security by obscurity is not optimal, but I don't see a good reason not to switch ports.

i think the main argument for leaving it default is [1] for any additional/new sysadmin that has to touch the system. but imo this goes both ways; i'd rather not have my house key hidden under my doormat, where every burglar can check for it in a few seconds. if i taped the key to a random tree branch (even in plain sight), it would be a much safer bet.

[1] https://en.wikipedia.org/wiki/Principle_of_least_astonishmen...

If you're going to insist on a security analogy involving keys and doors in the real world, a better one would be that the attacker has to search every single atom of the doormat to find the key. If you have PubKeyAuthentication enabled and PasswordAuthentication disabled, the world will end before the attacker finds your key (in expectation).

Changing SSH to a non-standard port is an inelegant solution and doesn't actually bolster security.

Does it _harm_ security?

No, but it is non-standard. Why do things you don't need to?

Perahaps not directly, but it does add to cognitive overhead and makes securing the system as a whole more difficult.

If your environment contains multiple ssh servers running on non-standard ports you would not notice when something out of the ordinary happens.

Something listening to an unexpected port should be something you must look into, not shrug at.

So standardize on a non-standard port, document it and when you do have ssh probes assume it's targeted rather than just another l33t child running du-jour crack(x).

Moving to non-standard means more sophisticated discovery and possibly a more sophisticated attempt.

One reason is because setting it to a random high port allows non-root users to turn on the SSH server if the admin turned it off. Probably not super likely to happen, but it's possible.

It doesn’t need to be on a random high port, just not port 22. 222, for example.

This is a silly opinion. Of course it's helpful and very useful when used in conjunction with other hardening measures.

The proper solution to this is having a separate management network that already requires authentication for getting into.

However, indeed. Minimized log noise is great for analyzing anomalies manually.

Seems kinda painful. Just turn off passwords and let them try all they want. I find rsync particularly painful with a non-standard port.

Your .ssh/config can handle that for you?

Sure, you could try to populate .ssh/config on every machine you ever will try to login from.

Or you could use the standard port, turn off passwords, and wait for the sun to burn out before someone brute forces their way in.

Do you sync your ssh keys to all the machines you login from?

How do you ssh-copy-id without some allowance for passworded logins? Especially if you're doing it from such distant machines that you can't scp over a .ssh/config file.

So, I just ssh to whatever machine I need to work on, but occasionally need to rsync or scp files to/from that server.

Sure, I could have puppet push a ~/.ssh/config file everywhere, just seems more effort than it's worth. After all if a hacker wants to know if you are running ssh, it's not hard to scan all ports for ssh.

Why inconvenience yourself more than the attacker?

Or someone can simply own your key holding device as you sip your beverage at the local hotspot or home wifi. Simpler to go after the secret privileged client than the server.

>Seems kinda painful

meh. Just means typing in the port in putty. Pick an easy port - 60k. I can live with that.

Right, but say you have a dozen machines you login from regularly and maybe a few 100 that you login from occasionally. You want to maintain a ~/.ssh/config on all of them?

Seems like a big inconvenience for minimal extra security. Not like scanning for all open ports that have an ssh listening is hard.

If you’re already typing in the full address for the server every time, since you don’t have an ssh config file, then is it really so much more work to additionally type the port?

Yes because I will forget to type -p 5223 at some point. If you set up a non standard port against my recommendation and I get banned for trying 22, you better be on call 24/7 for free to unban me.

If you are coming from 100s of machines, I would really hope you have your home on NFS or something anyways.

Or simply use sshguard.

Just use a jump server and put the rest of your servers on the private subnet for port 22 inbound. Expose your jump server to your vpn only and use 2fa on vpn.

This is so simple to set up and completely removes random attacks. If your jump server/bastion host is attacked... take notice.

Right. But only bring up your jump server when needed from a CSP. The rest of the time unseen and mostly unknown.

That's really terrible recommendation. You should add firewall rules and allow SSH access only from trusted hosts, ideally over VPN. It's not about login attempts but about possible bug in SSH (and other network services), because it's all just software, written in C, so believe it or not, but it's there, it's not if but when.

> That's really terrible recommendation

it would be if it was meant to be comprehensive.

Every little bit helps. I have hosts I access from "where ever."

It's fallen out of favor these days, but if you're running on firewalls that can be configured for port knocking consider doing that. A non-sequential knock pattern that unlocks the remote IP for X amount of time can prevent a ton of grinding attacks.

Add a TLS auth channel to the portknock with a wait window and OTP and you are really in business with this approach.

Absolutely this. Sad to see how many people refer changing ssh port to non-standard as "security" measure.

The firewall isn't immune to exploitable bugs: https://nvd.nist.gov/vuln/detail/CVE-2017-18017

Likewise for VPN stacks: https://nvd.nist.gov/vuln/detail/CVE-2017-7521

If you're not prepared to expose a service to the world, you probably should run it at all. Ad hoc, non-standard configurations add substantial complexity and maintenance burdens. Complexity is the enemy of security, and having less time to manage more complex configurations is not a good recipe.

If you really need a service, then choose the best one and move on. The rule of thumb is that if you can reach a service, you should assume anybody else can, as well. This is especially true regarding SSH. I've seen plenty of servers p0wned via SSH, but never by breaking SSH. Instead the vector was always through an SSH user's computer infected by malware.

You want secure SSH? Disable password authentication and force everybody to use smartcard authentication like a Yubikey. I do rate limit SSH access using OpenBSD PF, but only because the authentication failures fill up and pollute the logs.

Have you seen yubikey/smartcard being used for sshin a large team? Would love to hear about your setup and learnings.

OpenSSH is solid piece of software. It hasn't had any (public) authentication bypasses that wasn't configuration related since it started to look like it looks today. Multiple skilled teams have looked at it thoroughly since. It has also been hardened with things like privsep and sandboxing.

This is not OpenSSL. Language alone does not make a product insecure. For practical purposes it is certainly not just a matter of when.

Contrast that with your VPN product, which probably falls over if someone looks at it funnily. Unless your product is called Wireguard, and we are ten years in the future, perhaps.

The only benefit of this is clean logs. We had auditd running and the bots blew up our SSH logs. Was so ugly that I killed SSH on all but a single entry point server, which is better security anyways. Then I finally just killed logging failed attempts. Since we use two-factor authentication with a key (you have) + password (you know) I was not very worried about this. Prob should just use something like fail2ban though eh?

>The only benefit of this is clean logs.

Except no. Less attempts translate directly into less risk.

Less attempts of something that has zero chance of succeeding (password login against a system that requires keys) doesn't really translate into less risk.

>Less attempts of something that has zero chance of succeeding

All attempts have a >zero chance of succeeding. Honestly...struggling to see how this concept is even remotely contentious or how someone could credibly argue against it.

If the number of brute force attempts drops substantially then the chances of success drop too. This is true for a high security setup and a low security setup. In both cases there is going to be a proportionate drop.

Also cut down on the host keys (e.g. don't have an RSA/DSA/ECDSA host key if all of your clients support Ed25519 hostkeys) and key exchange algorithms (KExAlgorithms in sshd_config); almost all clients support curve25519-sha256 now, so get rid of all the DH fixed ones. Similarly ciphers; OpenSSH and PuTTY support ChaPoly, so you can get rid of aes-ctr, et al.

I'm not saying decrease your supported algorithms to the point where you break things you're using, but you'd be surprised at how much you can get away with not having and never notice it.

At the end of the day this means all those horrible libssh bots and worms can't even begin an SSH session with you, much less attempt any sort of exploitation or bruteforcing. It still generates an entry in your logs (key exchange algorithm mismatch or the like), but decent syslog daemons can filter out such messages by regexps, and redirect them to another debugging log file that you hardly check unless you're having an issue connecting somewhere. Metalog is an example.

I'd absolutely love a section that covers UEFI, secure and measured boot, and TPM integrations.

You can install Ubuntu with UEFI and secure boot enabled without any problems (personal experience). I believe that several other distros work as well, out of the box. All distros can do UEFI with minimal effort.

To be honest I consider things like BMC and the like and whatever goes on inside the BIOS and the like as far greater opaque problems than some of the more esoteric security "assurances" like measured boot.

While not part of the OP - there is some information on a common TPM use (LUKS unlocking) with the later TPM 2.0 options (TPM 1.x was not amazing with Linux).

https://blog.dowhile0.org/2017/10/18/automatic-luks-volumes-...

Even if you have a single server to run a bunch of services (web, mail, ssh, file server etc), best way to raise the security bar quickly is to isolate these services from one another. Use one of the modern virtualmachine/container technologies with minimal footprint to do this. Ideally, you would want physical separation between a node running firewall/loadbalancer and the servers behind them. If you are running a large cluster of many servers, then you definitely want to plan and separate your architecture into different layers of separations.

Don't forget your own laptop from where you login to administer all this is the best chance an attacker has to breaching all this security. So, ensure there is good role separation for administrators and good security hygiene on all the individuals machines (use password managers, do only official work on these laptops, don't visit random websites, run vetted authorised packages only).

For highly sensitive stuff (like CA or any root of your chain of trust), setup a SCIF and use only terminals inside a SCIF to access/administer them.

Since this guide started with datacenter and hardware security, it is important to discuss the merits of having a secure enclave (hardware security module) on each server to hold the security keys.

Securing the chain of trust is a critical element of securing any system. If the root of the chain of trust is not securely bootstrapped and continuously verified to be still secure, then all bets are off.

The provenance and integrity of binaries, config and data running on your servers should be fully known – their integrity verified with cryptographic methods tied to your chain of trust. There should be no way for these entities to be mutated without cryptographic hardware tokens also being broken. This raises the security bar significantly. (mostly targeted supply chain attacks mounted by highly motivated likely state actors can overcome this bar).

Without this, you are always standing on shaky suspicious ground no matter how much you harden the layers above.

I just keep my prod systems clean, updated and SELinux in enforcing mode. Behind strict firewall of course. This is sufficient for practical reasons.

In proper devops teams, there is no even ssh access. System is deployed from image and configures itself. And is killed in scaling events.

Grsecurity is for hardcore stuff. Or openbsd, ultimate solution :)

Yeah, let's not debug anything, ever.

That'll teach'em not to make mistaeks.

Oversaw a CTO 'harden' a linux box that was used as a firewall and exploited with;

chmod -R 600 /

I wouldn't recommend doing that.

Well, it certainly is secure, there's no denying that.

Especially after the next reboot.

Suggested additions: 1. Don't use a CPU equipped with Intel ME or equivalent, or other system-level management engines like HP's iLO. 2. Look out for keylog devices.

Also, from a practical point of view: sshguard is very useful and easy to install.

You mean, you want to drive to your data centre every single time something fails to boot, or have the awkward debugging session over the phone with remote hands? Or come in with a physical copy of the firmware update every time? Remote management is necessary in some situations. KVMs, ILO, and others have their issues, but it's a tradeoff. "Don't use" is not a great suggestion.

Ok, if you have to use something like iLO, then make sure to connect it only to an internal network. And update the thing frequently.

People expose IPMI to the internet?

lol you people. the internal network concept doesn't exist in any targeted threat model.

I do exactly this. Granted I only have 50 chassis to deal with and can emergency boot to rescue via PXE. After 1st generation ILO and the enforced insecurity of vendor KVM options I saw the light.

> Don't use a CPU equipped with Intel ME or equivalent, or other system-level management engines like HP's iLO.

This is a nice dream, but absolutely not practical in the real world. All the major hardware vendors have an equivalent, eg. CMIC for Cisco UCS, DRAC for DELL, etc etc.

Not to mention the impracticality of not having these when your equipment is locked inside a cage in a datacenter on the other side of town.

Google is purportedly aware of ME and working on removing it: https://www.tomshardware.com/news/google-removing-minix-mana...

But yeah, good luck convincing anyone that it's a good idea to run me_cleaner on production hardware.

I don't think you actually do this.

After thinking about it some more, I guess there are very special cases in which you might want to avoid Intel ME. Mostly if your adversary is the US Gov.

Why put /boot on a separate partition? It becomes a pain on Ubuntu if the size is too small, if you forget to run apt autoremove (or whatever the exact command is) that thing fills up with old kernals. Updates can fall because of this and your system breaks in weird ways.

I know this because the original developer of our system used a /boot partition of 500 MB. That's right, 500 MB. I noped out of that. Don't even get me start on 2 GB total disk space for everything else on our API server. This was built in 2013 mind you, plenty of disk space by then...

I think the advantage is to have /boot read only? so you can't abuse fs security bugs to infect boot files

No mention of 2 factor authentication. Humm

Have fun getting smartcard logon working on a desktop Linux machine (unless you've got a full-blown enterprise PKI in place -- and, even then, it's still gonna be fun).

(cue the "but I did it with a Yubikey" responses)

Or you could use a hosted solution like Duo which has a PAM module which is dead simple to use.

Or pam_google_authenticator.

Seems to be in progress. In the future it will cover 'automatoc security updates'. Anyone have insight into wether this is a good idea and how it's done on something as rolling as an ubuntu server?

--thanks hn:shocks for covering the debian side of things--

I know on centos, I put this in cron 2x a day:

yum update-minimal --security && curl https://logserver/api/$(hostname)/$(needs-restarting -r)

It grabs all the security based updates, and installs only those. And it reports the reboot required status. I watch that and make a reboot ticket if it hits the true endpoint.

Edit: the server part is just Node-red watching for those endpoint GETs. Once it sees a true, it fires a Jira ticket in the Change Management space for a security based reboot.

There is also a dedicated tool, yum-cron.

Also, I thought the --security flag did not have any effect on CentOS?

I believe you are right. The CentOS repos don't have the security metadata?

On Ubuntu and Debian there is a package for this: unattended-upgrades.

The default configuration should work fine for most.

I've had problems on some servers where unattended-upgrades would install new kernel versions without removing the old ones, and ending up filling my /boot partition.

sudo apt install byobu

sudo purge-old-kernels

Ditto, for this reason I leave it off. Also, I want to be able to verify patches in staging. Kill me if you want, but we only patch once a month for this reason. For critical things like heartbleed, meltdown, spectre we fast track them obviously. I find it helpful to subscribe to all security lists of core services to know if something really nasty is out there.

You can install Canonical livepatching, I think it deals with actually applying new kernels and removing old ones. https://www.ubuntu.com/livepatch

This seems good technically, but the writing needs a good once-over. Some directions are awkwardly written and obtuse:

For more security-focused situations is as follows

This is like having a car that you work on more than you drive. Great as a hobby but that's about it.

One thing I miss from that guide.

What about compiling everything with GCC security enabled like Google does on Android?

The fact is; your distro is likely doing this already.

There is a script which reads elf binaries and outputs security info: https://unix.stackexchange.com/a/89214

Not all of them enable ALSR for example, right?

Thanks for the script, I wasn't aware of it.

ASLR is implemented in the kernel; although some unscrupulous programs are able to bypass it entirely (or force it to be disabled system-wide as is the case for Dropbox on mac)

I think what they were getting at is that ASLR doesn't work at all (for the main program) if you're not running a PIE binary. The libraries still get it (they usually have to be PIC by design) but that's not much benefit when the application using those libraries has entirely predictable addresses.

So yeah, your binaries need to be built with CFLAGS="-fPIE" and linked with LDFLAGS="-pie".

incomplete and yet has 1600 stars now. good job. also it's top on HN.

this is not practical...at all.

I have to agree. Table of contents is a bit too ambitious. Anything that wants to cover "Air conditioning", "Fire protection" and configuring Nginx and Apache is not going to be practical.

This would make a great Ansible role too!

This is the beginning of a great list, but to apply everything without careful understanding / adapting to your situation would just end up confusing people and causing them problems. Some things are also very distro/environment specific (like the mounts layout).

There are tools which will verify the situation for you, but not force a specific solution.

There's https://github.com/openstack/ansible-hardening and https://github.com/dev-sec/ansible-os-hardening

Instead of one role for hardening, I try to write my roles with secure defaults: https://github.com/liv-io/ansible-roles-centos

There are already some automation packages around, in dev-sec project we have them for ansible/puppet/chef including automated compliance tests for inspec.

https://dev-sec.io https://github.com/dev-sec/

(disclosure: I'm core maintainer of dev-sec)

Does anybody knowledgeable know how hard would it be to patch the same linux kernel with per-application permissions? I know there is some 'OS' doing it but I am looking for a WSL like solution- 'patch' the existing kernel so I can run the binaries et al unmodified.

There would be a permissioning matrix where I can list allow-all-permissions and deny-all-permissions by default and then change the granularity

Subgraph might be a starting point, https://subgraph.com/sgos/hardening/index.en.html

That sounds like selinux / apparmor / rbac. (Or alternatively I don't understand what you mean by "patch Linux kernel with per-app permissions")

You understood exactly and apparmor clearly fits what I need (and possibly the other two too but I haven't had time to look...). I used to (incorrectly) believe that SELinux only dealt with networking...

Reply on news.ycombinator.com