Hacker News Reader

Comment by chmod775 | Hacker News Reader

No. You can't assume your logs are trustworthy if you are compromised. You have to assume an attacker will remove any traces of themselves from those logs or will stop them from being written in the first place.

It's pretty standard practice to do this post-exploitation.

Further this requires you to regularly check your log files for suspicious activity, which is way more work than just rotating your ssh keys - which can be easily automated. Running a script that will automatically rotate your ssh keys on all servers in your .ssh/known_hosts is trivial.

Also, rotating your ssh keys is something that has a chance to prevent intrusion, whereas if you see something in your logs it's already too late.

View on HN

> You can't assume your logs are trustworthy if you are compromised.

It's not that simple. In a larger/mature environment you'll have log aggregation where the initial login is close to certain to be forwarded before it can be messed with. (Unless someone can log in, escalate, kill the right daemon, and somehow prevent monitoring from noticing a missing endpoint - all before the log gets forwarded)

And that's just host logs, without the networking, potential forwarding, etc.

You can assume a lot of logs are not compromised. For example, logs from your network infrastructure. In this case you could trust sshd logs (so long as you ship them off quickly) and network logs.

As for endpoint logs, yes, a privileged attacker could disrupt them. But honestly, even with regards to your endpoint, attackers often don't disable logging - though I do see it, for sure. For "blessed" logging like Windows Event Log you'll have an even harder time - they do take measures to protect the files on disk, even against privileged attackers, and supported methods of deleting the event log actually themselves generate a "Someone deleted the event log" event, which I would highly recommend you watch out for :)

The simplest advice for dealing with this is to ship logs off of the device ASAP and to make sure that disrupting the service requires privileges.

> For "blessed" logging like Windows Event Log you'll have an even harder time [...] even against privileged attackers

Trying to prevent a determined privileged attacker from doing something is an exercise in futility, since it's impossible in every sense of the word. If they have total control over a system, they can do whatever they want, even if you put up a bunch of stopgaps.

Most post-exploitation frameworks (prominent example: DanderSpritz) have modules to remove stuff from the windows event logs without leaving traces.

It's pretty basic stuff.

A log immediately stored on a "physically" write-once device, or printed, may be more difficult to dispose of (w/o physical access to the premises).

> Trying to prevent a determined privileged attacker from doing something is an exercise in futility,

That's not really true at all.

> since it's impossible in every sense of the word

Ok? Lots of things are impossible, and lots of those things are also still very very hard and costly. Hash collisions aren't impossible, and yet here we are, with a world hinging on them being very hard.

> , they can do whatever they want

Not really.

> Most post-exploitation frameworks (prominent example: DanderSpritz) have modules to remove stuff from the windows event logs without leaving traces.

You realize that:

a) DanderSpritz's logic to bypass the event log was a huge deal

b) It's literally an NSA leaked exploit????? Like are you kidding me using an NSA developed exploit as "pretty basic stuff" lol

Sorry but I think I'm gonna stop responding here.

> b) It's literally an NSA leaked exploit????? Like are you kidding me using an NSA developed exploit as "pretty basic stuff" lol

It must have been very expensive and innovative, does that make it hard to copy into your scriptz folder?

I don't understand the point you're trying to make. That because one exploit exists, and using that exploit is cheap, the entire class of attacks is cheap?

Sorry, but this entire thread is nonsense, and it's just a clear demonstration of a lack of threat modeling and frankly a lack of understanding of what attacker capabilities are.

The basic principle of real security models is once an attacker has full access all data on the system is potentially tampered with.

You can try to play games with ensuring logs leave the system, but everyone takes shortcuts to make sure they can recover the system when networking is down etc.

Everyone and their grandma has access to complex scripts, etc that were once very expensive. Whether they invest the energy in learning methods to hide their presence or just go straight to some other goal is going to depend on how they intend to abuse your systems.

I think most "security professionals" pretend they are going to catch an oddity that occurs that doesn't match what their automated tools would catch and occurs in the middle of some other crisis or holiday break. I would say good luck with that.

> The basic principle of real security models is once an attacker has full access all data on the system is potentially tampered with.

That's not true at all. Like, not at all.

> You have to assume an attacker will remove any traces of themselves from those logs or will stop them from being written in the first place.

You’re assuming the attacker has write access to the log storage of the system they ssh into. This is not the norm for production systems. If your auth processes aren’t shipping off logs immediately, your system is broken regardless of ssh.

Ship them off to where? A system that also can be accessed with your ssh key?

If not, can an attacker gain access to your email server or DNS with your ssh key? If either is true, they now have access to everything not protected by 2FA that uses an email address they now control.

There's so many things to get right.

You can design a system such that there's a very high likelyhood even a nation-state attacker won't be able to intrude without leaving traces - if you make no mistakes.

Or you know, you could also just rotate your ssh keys in addition to everything else. "I have logs" is really no excuse to forego something that is this easy.

> Ship them off to where? A system that also can be accessed with your ssh key?

If the ssh key you lost has access to all the critical infrastructure, then, certainly you have a problem. The solution is to not give away write-access keys to your entire system.

> If either is true, they now have access to everything not protected by 2FA that uses an email address they now control.

The initial question was not whether losing a key would cause a breach, but whether a detection mechanism is reliable.

> Or you know, you could also just rotate your ssh keys in addition to everything else.

The question is how does it help? If you can't detect a breach, it will live for as long as your key rotation policy, if not longer. If you can detect and mitigate a breach, it will be closed quickly regardless of your rotation policy.

I've not seen an enterprise environment where "ordinary" sysadmins also have access to the log hosts. Must be extremely rare setup in a healthy environment. Likewise selldom to see "logging staff" to actually login to a shell. Logs naturally aggregates to some kind of a service with visualizations, alerting and filtering capabilities.

In those special cases an "extraordinary" sysadmin gets onboard a log host, it is not through the ordinary access ways, such as SSH from where the other sysadmins play around.

Ship them off to where?

The usual networked logging systems that don't have ssh logins, e.g. syslog into ELK stack.

How do you set up elastic on a server that doesn't run sshd?

For real? console (in various ways/flavours) is one way.

Console. Cloud login. Different VPN connection. Different credential set then other hosts. Using a hosted service from a completely separate provider. Setting up infrastructure from images without external access.

And I'm sure I missed some - there's a thousand ways to do it.

  curl -X "DELETE" http://log-server:9200/logstash-*

Probably works for a large percentage of deployed ELK logging stacks.

Are people not using elastic authentication?

Elasticsearch existed for years without any authentication on its community tier version, with xpack behind a call with a sales rep and a heavy pocketbook. These days, authentication is provided after you enable the free xpack plugin, but it is not enabled by default and their install guide doesn't exactly point it out to you right away. It starts delving into JVM tuning options before it even references there's this xpack thing you may want to look into.

I always just put it behind an authenticating proxy.

Holy shit you should not have access to your logging systems/DNS servers with the same exact credentials as your main app servers. Do you let devs onto the HR system with the same ssh key as well?

Reply on news.ycombinator.com