Posting successful SSH logins to Slack | Hacker News Reader

Posting successful SSH logins to Slack

sandrinodimattia.net

133 pointssandm66 comments

Open articleView on HN

This is neat! Threw together an Ansible role for it:

https://github.com/duggan/ansible-slack-notify-ssh

Is Slack really the right place for security-critical notifications?

yes?

I mean the next step is to have an automated phone call go out to people (which is what we do for critical alerts).

Short of that, slack is on my desktop, laptop, and phone. If i don't have one of those around me at the time, you aren't getting ahold of me for any reason.

So yeah i think it's perfectly valid for security-critical notifications. Plus this isn't as security critical as you'd think. I don't want klaxons going off every time someone sshs into a server... This can just be an additional layer of security.

Slack is terrible for auditing, though. What's wrong with email?

people are more likely to be looking at chat windows as opposed to emails

Not in my book. Slack seems to be really cool but since it's not self-hosted and owned by a US entity, I'll stay clear.

It's an ssh login notification with a user and IP address. It's not notifying everyone what the new launch codes are. Let's not overstate it.

I'm not referring to the SSH logging, I mean slack in general.

What would you use instead?

Mattermost

or Actor.im

IRC on a server I control, or some slack clone with on-premise hosting. Call me paranoid, but I'm paranoid.

I still don't understand why everyone is so excited about slack, it really doesn't offer that much more than IRC.

It offers marketing and support, that's about it.

Nobody is selling IRC.

cough www.grove.io would like a word with you ;-)

Looks cool, but

>We're sorry but Grove isn't currently accepting new customers right now.

I guess with that pricing structure they don't need to.

Well shucks, that is news to me. Thanks for the update.

I use IRC, but come on. It offers a lot more than IRC right out of the box.

* File uploads

* Embedding portions of links (tweets, images)

* A very good search

* Multi-line posts

* Code-formatting, including multi-line posts, and also snippets.

* A mobile client that alerts you when someone mentions you.

* Scroll-back history when you sign on at any time.

* Syncing between multiple clients.

Yes, you could create a bot or modify an irc server to do this, and then find or write a client that will do all that stuff, and an irc bouncer can fill in for a lot of this.

But Slack does it out of the box. Zero extra work needed.

I like IRC, but if you claim that Slack doesn't offer anything more than IRC, you're either delusional or using an incredibly broad definition of IRC.

We just discovered that some developers at my workplace use their domain user account credentials for Slack as well. My manager was not happy.

Are you asking if a communication platform is a good place for communications?

Yes.

If you want to send email on login, add in /etc/profile :

echo "`whoami` logged in at `date` from `echo $SSH_CLIENT`" | mail -s "`hostname` login" youremail@example.com

Note that people can still ssh execute remotely etc.

You'll definitely want to add a '&' at the end of that line so that you don't delay user logins if the network is down or mail barfs.

right, actually I do have a delay in a server with no mail where it fails :-)

pam_exec seems to be a more robust solution to login notifications than /etc/profile http://blog.stalkr.net/2010/11/login-notifications-pamexec-s...

I recommend using something like OSSEC to watch your logs and also tie it to Slack/Pagerduty.

This post explains how to set it up:

https://blog.sucuri.net/2016/01/server-security-integrating-...

I did a similar thing a couple of days ago. I just added this (with the correct values) in the `sshrc` file inside `/etc/ssh` and enabled a webhook. that's it.

ip=`echo $SSH_CONNECTION | cut -d " " -f 1`

curl -X POST --data-urlencode 'payload={"channel": "#<your channel>", "username": "SSH Login watcher", "text": "User '${USER}' just logged in from '${ip}'", "icon_emoji": ":robot_face:"}' https://hooks.slack.com/services/<rest of the webhook>

I do the same thing, except I email the logins to myself with SEC: https://simple-evcorr.github.io/

Elegant. Thank you! Is there a way to extend it to override DND?

That's a great idea. But I didn't find any docs that explain how to override DND. I opened a feature request instead, so we'll see how it goes :)

That kind of gets you into this sort of problem: https://blogs.msdn.microsoft.com/oldnewthing/20110310-00/?p=...

If you start overriding DND, now the user is going to want super-DND. Which somebody will then want to override, and so on. The correct solution is that your users need to not set DND when they in fact need to be disturbed, and your systems shouldn't be disturbing unnecessarily, and to the extent that's a really hard problem, well, yes, it very much is, but an unboundedly-large hierarchy of "bother that person, no don't bother me, SUPER bother that person, no SUPER don't bother me, SUPER MEGA bother that person" isn't part of the solution set.

The biggest problem with DND in my book is that when first introduced it was enabled automatically, and not super obvious that it was enabled.

This caused more than a few missed announcements and made escalation hard for a bit.

Is that a good idea? You don't necessarily want to wake up the entire team with an alert if there's a dedicated on-call person.

Love this, great idea! I've been trying to setup useful Slack integrations lately and this is a really clever use of them.

Excellent. I've been thinking about having a SSHD keylogger post to slack (or some other log). It's crazy that sshd doesn't have this functionality built-in. It's so important to know what your admins are executing on your machines. Aside from the fact that they might have been compromised, it's just good to know what sort of general administration is being done.

What about logwatch? That can do the same and a lot more. You can set it to mail you daily, which gives you an overview who logged in and how often. With a weekly mail you don't get these details, but it might be good enough.

If you don't want to install a "real" solution like Snoopy Logger that works for all users/shells/edge cases, you could always adapt the bash prompt to write the command out. E.g. I personally use something like this in my .bashrc which logs everything per user, but you could easily adapt this to post to Slack instead:

    # Adapted from https://unix.stackexchange.com/questions/207813/how-to-log-every-command-typed-into-bash-and-every-file-operation
    export ETERNAL_AUDIT_LOGFILE=~/.bash_eternal_auditlog
    PROMPT_COMMAND='RET_VAL=$?; history -a; echo "$(who am i | sed -e "s/[[:space:]]\+/ /g") [$$]: $(history 1 | sed "s/^[ ]*[0-9]\+[ ]*//" ) [$RET_VAL]" >> $ETERNAL_AUDIT_LOGFILE'

Output including return code and all parameters:

    ubuntu pts/0 2016-01-22 13:24 (example-loggedinuser-rdns.yourisp.com) [4379]: [2016-01-22 13:25:37] ps aux | grep python [0]

If you assume no malicious users this will work just fine.

I've looked into both Snoopy and a bash script as you said. They both have merits, but I think for it to be really reliable there's no better way than to just log keyboard input. Assuming there are malicious users for me is a big part of the motivation.

(disclaimer, I'm a co-founder of ScaleFT)

If you are interested in a commercial solution in this space, check out ScaleFT. Besides the dynamic SSH certificates, we also track both SSH keys used and access events, which you can then pull via an API across all your machines. We also have a Slack notifier, though It needs a little love & cleanup.

We're currently in what I could best call a beta: https://www.scaleft.com/

> It's so important to know what your admins are executing on your machines...

Micromanagement at its finest!

> it's just good to know what sort of general administration is being done.

Your change management process will give you an overview of what your admins are doing.

Could be used for micromanagement. But its required in some environments by some common regulations.

Eh, the sysadmins would be monitoring eachother. I'm not saying they should have a manager that keeps their commands in check.

And yes a change management process is very nice and all, and I suppose that at Amazon no line is entered into a root sshd shell without each character being vetted thrice, but at your regular shop you can bet that there's loads of admins that type "ps aux" three times before getting it right. Not that that's terrible, but if you want to look at system administration as an engineering problem you have to know what's going on.

Incident response. When one of your admin accounts is compromised, you'd want to know what the attacker executed.

Yes you would - but why just SSH? Wouldn't auditd execve syscall logs sent to a logstash server be better? It'd handle compromises other than SSH too.

Yes - though there's more to a SSH session than executing commands (interacting with interactive editors, port forwarding, etc.)

There's a github talk about using Hubot to do sysadmin. One of the advantages of having everything happening in a public forum is teaching.

Some shells do have this built in, for example bash's history file, typically set to ~/.bash_history

Which is easily defeated, of course ...

Some file systems support append-only permissions, although I've never used them. I wonder if that would work with bash's history file?

No, since you can just turn it off or prepend all your commands with a space to hide them.

You can remove ignorespace from HISTCONTROL for the latter, and for the former, can't you just remove chattr's execute permission for that user?

The user can point $HISTFILE elsewhere (e.g., /dev/null), `unset` it, use another shell, ...

It looks like you could lock down the environment variables with rbash http://unix.stackexchange.com/questions/66627/is-there-anywa...

But that's getting into no-man's land I guess

man syslog

That's some very sensitive information to put in Slack. If Slack has a security breach or if any of your slack accounts are hacked then an outside party could see this channel.

Ever accidentally typed your sudo password at the wrong prompt?

How quickly we forget that Slack was hacked not more than a year ago.

How can we forget that? Literally any time Slack is mentioned on HN, people come out of the woodwork to let everyone know they don't use it cause it's hosted by someone else.

People are _way_ too trusting of Slack and similar services.

You could run your own IRC or Mattermost server accessible only in your LAN or over VPN and this would be fairly safe. Heck, even SSL-only with cranked up SSL options would be fairly safe.

Instead, people ship all their data off to a 3rd party company, often letting that 3rd party have control over their servers and such. Their customer data, their payment data, even code exec on all their servers is exposed and out of their control.

It's insanity, I can't believe people have so little regard for security. Especially with a 3rd party company with a less-than-great security record.

People will always feel comfortable trading privacy for convenience!

Very unfortunate but I feel it may be true.

And liberty for the feeling of safety.

I use LogWatch, gives me daily emails with nice summaries of that happened on my Ubuntu Droplet: https://www.digitalocean.com/community/tutorials/how-to-inst...

Look into auditd for logging execve() syscall instead. OSSEC can (directly) report or act on any thing reported through logs.

And send it to Slack now as well:

https://blog.sucuri.net/2016/01/server-security-integrating-...

Surprised at how many people didn't read parent's post as satire...

The author didn't walk it back when challenged.

srvAudit also does this, though it's still early in development. srvaudit.com

Reply on news.ycombinator.com