Tell HN: HN Replies Data Security Incident

I'm not an hnreplies user, but IMO this is exactly what an incident disclosure should look:

  - What happened  
  - The extent of the damage  
  - Why it happened  
  - Why it is less likely to happen again in the future

Software is made by people, and people make mistakes. I don't think anyone is asking for perfect software (particularly not on this forum, and particularly not for free services offered by the community). But a clear picture of the breach and why it occured allows the user to make an informed decision about whether a service is worth using.

Maybe this is evidence of how low the bar is for incident reporting, or maybe its evidence of of "no publicity is bad publicity," but I just signed up for hnreplies.

Great service idea, and thanks for the honest and helpful disclosure!

I agree and it's why we must be prepared for WHEN "it" happens, NOT IF "it" happens.

"I am convinced that there are only two types of companies: those that have been hacked and those that will be. And even they are converging into one category: companies that have been hacked and will be hacked again."

-- Robert Mueller, 2012, when he was Director of the FBI [yes, that Robert Mueller]

https://archives.fbi.gov/archives/news/speeches/combating-th...

Plus it does not contain the classic line "At [...] we take security very seriously" which to me is one of the most stupid things to say when you've just been breached.

To be fair, accidents do happen. Nuclear reactors take things pretty seriously but they get breached / melt down occasionally too.

> a .git directory on the VPS hosting this site was exposed to the internet by a configuration mistake, providing enough information to allow an attacker to access the site's database.

I had trouble figuring out how they got from point A to point B. My best guess is that a file containing database configuration settings had been committed to the Git repo at some point.

Lessons:

- Don't commit secrets to your Git repo. Store them in config files which are listed in .gitignore. Yes, this means you have to manually copy or write config files when you set up different servers, but those servers are likely to have different database credentials and the like anyway, right? If you've found that you've committed secrets by accident but haven't pushed anywhere yet, revert your branch to before the commit and redo or cherry-pick any subsequent commits safely. If it is too late, change the credentials to access the database or whatever other secrets you've revealed.

- Don't put web-accessible files in the root level of a Git repository. Instead, have a subdirectory contain the web-accessible files, and configure your web server to serve from that. That ensures that the .git directory won't be right next to web-accessible files, and has other benefits as well - you can also easily have subdirectories for docs, CLI scripts, and other things that shouldn't be web-accessible right there in the same repo. But if that isn't an option (or even if it is), configure your web daemon to not serve files from .git, or probably any other file or directory that starts with a dot.

- Edit to add one more: As much as possible, lock down access to your database server. An attacker having your database credentials is bad, but they still won't get far if the database server is configured to only listen to connections from the IP address that the web application server is hosted at, or, if they're on the same server, only over local sockets.

Of course, the same applies to other VCSes besides Git.

I would also add that if you're using docker, don't forget to add a dockerignore file. I often see people miss this step and end up with exposed secrets and git folder (and poor build times).

> have a subdirectory contain the web-accessible files

You can also separate your git repository from the working directory. I have my git repositories at ~/repositories, with a post-receive hook that checks out the working directories at ~/projects and then _actively_ copies only relevant files to ~/html. Basically a "poor man's CI/CD", with the bonus of having to opt-in to publish any files.

You can also tell git that the .git dir is completely separated from the working dir:

    alias dogit='GIT_WORK_TREE=/path/to/src/dir GIT_DIR=/path/to/dotgit/dir git'
    dogit pull remote treeish

---

I set this up as a "poor man's version control" on a large client's server back in 2016-ish. I was lost for words when last year I saw my colleagues still using Ctrl-R to rerun that alias line.

Yes, that's roughtly what I am doing in my post-receive hook:

    export GIT_DIR=/path/repo
    export GIT_WORK_TREE=/path/working-directory
    git checkout -f

Personally my lazy solution to config files is using ansible-vault.

It basically lets you encrypt your secrets and store them in your git how much you want. You then only need one password when decrypting the vault.

> Yes, this means you have to manually copy or write config files when you set up different servers, but those servers are likely to have different database credentials and the like anyway, right?

No, you can use GitHub Secrets, AWS Secrets Manager, etc ... or encrypt your config files and add a decrypt step to your build. [1]

[1] https://docs.github.com/en/actions/reference/encrypted-secre...

Just want to say thanks to Dan Grossman for providing this free service. I find it really useful.

Same here, kudos Dan! HN Replies has allowed a lot of interesting conversations to happen that probably would not have taken place otherwise.

Best of luck with everything

Came here to say the same. I really appreciate the awesome service. I also really appreciate being upfront with the incident.

I badly hope nobody is going to be offensive here. (Mind you, I’m watching!)

THANK YOU, Dan!

I also just got this email and it actually fills me with confidence. Everyone makes mistakes and while I'm a lot better about using ENV vars and the like to pass secrets into code, it was wasn't that long ago that I was committing secrets to git because it was way easier. Thankfully this was a rather low-stakes mistake for me as my email and username are already linked (though I understand that is not the case for everyone). I'm also very happy with how Dan handled this. A user reached out to warn him about the breach, he investigated, found the issue, and communicated it to his users. A /very/ different situation from how Ubiquiti handled a much higher-stakes breach.

>For a period of several days earlier this year, a .git directory on the VPS hosting this site was exposed to the internet by a configuration mistake, providing enough information to allow an attacker to access the site's database.

Storing credentials in git. Ahh a classic!

Binding the database to 0.0.0.0 is not a good idea.

Thanks for the free service, email getting leaked is the least of my concerns. Can't get mad, thank you Dan. HN should have this natively

Promotes rapid responses, which doesn't promote well considered, goal-oriented, and empathetic responses.

My identity is clear from my profile link so this is no hassle for me, but I assume there could be at least loose repercussions for people who followed their own, anonymous-but-opinionated accounts?

some HN user once observed that this was a fun single point of failure to pwn, mapping HN usernames to emails. people who intend to stay anonymous need to be aware this is a risk when using these services.

I signed up for the HN replies fully expecting it to be pwned sooner or later. Actually, I assume pretty much every service to be pwned at some point, and sign up according to how it'd impact me. If I don't care at all about the association, I use my real address, if I care somewhat I use a "+something" gmail address.

And if I REALLY don't want to be associated, I use a relay email address.

TIL hnreplies exists! It's exactly what I've been looking for.

There is no such thing as bad publicity

https://idioms.thefreedictionary.com/there+is+no+such+thing+...

Got the same email. While this isn't a huge deal to me, for those that'd prefer to run their own RSS feed of their comments, I wrote this a while ago, before I knew about this service:

https://gist.github.com/Q726kbXuN/15e61acc003bb6d46a458001fe...

I updated it to look like the emails this service sends, though it should be straight forward enough to change.

(Standard "this was written in a day with no care for maintainability and code prettiness" warnings)

I've been using this RSS site to get notified of replies:

https://hnrss.github.io/

I did write a simple rss2email application, which I run as a daemon to trigger the notifications, but there are plenty of other rss2email applications out there:

https://github.com/skx/rss2email/

EDIT: my comment is not relevant to this thread. Sorry!

Since 2014[0], the HN search API has supported an RSS feed of a user's comments, e.g.

https://hn.algolia.com/userfeed/banana_giraffe

[0] https://news.ycombinator.com/item?id=7527759

Fair enough, though the site mention in this post here, and my little python script don't provide a feed of a user's comments, but a feed of replies to a user's comments.

Oops. My comment was irrelevant :)

Also wanted to chime in and say thanks for the service! As others have mentioned it has allowed me to have interesting conversations I wouldn't otherwise have had.

And kudos for only storing truly necessary data!

I hope Dan doesn't get much hassle or negative feedback from this. As a free service it's always going to be all care and no responsibility and as someone from outside of a major timezone, I find it very useful.

Good time for HN to implement its own replies notification feature. The best time to do so was a decade ago. The second best time is today.

I think they’ve explicitly decided against such a feature to reduce addictive engagement patterns, and limit potential for long running flamewars.

The feature exists, as shown here. But it is clunky, provided by a third party. The more elegant solution would be to be implemented in the core product but turned off by default of the concern you’ve mentioned is the actual reason.

“ I think they’ve explicitly decided against [providing] such a feature”. Of course it exists, I don’t think they want to make it a part of the experience they provide.

me, too.

:(

https://www.thecscience.com/