Hacker News Reader

Comment by xtracto | Hacker News Reader

This happened to me (someone in my team) a while ago but with mongo. The production database was ssh-tunneled to the default port of the guys computer and he ran tests that cleaned the database first.

Now... our scenario was such that we could NOT lose those 7 hours because each customer record lost meant $5000 usd penalty.

What saved us is that I knew about the oplog (binlog in mysql) so after restoring the backup i isolated the last N hours lost from the log and replayed it on the database.

Lesson learned and a lucky save.

View on HN

Same happened to me many years ago. QA dropped the prod db. It's been many years but if I recall, I believe in the dropdown menu of the MongoDB browser, exit & drop database were next to each other...Spent a whole night replaying the oplog.

No one owned up to it, but had a pretty good idea who it was.

> No one owned up to it, but had a pretty good idea who it was.

That sounds like you're putting (some of) the blame on whoever misclicked. As opposed to everyone who has allowed this insanely dangerous situation to exist.

Misclicking is a tiny forgivable mistake.

Not immediately calling up your boss to say "I fucked up big" is not a mistake, it is a conscious bad action.

It sounds like whoever did it might not even be aware they were responsible:

> in the dropdown menu of the MongoDB browser, exit & drop database were next to each other

So maybe they signed off for the night without realizing anything was wrong.

MongoDB can sell an enterprise version with the buttons further apart

This. The person that erased the database in my case came forward to me as soon as we realized what had happened. At that moment I was very happy it was an "inside job", it meant I could discard hacking.

As its said before: he made a mistake. The error was allowing the prod database to to be port forwarded from a non prod environment. As head of eng that was MY error. So I owned to it and we changed policies.

How do you prevent forwarding ports? Then one needs to disable ssh access?

Nice that you were a person he felt ok with sharing the mistake with, I suppose that's an important part of being head of eng.

`AllowTcpForwarding No`

There are ways around it, of course, but it prevents the scenario described above.

Thanks

Nope . The solution is to password protect and not give the pass to developers. Or only give read only access.

Another thought: the company culture and approach to hiring and firing, can cause people to try to hide mistakes, although they don't really want to?

Which MongoDB browser or admin tool are you referring to?

I haven't seen this design in practice using MongoDB Atlas or Compass, but would hope for an "Are you really sure?" confirmation in an admin UI.

A dangling port-forward was my first thought to how this happened.

Reply on news.ycombinator.com