Hacker News Reader

Comment by unnouinceput | Hacker News Reader

Quote: "Note that host is hardcoded to localhost. This means it should never connect to any machine other than the developer machine. Also: of course we use different passwords and users for development and production. We’re too tired to figure it out right now.

The gremlins won this time."

No they didn't. Instead one of your gremlins ran this function directly on the production machine. This isn't rocket science, just the common sense conclusion. Now it would be a good time to check those auditing logs / access logs you're suppose to have them enabled on said production machine.

View on HN

> Instead one of your gremlins ran this function directly on the production machine.

Exactly my first hypothesis too. But then keepthescore claims,

> of course we use different passwords and users for development and production.

How would this hypothesis explain that?

---

Metadialogue:

John Watson: "I deduce that someone changed the dev config source so that it uses the production config values."

Sherlock Holmes: "My dear Watson, while that is sensible, it seems to me that the balance of probability leans towards their production instance also having the development access credentials."

---

Just my way of saying, I think this case isn't as shut and closed as most comments (including parent) imply. I personally find the /etc/host mapping a likelier hypothesis but even that can't explain how different credentials failed to prevent this. Without more details coming from a proper investigation, we are just piling assumptions on top of assumptions. We are making bricks, without enough clay, as Holmes would say.

Agreed, it seems like most people making suggestions above are missing the point about credentials. The code they present explicitly references `config.DevelopmentConfig`:

    database = config.DevelopmentConfig.DB_DATABASE
    user = config.DevelopmentConfig.DB_USERNAME
    password = config.DevelopmentConfig.DB_PASSWORD

One way this could happen is if the objects under `config` were loaded from separate files, and the dev file was changed to a symlink to the prod file. So `config.DevelopmentConfig` always loads /opt/myapp/config/dev.cfg but a developer had dev.cfg -> prod.cfg and the prod credentials and connection details were loaded into `config.DevelopmentConfig`.

Just an idea.

If managed DBs on DigitalOcean are anything like those on AWS, you can not run commands directly on them since SSH is prohibited. EDIT: there's also the deal with different credentials for dev and prod envs.

2 cents his hosts file points localhost to the prod db IP

Yep. Nowadays a kubectl port-forward makes something like this all too easy. They accidentally had the kubecontext point at the production cluster instead of dev, set up the port-forward to the database, and whoops! At least that's how this could happen to me, even with my years of experience in doing unexpected things to production databases.

That whole ecosystem is a devexp nightmare. I try to stay away from it entirely having worked with it extensively.

Docker + Kubernetes are the biggest socially-acceptable hacks in the industry at the moment.

Why do you think they are hacks? Could you please elaborate?

In my opinion, they nicely abstract over server hardware and services running on them, so one is able to have simple infrastructure-as-code management of otherwise complicated setups.

I think so as well.

It doesn't even make sense to connect to a managed database using 'localhost'.

Managed databases are never localhost. They are hosted outside your VPS and you use a DNS name to connect to them.

That could've happened if the database is not accessible from the Internet and they were using a tunnel which binds to localhost (e.g. `ssh -L`).

It does make sense to connect to localhost on dev machines. But if that's the setup, I guess one should avoid from tunneling to localhost to avoid potentially dangerous confusion (hmm... I think I'm guilty of that on a couple projects, need to check if that's true...)

Yeah, my guess would be that the script got executed on prod server by, ups was I in that terminal window, accident! Localhost is after all the-local-host, no matter what server it's on. Better to also have clear convention regarding name of prod db name versus test db (i.e. "test_mysaas" versus "mysaas").

Plus of course using git with a hook specifically for preview versus production (i.e. "git push production") that way local specific scripts can be stripped even if in same repo.

Yeah this is how I once deleted my production database. One thing I did to mitigate this was colour code the prompts for local/staging/production.

Yeah, if one has to regularly interact directly with prod, some sort of visual indicator is super helpful.

I'm betting on a tunnel, myself. And grandparent is probably wrong, they most likely have dedicated mysql machines so "localhost" will never be the db.

One solution to this is to make sure only the production servers can connect to the production database.

My guess would be something like PgBouncer. Someone may have installed it to the production server at some point in the past.

As someone else said it below: "hardcoded to localhost" doesn't mean it's hardcoded. It means it goes to whatever localhost resolves to. Really hardcoded should ALWAYS mean: 127.0.0.1

Reply on news.ycombinator.com