Hacker News Reader

Comment by fredthedeadhead | Hacker News Reader

The blog post is 404ing, here's a Web archive link

https://web.archive.org/web/20240610032818/https://asim.bear...

The author has added an important edit:

> I want to preface this by saying yes the practices here are very bad and embarrassing (and we've since added robust unit/integration tests and alerting/logging), could/should have been avoided, were human errors beyond anything, and very obvious in hindsight.

> This was from a different time under large time constraints at the very earliest stages (first few weeks) of a company. I'm mostly just sharing this as a funny story with unique circumstances surrounding bug reproducibility in prod (due again to our own stupidity) Please read with that in mind

View on HN

Could they have deleted it because of all the negativity?

They did make a silly mistake, but we are humans, and humans, be it individually or collectively, do make silly mistakes.

If you code for a hobby/fun, yeah, sure, it's a silly mistake.

If you're earning past six figures, are part of a team of programmers, call yourself an professional / engineer, and have technical management above you like a VP of Engineering, yadda yadda....then it's closer to systematic failure of the company's engineering practices than "mistake."

There is a reason we call it software engineering, not software fuckarounding (or, cough, "DevOps Engineeer".)

Software engineering practices assume people are going to make mistakes, and implements procedures to reduce the chances of that making it into production, and reduce the impact of those mistakes if they do make it into production.

This is how I felt when a Gitlab employee deleted the production database by doing it in the wrong terminal window.

Get into the habit of colour-coding your important SSH sessions, red/green/blue can give you a very powerful subconscious check before doing something very silly.

How do you do this so that it lasts? I find mine resets to default

Can’t you set it in the .zshrc file?

I use iTerm2 and have different profiles for different directories/remote hosts. iTerm switches the background colour automatically for me. I'm sure other terminal emulators would have similar features.

https://bsago.me/tech-notes/change-ssh-background-colour-wit....

what phist wrote. Color code the backgrounds of your servers, different colors. So anyone who connects to 'take console' in any system is hit by a blinding electric Green/Blue/Red/Yellow and other striking colors.

I assume that all systems already have descriptive names App_DEV_Server1, App_PROD_Server5, etc.

It also helps if (ofc they would be right??) in separate IP groups/WLANS?

If you are running Windows, it's a good idea to use BGINFO.exe by SysInternals (or Winternals as we old people still call it), and display the most relevant info (showing Dev/Prod/UAT/etc.) with big-big-big letters.

Why do employees have write access to production DB?

Lazy attitude towards proper role management and poor engineering practices. More common in small companies or small teams managing their own service (db and app)

Really all you need is logging and potentially temporary read access to the db if you need some info that you can't derive from the logs.

Post mortem for the outage I referred to.

https://about.gitlab.com/blog/2017/02/10/postmortem-of-datab...

> If you're earning past six figures, are part of a team of programmers...

Compensation is in no way correlated with good engineering practices.

They might be paid much because they're developing something which people are willing to pay for, it doesn't have to be "real engineering".

I agree, but in fairness, engineering mistakes do happen all the time, in every organisation. A good engineering culture enables mistakes to be acknowledged and reviewed in an emotionally neutral manner, ideally leaning to a learning experience.

Being on the receiving end of an internet pile-on of "OMG you idiots everyone knows the first thing you do when setting up a flerble cluster is spend a week installing grazoono monitoring!" is not conducive towards building a good engineering culture.

If I configure yaml or json files with a markov chain am I a idiot? What if I use chat gpt?

Calling yourself an engineer doesn’t make you one.

If they would have made that mistake by writing code and just misunderstood something or oversaw the problem, fine. But making this mistake by copy-pasting from ChatGPT without proper review is just terrible.

I don’t find the source of the error being a careless human writing original code without proper review vs a careless human copy/pasting code without proper review to be significantly different.

The difference is that everyone knows about hallucinations, so an LLM never can be trusted by default, they still trusted it blindly.

Is that really worse than trusting blindly the code of a human, about which "everyone knows about" the bugs that humans write as well.

Seems like he changed the subdomain or something, the article is still up

https://0912i390129ionkjan.bearblog.dev/how-a-single-chatgpt...

Also 404 now

the page on archive.org could not be loaded now...

but google cache still serves a copy...

https://webcache.googleusercontent.com/search?q=cache%3Ahttp...

Reply on news.ycombinator.com