If you haven't fixed that alerting deficiency, then you haven't really fixed anything.
Programming when everything works is easy, it's handling the problems that makes it hard.
"Under construction "
Looks like the OP removed the post?
EDIT: Found archived copy of post: http://web.archive.org/web/20240609213809/https://asim.bearb...
As soon as you expect paying customers in your system you need to have someone with the knowledge and experience to deal with infrastructure. That means logging, monitoring, alerting, security etc.
DevOps.. amateurs.
But not having error logging/alerts on your db ? That's the crazy part.
This is a new product, is not legacy code from 20 years ago when they thought it was a neat idea to just throw stuff at the db raw, and check for db errors to do data validation, so alerts are hard because there's so many expected errrors.
Now granted, if they'd carried on doing stuff (but not redeployed production) it may have shown up in say mid-aftenoon, or not depending on volume.
And of course the general guideline of not deploying anything to production early in the day, or on Friday, is still valid.
And it's not trivial to set up a correct alert for this one. Simple HTTP 5xx threshold wouldn't work because it's high enough to not wake you up whenever cloud provider restarts Postgres, it's too high to catch this. You need either per-endpoint failure rate alerts or something else more clever.
One of the reasons I use Go whenever possible is that it removes a lot of the classic Python footguns. If you are going to rewrite your backend from Javascript, why would you rewrite it in another untyped, error-prone language?
In go, I've definitely seen:
tx, err := db.Tx()
defer tx.Commit() // silently ignores the error on committing, which is the important one
In python, if you ignore an exception entirely, like I did that error above, you instead get an exception logged by default.
Python's exceptions also include line numbers, where as Go errors by default wouldn't show you _which_ object has a conflict, even if you logged it.
In general, python's logs are way better than Go's, and exceptions make it way harder to ignore errors entirely than Go's strategy.
I suppose there are probably similar checkers for Python that would have caught the passing of a scalar value instead of a function.
Perhaps this is an argument for mandatory linting in CI.
What did they say was the thinking behind it? defer tx.Rollback() would make sense, but defer tx.Commit() is nonsensical, regardless of whether or not the error is handled. It seems apparent that the problem there isn't forgetting to check an error, but that someone got their logic all mixed up, confusing rollback with commit.
It's not pure nonsense, it works in the happy path, and it matches the pattern of how people often handle file IO in go.
f, err := os.OpenFile(...)
defer f.Close()
Those two patterns are so similar, python handles them in the same way:
with db.begin() as conn: # implicit transaction, gets automatically committed
with open(...) as f: # implicit file open + close pair, automatically closed
If you are only reading, then sure, who cares?
Okay, sure, but Rollback is the cleanup function. You always want to rollback – you only sometimes want to commit. I suppose this confirms that someone got their logic mixed up.
This phrase highlights the confusion. If you learned SQL before Go, you want to rollback only on error.
Every time I write `defer tx.Rollback()`, I cringe and have to remind myself that yes, it's actually ok to call a method called `Rollback` after succesfully writing data.
BEGIN;
INSERT INTO ...
COMMIT;
ROLLBACK;
Perhaps you mean learned SQL in the context of languages that consider a failed rollback an exception? In that case one needs to be careful to not rollback, else be stricken to handling the exception, which programmers seem to hate doing.
No: https://github.com/golang/go/blob/beaf7f3282c2548267d3c89441...
BTW I checked and it's not an error in Postgres, only a warning. Still not something I would want in my database logs for the happy path.
But it's a good sanity check/safety measure to call it anyway incase you made a mistake elsewhere. An errant rollback is more likely to be caught in testing than a dangling transaction.
Unit tests are good, yes. Monitoring is also good. But just taking 30 seconds to do some manual testing will catch a LOT of unexpected behavior.
1. Get it working: write the code for the desired behavior, not worrying about making it beautiful, testable, whatever.
2. Get it working well: manually testing and finding edge cases, refactoring to get it testable and writing tests to solidify behavior.
3. Get it working fast: optimizing it to be as fast as I need it to be (can sometimes skip this step). No tests should change here, but only new tests.