Hacker News Reader

Comment by lmm | Hacker News Reader

Just another reminder that it's never worth bypassing the normal deployment process. Every year or two I learn a similar lesson myself: it's so tempting, the fix is so small, it couldn't possibly break anything (heck, I once had a data-gathering script that made a bunch of read-only calls to our system's API cause a live issue). Just say no.

View on HN

> heck, I once had a data-gathering script that made a bunch of read-only calls to our system's API cause a live issue

I don't think you can just throw that out here with no story. Well...I guess you can, it just makes me sad.

The script was a permission checker; roughly, for each client check whether they had any object types they didn't have permission for, plus a bit more logic for things like "they have permission A which is a superset of B but don't have permission B", and output a report. There was a bug in the DB cache used by our security service (I think caused by a broken clone() implementation that shallow-copied when it should have deep-copied, but this was a while ago now) that meant that if permissions for two different clients were requested in quick succession, both would end up with the intersection of their permissions - which were then cached in the main application (I guess there must have been some logic like the API always triggered a refetch to ensure it returned the canonical value, whereas for the rest of the application performance was more important?). In theory it could've been happening already, but I guess we never had two clients loading the same webpage close enough to the same time (and they would have to both have had the local cache expire - plus almost all of our live clients had very similar permissions, whereas we had some "test" clients with no permissions at all), whereas of course my script just called as fast as possible.

It was fairly easy to track down the bug once we passed the initial disbelief stage, and I don't think many clients noticed. But it could've been a lot worse - I'd been so close to running this last-thing on a Friday before I left.

And I guess that meant that the clients would find themselves unable to access things since the permissions were cached, so you were actually seeing issues outside of your script results. Yea I can see how that would be truly baffling, and provides good motivation for not modifying interaction with production on a Friday evening. Thanks :)

Reminds me of a time a coworker at a previous job almost brought down a production server by opening a file in vi.

That file was a customer log that had grown to 100+gb in size due to an error that she was debugging. She failed to check the log size, instead assuming that it was a small file left over after that night's log rotation. When vi tried to load the file to memory, it almost crashed the box before we could kill it (we still got calls about degraded performance though).

I've made this mistake, only I was too naïve at the time to realise my mistake, thinking that vim, because of its reputation, could surely handle it. A server reboot later and I was enlightened.

Hmmm. I dunno.

Why wouldn't currency by handled by the type system? You could still have an overrun. But it'd be handled more appropriately.

Long ago, I wrote a budgeting / estimating tool. Costs were represented with binary coded decimals (BCDs). Not floating point numbers. Just like an accounting system.

Competing products could have weird roundoff errors. Not mine.

For accounting, why not just use scaled integers? Isn't there a limit to many many digits you want after the period? BTW, I think BCD is outmoded. Seems it's getting replace by DPD which has far less overhead (http://en.wikipedia.org/wiki/Binary-coded_decimal#Higher-den...).

To be fair, I am not sure if anyone would try posting 6 billion gold onto a PTR equivalent of the RMAH or whether the RMAH is even available on the PTR, which might explain why it wasn't tested.

If you set the limit to 10 billion of something, you need to test that 10 billion actually works.

A public test should not be your only line of testing, especially when said feature to be tested deals with real money.

Reply on news.ycombinator.com