Hacker News Reader

Comment by Fatnino | Hacker News Reader

I was at a place where we had "worker" machines that would handle incoming data with fluctuating volume. If the queues got too long we would automatically spin up new worker instances and when it came time to spin down we would kill the older ones first.

You can probably see where this is going. The workers had some problem where they would bog down if left running too long. Causing the queues to back up and indirectly causing themselves to eventually be culled.

Never did figure out why they would bog down. We just ran herky jerky like this for a few years till I left. Might still be doing it for all I know.

View on HN

> The workers had some problem where they would bog down if left running too long.

So you just automatically replace the instances after a certain amount of runtime and your problem is gone.

Yeah, fixing a problem without understanding it has some disadvantages. It works sometimes, but the "with understanding" strategy works much more often.

Is this really a prevailing attitude now? Who cares what happened, as long as we can paper over it with some other maneuver/resources? For me it's both intellectually rewarding and skill-building to figure out what caused the problem in the first place.

I mean, I hear plenty of managers with this attitude. But I really expect better on a forum called hacker news.

For me, there’s a certain threshold involved.

If it happens extremely rarely (like, once every 6 months) or it’s super transient and low impact, we kick it and move on.

If it starts happening a 3rd or 4th time, or the severity increases we start to dig in and actually fix it.

So we’re not giving up, and losing all diagnosis/bugfixing ability, just setting a threshold. There’ll always be issues, some of them will always be mystery issues, so you can’t solve everything, so you’ve got to triage appropriately.

I said nothing about not understanding the issue. Even with understanding just “turning it on and off again” might be the better solution at the moment. Because going for the “real” solution means making a trade-off somewhere else.

Plus just because the only visible symptom of the bug is a perfomance issue right now doesn't mean that there won't also be other consequences. If something is behaving contrary to expectations you should always figure out why.

Reply on news.ycombinator.com