Hacker News Reader

Comment by rbanffy | Hacker News Reader

> But I’m still going to be the Marie Kondo of IT and ask if that specific data point brings you joy.

There seems to be a strong "instrument everything" culture that, I think, misses the point. You want simple metrics (machine and service) for everything, but if your service gets an error every million requests or so, it might be overkill to trace every request. And, for the errors, you usually get a nice stack dump telling you where everything went wrong (and giving you a good idea of what was wrong).

At that point - and only at that point, I'd say it's worth to TEMPORARILY add increased logging and tracing. And yes, it's OK to add those and redeploy TO PRODUCTION.

View on HN

> There seems to be a strong "instrument everything" culture

Metrics are the easiest way to simply expose your application internal state and then, as a maintainer of that service, you’re in nirvana. And even if you don’t go that far you’re likely to be an engineer writing code and when it comes time to add some metrics why wouldn’t you add more rather than less, and once you have all of them why not adding all possible labels? And in the meantime your Prometheus server is in a crash loop because it run if of RAM, but that’s not a problem visible to you. Unfortunately there’s a big gap in understanding between a code editor writing instrumentation code and the effect in resource usage on the other end of your observability pipeline.

I can only say, I tried to add massive amounts of data points to a fleet of battery systems once; 750 cells per system, 8 metrics per cell, one cell every 20 ms. It became megabits per second, so we only enabled it when engaging the batteries. But the data was worth it, because we could do data modelling on live events in retrospect when we were initially too busy fixing things. Observability is a super power.

This right here! Don't be afraid to over instrument. You can always down Rez or even just basic statistical sampling before you actually commit your measurements to Time series database.

As annoying as that may sound, it's a hell of a lot harder to go back in time to observe that bizarre intermittent issue...

This is very true for systems and components you don’t thoroughly understand.

By all means instrument batteries, motors, gearboxes, positioning systems, turbines, and so on. If your system is mostly a CRUD or some business logic automation, then the need to overinstrumenting it is much smaller. Databases are well understood after all.

On paper this looks smart, but when you hit a but that triggers under very specific conditions (weird bugs happen more often as you scale), you are gonna wish you had tracing for that.

The ideal setup is that you trace as much for some given time frame, if your stack supports compression and tiered storage it becomes cheap er

> you are gonna wish you had tracing for that

It’s never too late to add data collection and redeploy the affected service. If the bug is bad, you’ll be able to catch it then.

> but do you really need a Cadillac Escalade (New Relic/Datadog/etc) to go to the grocery store

Depends if your objective is to go to the grocery store or merely showing off going to the grocery store.

During the ZIRP era there was a financial incentive for everyone to over-engineer things to justify VC funding rounds and appear "cool". Business profitability/cost-efficiency was never a concern (a lot of those business were never viable and their only purpose was to grift VC money and enjoy the "startup founder" lifestyle).

Now ZIRP is over, but the people who started their career back then are still here and a lot of them still didn't get the memo.

> During the ZIRP era there was a financial incentive for everyone to over-engineer things to justify VC funding rounds and appear "cool".

Yep, and what’s worse is that…

> Now ZIRP is over, but the people who started their career back then are still here and a lot of them still didn't get the memo.

…folks let go from BigTech are filtering into smaller orgs, and the copy-pasters and “startup lyfers” are bringing this attitude with them. I guess I got lucky enough to start my interest in tech before the dotcom crash, my career just before the 2008 crash, and finished my BigTech tenure just after COVID (and before the likely AI crash), and thus am always weighing the costs versus the benefits and trying to be objective.

> folks let go from BigTech are filtering into smaller orgs, and the copy-pasters and “startup lyfers” are bringing this attitude with them

Problem is, not all of them are even doing this intentionally. A lot actually started their career during that clown show, so for them this is normal and they don't know any other way.

Yeah, very true, and those of us with more life and career experience (it me) have a societal contract of sorts to teach and lead them out of bad habits or design choices. If we don’t show them a better path forward, they’ll have to suffer to seek it out just like we had to.

Reply on news.ycombinator.com