Hacker News Reader

Comment by taeric | Hacker News Reader

I can't really compare to Meta, but I will confess that "industry standard" tooling has been very disappointing for me. Trying to get a build setup that automatically pulls in dependency changes once a week is something I'm still not entirely sure how to do in a way that everyone agrees with. Seems most tools bake that into a code commit with a lock file nowadays. But even that is amusingly recent. Years ago, you had to mirror any repository system locally and learn how to set that up. Probably getting it wrong in the process.

Mentioning splunk, I'm assuming you are using more paid industry tools. I suspect that opens things up a bit more, but realize that most of the industry doesn't use those due to prohibitive pricing. And I don't even know what the appropriate paid tool for building would be.

We seem remarkably primed to just hate whatever tooling we have at our disposal in ways that baffles me.

View on HN

I remember when Cisco mentioned potential acquisition of Splunk for $20B, the jokes about whether it was to buy the company or to pay the running Splunk bill wrote themselves.

I know splunk is expensive, especially if you don't have somebody who is actively monitoring your spend, but sumo is a pretty good alternative. It's actually something I look for in the interview.

Regardless, many companies underspend by an order of magnitude on dev tools. If a tool makes you 1% more efficient, it's worth them putting 1% of your annual salary into it (multiply this by engineers at the company who need the tool). Soon you'll see that a tool costing hundreds of thousands to license a year is often a steal.

So you may say "oh splunk is overkill and unfair expectation" but in their CI/CD system, if your job failed (frequent occurrence) and you wanted to search for something in there you'd manually DOWNLOAD a 300 meg file to your machine and grep it for errors.

Basically a fang engineer's time costs $3 a minute minimum, so if a query could save 1 minute then the break-even cost is paying up to $3 to run that query.

Even in a fang, most money spent on log storage is fully wasted. The article yesterday about monitoring being a pain was spot on.

More amusing is when you see someone build a giant Elastic Search pipeline to a Kibana dashboard so that they can get what would have been table stakes metrics if they had used the normal service templates. Without a ridiculously large bill.

Or folks that think they can warp around the high cardinality traps of making a metric out of everything. Assuming if you can make a system that works for the testing environment, of course it will work when you open the floodgates.

Seen people argue that sampling shouldn't be used, "because you could miss data?" Reservoir sampling is a thing, for very real reasons.

At any rate, I don't mean to just yell about splunk, from all I've heard it is nice. I am annoyed that folks seem to ignore the OLAP and OLTP divide, such that they think your metrics system should somehow be optimized for both. At the same time.

Pulling back to builds, though. What, exactly, is the gold standard in industry? I have yet to see it. Python builds, in particular, strike me as not good.

> If a tool makes you 1% more efficient, it's worth them putting 1% of your annual salary into it

Hear me, nameless internet stranger, that you might learn from my mistakes: This is not true. Efficiency is only worth money if it increases profits - concretely this means your efficiency gain must result in the following:

1. Delivering features faster

2. Delivering features with meaningfully higher quality

3. Delivering the same features with lower headcount costs

(2) is hard to measure, so you can generally only sell on (1) and (3). During the 80s the business world learned the hard way you can throw away a lot of money on useless efficiency. If that interests you I recommend "The Goal" by Goldratt for a fictionalized account of those learnings.

* There's also a subtler point that corporate finances may mean that even if efficiency is perfectly captured, a 1% efficiency increase may only be worth 0.1% of your salary.

100% this... profit, or it's not worth it

Splunk can save money, as long as you’re intelligent about processing data. I can think of dozens of security and operational incidents where splunk saved the day.

I’ve been through a few savings exercises where everything is gonna go open source dashboards etc. Once the guy who understands how it all works disappears, it migrates back to splunk.

One security incident response will pay for the entire splunk infrastructure.

This isn't a great argument... sure, Splunk "saved the day" because it caught that really bad security thing... but what other tools, at half the cost, could have also "saved the day"? If most of them could have, then it's not really worth the price... right?

Sure. I can save a ton of money by fixing my own car too - if I have the ability to execute.

If you can, that seriously awesome. The point is, many can’t.

FB's log shipping and parsing is much better than "industry" however we ship billions of logs to get a metric, because the internal graphite service isn't easy to use. and unidash really sucks compared to grafana for non-SQL based metric discovery and prototyping.

Reply on news.ycombinator.com