Hacker News Reader

Comment by icambron | Hacker News Reader

Putting aside questions of monorepos, I am always astounded by how much big companies invest in their build/change management/review systems. Shipping becomes a whole product in itself. This appears to be a serious piece of engineering, but it’s directed at Uber’s tooling, not Uber’s business problems or even Uber’s product.

I’m sure it’s fascinating for people who are into that, and I’m not saying there isn’t a substantive problem to solve here, but I’d personally never go anywhere near a problem like this. It’s too many steps removed from the value. I’d keep thinking “we are down a very deep rabbit hole. A side quest off a side quest. Should we be doing this?”

But I’ve always worked at smaller places, where things like toolchains and sales ops and marketing tech don’t take on huge lives of their own.

View on HN

I've repeatedly seen both high quality and low quality tooling act as massive force multipliers even for small teams; the question is just which direction you want to multiply. They say an army marches on its stomach—well, a tech effort marches on its tools. I interned at a trading company with, at the time, maybe 100 developers that built almost everything in-house and yet was quite a bit more successful and productive than much larger teams I've worked on since. The larger teams had incentives and structure to "focus" on what's "important" for very narrow notions of important... and so they ended up with lower-quality internal systems, a bunch of day-to-day drag on development and, both technically and organizationally, far less adaptability. (Turns out, one of the major advantages of investing in your own tools is that it builds up deep institutional knowledge about tooling that is hard-to-impossible to maintain otherwise.)

Only valuing easily-measurable work is, frankly, a modern organizational disease. It's like searching for the keys under the streetlight—and why? As a substitute for human judgement or resolving disagreements? As a way to make work more legible for executives? It leads to unforced errors and systemic problems, but we can't seem to do anything about it.

> Only valuing easily-measurable work is ... a modern organizational disease

Recently I showed a development team how to use a debug tool that can automatically collect debug snapshots from production that can be opened with one click in the IDE. It’ll jump to the line of code and show the state of everything — not just the stack but the heap too.

My demo is to pick a random crash and diagnose the root cause while talking. The mean time to resolution is low single digit minutes.

I showed this to an entire team, one person at a time, solving bugs as I went. I showed the junior devs, senior devs, and their manager.

No interest. None. Just… silence.

The tools are amazing, but the lack of motivation from the typical developers for learning to use them is even more amazing.

Pointer to the tool? It sounds awesome.

Seconded!

Azure App Insights can do Debug Snapshot collection, but it’s not the only such tool. It just makes the workflow quick and simple.

Ref: https://docs.microsoft.com/en-us/azure/azure-monitor/snapsho...

It works even better with DevOps source indexing added to the build pipeline:

https://docs.microsoft.com/en-us/azure/devops/pipelines/task...

A vaguely similar feature is Azure App Service memory leak diagnostic tools which take memory dumps on certain triggers or at intervals.

You can open these in Visual Studio and it’ll show the heap deltas over time.

Playing devils advocate here. It could just be that they already have the stack trace in logs somewhere, so how does this help you any faster? Bulk of the time to resolution will be figuring out what to do anyways.

These were all bugs that were nearly impossible to solve with just a stack trace. For example, there was a "format" exception on a web page with hundreds of uses of string formatting (to generate a report). Another example was a function call complaining about a null argument. Which instance? The page had 40+ calls to the same function, each with 6 arguments.

Most of these couldn't be reproduced either. As in, you'd get a crash once a day in a page that would otherwise work successfully thousands of times.

How would you fix a problem where there's a stack trace only from a release? The scenario is: you can't reproduce the errors, you don't get line numbers, you don't even get function argument values.

I could solve these in minutes using this tool. Could you match that without such tooling?

I’m guessing the organization incentives are against investigating/fixing random crashes. It might be unscheduled, or seen as “test” or “qa” or “ops” work. Try working with management to set up a way to reward the behavior you want to encourage.

It is really just a matter of scale. At big companies it is easier to see the value of having people on these problems, so the teams have a clear reason to exist.

As for your "side quest off a side quest" observation, I think that is a matter of perspective, about what personally motivates you, and about how work is organized in society. I find that working at a small company makes it easier to have a feeling of focus and purpose -- everybody is focused on one thing. But, I find that working at a large (profitable, well run) company makes it easier to feel like you can actually move the needle and have the resources do large things (and yes, this includes a connection between something like a 'build tools' team and being able to ship a quality product more quickly).

It is easy to miss that every race car needs a pit crew, every pit crew needs equipment and parts, getting parts requires suppliers, parts need to be built, tested, etc., all those people need to eat, all that food needs to be grown, and so on. Organizing all of that "work" into small single-purpose companies, that buy each others' stuff, is one way to go. Another way to go is to organize it into larger companies with sub-teams/groups/departments. So a company's larger mission might be "win races" but, if it is large enough, smaller support efforts in support of the larger goal can make a lot of sense.

Tech companies pay for few things more than for developer time. In a small company, spending one day to speed up a 30 minute build of a codebase used by 10 developers by 1 minute is not going to unlock much developer time, but imagine you multiply it by a thousand or ten thousand developers. Suddenly you save 10 thousand minutes per build, or 166 hours. Imagine the average dev runs a build three times per day, your 1 minute saving is now saving 498 hours of developer time per day, or 62 8 hour work work days per day (and there is the saying that in FANG it's way less than 8 hour work days). These are continuous cost savings, so the value your 1 day investment created is equivalent to 62 fully paid developers. Isn't that amazing?

I think tech companies should spend a lot more on speeding up their stack. I would love to do something like this (improving developer workflows in a giant organization) as my day job.

The counter argument here is that that 1 minute difference doesn’t correspond to a minute of extra work in the way that ‘62 days of work saved’ conveys. But the counter-counter argument is that sufficiently fast/good tooling is qualitatively different and increases productivity by enabling new workflows (eg build it vs think carefully about whether it will compile right because builds are slow; break up changes into smaller chunks vs write big hard-to-review changes because CI is slow) such that speeding up tools or maintaining performance as the code base grows are valuable.

I remember reading once that the larger your engineering org, the larger the percentage of your engineers should work on tooling. As the complexity of your system grows, the ability to know if your change causes an ill effect becomes more difficult. Large systems have large and complex tooling because it becomes a necessity for survival.

Do you remember where you read that? I'd love to read that article/book.

Investing in that is vital if you want to grow as an engineering company. I work at a company with 2000 engineers, and the infrastructure and our build/change management/review system is horrible. It takes me all day to check something in, the integration tests are flaky and waste a lot of time tracking down errors, etc. Investing in this would collectively save so much time, but the people in charge think we can't afford the investment so everyone suffers.

Sounds more like an organizational issue than a technical one.

I’d say that this is just internal platform / infrastructure support, aimed at making the whole organisation be more effective. It’s absolutely adding a lot of value to the organisation, it’s just not visible from the outside.

But saying that “it’s too many steps removed from the value” appears to be a misrepresentation of the value it adds.

The amount of people dedicated to such projects is often a very small overall percentage of the engineering force, and they act as a leverage point. Invest in X staff, make all eng at the company 1.X more effective, therefore giving you a large return on investment.

> It’s too many steps removed from the value.

Don't be worried about being too many steps removed from the value. With that logic, CEOs, CTOs, etc are also too many steps removed from the value, yet they are valued very highly. Look at such infra work as multiplying the value of the entire company and in a way, being part of a "CEO team".

> It’s too many steps removed from the value.

And this kind of reasoning is why tooling in most shops absolutely sucks

YAGNI is always a valid strategy but what you are and are not going to need will change with scale. An inconsequential case can turn out to be something that impacts several teams.

At scale, rabbit holes can be quite relevant. At scale, you can no longer trust that your personal view of the org is complete enough.

There's still plenty of waste but just because you don't need something doesn't mean it isn't in another team's hot loop. Best to come in with trust first and incredulity second.

Do you also find the existence of accounting or janitorial departments in companies whose core business is neither accounting nor custodial services strange?

I would find it pretty strange if the janitorial department were smelting iron ore to make their own trash bins.

And yet accountants mostly adhere to GAAP no matter the industry, and the janitors are using the same tools their counterparts are.

Per DORA and the information in Accelerate, shipping code frequently and reliably contributes directly to good outcomes. It seems like a good investment.

I don't think the commenter is challenging this idea. Many companies do this with "off the shelf" CI/CD solutions without committing large amounts of money into bespoke tooling.

From my experience, the off the shelf tooling works poorly with monorepos. Everything is still optimised for the polyrepo microservice craze of 2017.

There are tools out there, but they're not free and they involve some level of vendor lock in. I've worked on some monorepo build scripting myself.

In my opinion, git in general works poorly as a monorepo.

My ideal VC solution would have the branch control of git with the checkout model and hook configuration of SVN.

I am not familiar enough with the version control ecosystem to know if there’s another product that does it, but I doubt it since I kind of suspect it would violate the CAP theorem.

Personally, Uber doesn't come off to me as an engineering culture that values simplicity.

And in fact, in this very article, the last paragraph actually states that they are aiming to increase the complexity of the change validation process?!?

> Additionally, it helped our team focus on increasing the complexity and features of our change validation process.

Personally, I haven't been super impressed with any of the engineering projects I've heard come out of Uber, but what they're doing obviously works to some degree so I can't be one to judge.

Deck.gl is a fine map visualisation library.

> It’s too many steps removed from the value

Uber is, partly, a software business. They need to ship software. They get direct value from shipping good software efficiently.

People drive cars to work, school, etc.

Look at the infrastructure required for that and what we spend on it. It’s massive.

Best practices in software tend to not be confined to a specific domain, so the amount of money spent in the industry being targeted really isn't relevant to need for custom CI vis-a-vis off-the-shelf solutions.

Wait til the regulators come for your industry. Modern IT has to prove to Auditors amongst other things that no one rogue person can contribute malicious code. I've worked in companies that let people do their own thing but then let them deal with Audit. A much better company had a sophisticated custom SDLC process that meant Audit just checked you used the process.

Reply on news.ycombinator.com