Hacker News Reader

There was a bug in an embedded system I worked on that several engineers, software as well as hardware, spent months troubleshooting.

I eventually got called in and fixed it with a while loop and one function call inside of that loop. 4 lines in total, counting the brackets.

A very trivial change if you didn’t know why it was there, what impact it had on the system or how much it would have continued to cost in engineering hours if I hadn’t figured out what went wrong, why it went wrong and how to fix it.

I’ve spent a good chunk of my 15 years as a software developer as a pure “bug killer” and then you don’t really get to write that many lines of code, but the impact per line is big.

Agree fully. Yours isn't the case I'm talking about. I'm talking about the people that aren't hunting big bugs, aren't serving as advisors or architects, aren't deleting unused code, and aren't implementing a significant volume of features or functionality, but who instead trickle out a small amount of simple code month after month. I claim that when inspecting commit histories, a very low LOC or very low commit-frequency is something a manager should look into.

All you have to do is read commit messages.

Reading commit messages is a poor substitute for reading code.

The main issue is the majority of managers arent especially qualified to make a judgement of the code itself which makes the metric spurious at best. Lines changed is at least a better version than lines added. But it is fairly devoid of meaning removing ten lines can be extremely hard and impactful compared to adding 1000 lines of deeply nested ifs written poorly. Lines changed multiplied by number of files touched is possibly better too.

Any generic measurement of productivity is a sign that the manager is incompetent.

If you're a manager and can't tell what your employees are doing, you're a terrible manager. Same reason they hate people working from home, they can't actually tell if someone's a good employee, they have no clue whether anyone is good at their job because they don't understand their job

My entire premise is:

1) A good manager needs to look at code to see what employees are actually doing. They can't just rely on the employees' verbal description.

2) One (correlated though not guaranteed) indicator that an engineer is struggling is when they are producing much less code than their peers or compared to any natural expectation of the role. Yes of course it's 100% that some bugs are super tricky and take a long time to find the magical one-liner fix. But statistically those are not common.

This is premised also on my belief that every engineer manager should be a very strong engineer themselves. This is common at most of the big tech companies.

Here's some counterexamples to that idea:

- a bugfix which takes ages to find and ends up being one or two lines of locally trivial code

- a senior who spends most of their day unblocking junior devs and keeping the team on track: 0 lines of code

- overzealous code formatters: lots of lines of code

- a junior writing an overly complicated mess of a solution to a problem that could be solved much more simply: ungodly amounts of code

- a developer talking to the client directly and finding out that problem that they're trying to solve is already solved by the product in a way that the client didn't think of, rather than immediately wasting company resources on a feature that's ultimately unnecessary (0 lines of code over lots of lines of code)

There is 0 correlation, none whatsoever, between lines of code committed by a dev and their value they provide to the business.

> There is 0 correlation, none whatsoever

This is quite unlikely to be true. No correlation at all?

Especially if we shift to ΔSLOC rather than +SLOC, I don't believe that for a second.

You've made a good case that it's a bad proxy for productivity. It is. But there's no need to over-egg the pudding.

I think it'd help to go over what the word correlation means, and then go over the examples again, and then look at the claim that there's a correlation between the value a dev provides and lines of code.

> a bugfix which takes ages to find and ends up being one or two lines of locally trivial code

I once had a junior engineer who took maybe 4-6 weeks to fix an intermittent bug in dynamic lib loading. When he finally fixed it, the fix was <5 lines of code. But it was obvious that (1) the bug was insanely deep and hard to find, and (2) he did amazing kick-ass debugging to uncover it deep in Linux code. It was great work, and he was one of our top rising talents. But that's not the kind of cases that I'm talking about.

No, it's instead the engineer who gives the appearance that they're always doing tough work, but then the commit history shows that they're simply not. The person who says they're implementing a new data structure in the code, but really then take a month to replace a list with a dict in python, or whatever. This happens.

> a senior who spends most of their day unblocking junior devs and keeping the team on track: 0 lines of code

I'm not talking about them at all. I'm talking about people who aren't doing mentoring, advising, architecture, forward-looking design docs, etc.

> overzealous code formatters: lots of lines of code. A junior writing an overly complicated mess of a solution to a problem that could be solved much more simply: ungodly amounts of code

Yup, this happens. This is also something managers will learn by looking at code.

- a developer talking to the client directly and finding out that problem that they're trying to solve is already solved by the product in a way that the client didn't think of, rather than immediately wasting company resources on a feature that's ultimately unnecessary (0 lines of code over lots of lines of code)

Absolutely. The best way to get more work done is to figure out what work you don't have to do.

But if you have an engineer on your team make $200+K/year and they're not writing much code, nor writing design docs, nor advising other engineers, nor guiding the technical strategy, etc... if all the engineer does all day is "not wasting company resources" by doing very little whatsoever, you have a problem on your hands.

All you're doing is affirming the antecedent.

> we can't pretend that the amount of code that a developer writes is devoid of any meaning

The scalar of «amount of code that a developer writes» is «devoid of any meaning» outside frameworks of streamlining of such code (which would themselves not be a good idea, as you would in fact actually have classes of code, to be judged differently). The submitted specifies it was a case of a simple scalar field in a form.

Your argument is of course valid, but your conclusion disregards it. Outside a defined framework of quality adherence, the «amount of code» is «a random number».

The interesting part is all in the productivity (bulk of quality). That is the area that should be investigated - in both discussion and assessment.

And clearly, "amount of code produced" is a bad incentive, against quality. Aaaalways take care and beware of incentives.

LOC as a datapoint provide no value that a manager couldn’t get quicker and with less ambiguity via other sources. For example, identifying engineers struggling for a month on a topic, as you described, can be uncovered way earlier by replacing a reporting-Like „standup“ with collaborative structures that help the team, and the manager, understand progress and potential blockers. LOC are lagging way behind on issues like these and are devoid of any meaning in isolation. I know your counter argument here is „one data point among many“, but mine is: if your other data points are solid, LOC do not add any value.

With the exception of engineers whose role is strictly advisory, architectural, etc, for most engineers the direct literal expression of their work is the code that they commit. Not the design document, not the ideas in their head, but the code in the repo. Managers should be looking at this code in their evaluation of everyone's work. Someone with an unusually-small amount of code shouldn't be "punished" but it is something that should be noticed and examined.

That is the equivalent of saying that the direct expression of a musician‘s work are the sounds they produce and therefore they should be heard in isolation. But as much as an orchestra produces symphonies as a group, development teams produce software as a group. This output matters and should be looked at, the individual code contributions to that output matter less and looking at them in isolation doesn’t provide insights that looking at the team‘s output in terms of working software, bugs, incidents and more wouldn’t.

A manager looking at the individual LOC contributions is looking into the wrong direction.

Do you really expect most managers, if any, would do this?

No, probably not. And most don't try. My point is that --with proper context and thoughtful analysis-- it is data that can be useful.

At the end of the day nothing about the number of commits was used in your thoughtful analysis, it was only the contents of the commits which you attribute any value.

Yes and no. Certainly I know lots of great developers who prefer to commit work frequently and incrementally, others who do slower bigger changes. But I don't think I know any engineers who can create a substantial amount of software with only infrequently and tiny changes.

I encourage you to look at the code produced by whoever you think is the top engineer in your team or company (and I'm talking only about the engineers whose job is still primarily hands-on coding, versus advising/mentoring/architecting etc). Now look at the code by that one person who you know isn't pulling their weight and who you think isn't getting much done. I'll wager that your top engineer is pumping out a ton of strong code, and the other one is a trickle.

> Now look at the code by that one person who you know isn't pulling their weight and who you think isn't getting much done.

This is bad, but it’s not the worst type of engineer. The worst engineers often have huge deltas and can kill companies singlehandedly.

It is clear that the weight attached to LOC will decline as management acquires more directly relevant information. The difficult issue is whether there is a zero lower bound.

Imagine buying in software, against a specification. A fully detailed specification with tests both for logical correctness and performance. If there is a choice of vendors, at similar prices, one naturally chooses the package with fewer lines of code. The weight attached to LOC is negative.

The anecdote is interesting because it invites the following speculation. Question: How many extra data points, beyond LOC, does one need to manage a software project? Answer: Enough that the weight attached to LOC, given optimal use of the other information available, is negative.

Sorry but your argument makes no sense. The 10LOC/day guy is not to be singled out from the 1000LOC/day people because 1000 LOC could be just as trivial; 1LOC could have taken hours to discover.

Exactly! I've myself spent days investigating an issue only happening on a specific Android device (but used by an important client) only to find that the solution was something along the lines of using "opacity: 0.1" instead of 0.

That bug was left unfixed for months but now it's solved. Should I have received a talk because of that?

(Of course the commit would have had at least 10 LOC more, explaining the reasons and providing references to future devs/self)

> Should I have received a talk because of that?

Of course not.

But, if your commit history for last 6 months is just one or two of those one-line changes a month, that might hint that there’s a problem.

That makes your point clear, but then what you are looking for is not a scalar value (that may be for instance be subject to a proportional judgement) but a fuzzy threshold.

Might that hint that the problem that is there is that the metric is bad?

So if you managed a team, and when you look at commit histories you notice that one person has <1 commit a month and you look at the commits and they're trivial... you would disregard any concerns?

I would wonder how it got to the second month, certainly - getting to the point where this kind of post-hoc analysis can actually be done is a serious failing of management.

The answer is still "it depends", I would imagine that issues would arise well before you had to look at their commit history.

I believe that if you are a good manager, you would just need to look at what tickets get done during the sprint and that's it. If you complete your tickets and pass QA/UAT in 10 LOC, so be it. There is no need to micromanage.

This is exactly why KLoC is not a good measurement. If you know about this, you will make sure you match everyone else by embellishing your line count with comments, copy and rename functions or extra deep nesting. The manager will still not know anything without looking at everyones code.

A good manager should look at everyone's code.

A good manager should recognize that if a developer does nothing but annotate code with comments, and that's not what they're expected to be doing, there's a problem.

Ofcourse, but if the manager is using the KLoC to decide who to look at, that won't happen.

I mean, seems highly role-dependent. If, in practice, your role is to investigate and fix very difficult problems, that’s precisely the sort of commit history one might expect.

Right. One data point among many. And managers should be looking at everyone’s code, not just the low LOC ones.

But there are people (I’ve seen this frequently) who constantly represent their work as having been “much more difficult than expected” but then you discover that they actually are struggling with what should be easy tasks, and looking at code (complexity and volume) is a data point. For me, more often than not, it serves as a confirmation of a problem I’ve started to suspect.

confirmation bias indeed works that way more often than not: it confirms what you already believe

I'm confused which part of this is so off-putting.

Joe is always the one in every standup saying the thing he's working isn't done yet because he's wrestling "one last tough bug." As a manager, you wonder if the technical problems are really that tough or if Joe is just struggling. Let's look at the commit history... yeah, something is off here, Joe has 1/10th the commits of his peers and they really don't seem more complex in any way, but look rather trivial. Time to talk to Joe and look at these commits together and see what's going on.

Is that really such a troubling proposition?

So troubling that you can't propose it cleanly: if you've already defined that "they really don't seem more complex in any way, but look rather trivial", why would you need to count? (another troubling sign is going for the order of magnitude assessments, and another minor point: you've mistakenly replaced volume of code with # of commits)

So why wasn’t the response after day 1 of the “tough bug” something like “hey, why don’t you pair with X to get to the bottom of it?”?

If you declare by fiat that every confirmation is confirmation bias, you've made knowledge impossible.

What happens if you declare by fiat that every mention of confirmation bias is declaring by fiat that eveyr confirmation is biased?

I wonder what the grandparent would think about a few LOC/month, there were a few rather gnarly performance issues in the codebase at my work for which the fix was usually a one or two liner, but that took an inordinate amount of research to even know where to start looking.

I guess I should be grateful for having a workplace that has sufficient understanding for the effort it takes to debug such issues.

Yes, but I'm guessing your commit history over the last year is not just 1/2-liners once a month, right?

> Why have they reported at every standup the last six months

Every what?

Standup. Daily meetings where teams talk about their blockers and so forth.

If six months of daily meetings did not lead anywhere then cancel those meetings.

Many teams have short daily gatherings — especially now that so many teams are fully distributed, it’s good for everyone to come together like that.

But while every team member talks about the difficult bugs they’re facing, sometimes the actual code tells a different story and can reveal that a team member is struggling to get working code together in any reasonable amount of time.

It sounds like you could just read their code and the meeting was unnecessary. Linux kernel developers are distributed but don't do daily status update meetings.

heh, I note your recent comment history is you being salty about everything you deem "cargo cult agile", and people patiently responding to that.

Yes, having a chat between the team for fifteen minutes every day is in fact quite useful, believe it or not.

We're not talking about hour long waterfall-y management progress review meetings here.

Daily status updates are not useful. Linux kernel developers don't do them. I trust that nothing is stopping your engineers from having a chat and that their manager doesn't need to schedule their chats for them.

You speak in absolutes. What works for me, works for me :)

Cargo-culting someone else's process is a bad way to develop good process for your specific team and its needs.

You could use the same logic to claim that "issues and PRs are not useful" because the Linux devs use a mailing list and patches. I think that's obviously absurd, all that this example shows is that a mailing list and patches can be a useful way to develop programs. It says nothing about alternatives at all.

What specific needs are met by daily status updates? Professionals outside of software don't do that. The best software projects don't do that. It's very popular in CRUD projects led by non-technical middle management with trust issues.

> Professionals outside of software don't do that.

How do you know this? Have you done an exhaustive search?

For one example, are you quite certain that no part of any military ever does this?

What about every factory?

You are not answering my question. No, you ask your acquaintances who work in law, finance or medicine. Neither does academia have any "stand up". What specific needs do CRUD software projects have that are met by infantilizing daily status reports? What needs for daily status reports do these projects have that the Linux kernel doesn't?

> What specific needs are met by daily status updates?

You are assuming (1) that standups are status updates, and (2) that they are scheduled by manager, and (3) for benefit of the manager.

In the last several teams I’ve managed, it’s the crew that has encouraged meeting daily, and in the standups they are mainly talking to each other. It’s where they cover hot topics and bugs for the day quickly, with the fluidity of spoken conversation instead of Slack, and also there’s a non-zero amount of friendly casual conversation.

Don’t force teams to have standups, but also don’t assume a team won’t like it.

They just recommended cross-checking status updates in "stand up" meetings with commit history to catch engineers underperforming. That is (1) (2) and (3).

A manager has a duty to "catch" (as you put it) underperformers, to help them improve.

If you ever move to management, you'll see that as lovely as it would be if every hands-on developer was an amazing and super-productive engineer writing high-quality code and delivering tons of value , the reality is that most teams have one or more people that are struggling and aren't getting much done, but aren't forthcoming about it. They'll represent their work as very challenging (which it is, to them), meanwhile others on the team are objectively better and faster at figuring out solutions and not getting stuck, and they understand the codebase better, and the architecture, and so on. The manager has a duty to spot when this is happening, and engage with the engineer immediately to figure out what's going on and try to help them.

Engineer managers should themselves be strong engineers (and in every company I've worked at, this has been the case up the senior leadership chain), so that they can form a realistic, fair, and reasonable assessment of whether there's a discrepancy between how an engineer represents their work, versus what is shown in the actual commit history.

The engineer did not choose to represent themselves in weird daily show-and-tell meetings. The commit history is there regardless. You could just look at that.

> Professionals outside of software don't do that.

What, and I mean this in the nicest way possible, the everliving fuck are you talking about?

Professionals do this constantly. Go sit in an ER and watch the hand off between shifts. Go see any serious manufacturing facility and see them review daily everything that happened the previous day. Go see an effective sales org in action.

The scrumbags may have ruined software, but that certainly isn’t the world.

You make your not entirely unreasonable points sound like the output of a zealot.

"Stand up" meetings are not hand offs in the ER. Continuous integration with automated testing is.

Why are there no laymen agile coaches at law firms, and no "today I did this, today I did that" breakfast meetings in finance, or in CS academia? Other professionals don't accept this kind of infantilization.

oh no. you're responsible for a group of skilled and not so skilled engineers in a domain that you don't understand. all is not lost.

what is your only goal - to maximize your teams contribution to the goals of the company. full stop. its not to stack rank your employees unless it serves that greater goal.

but you don't understand the domain. oh no. all you can do is develop a human relationship with the team. listen to them. you can't judge the quality of their work, but they can.

don't ask them to rank each other (I've seen this), but just be attentive. after not to much time you can really start to understand how people are helping the effort, and how they are undermining it. in broad terms, without understanding the gory details.

this is the only thing that works. everything else is just obfuscation.

What's all this business about ranking that you're talking about? I didn't mention it. And asking engineers to rank each other? Seems you've picked some bad companies to work at.

You should consider working somewhere where management is deeply technical. At everywhere I've worked, the entire eng-management chain (up to CEO) fully understood the domain (as you put it), the state of the art, the key system architectures, and so on. The idea that an eng manager's role should be limited to "developing human relationships and being attentive", would simply not fly.

Eng managers need to be able to go into code, and they must be able to gauge the complexity of the work under them, or they'll be unable to spot the difference between someone aggressively chasing a nasty heisenbug for weeks, versus someone struggling for weeks to put together a basic block of working code.