Hacker News Reader

Comment by busterarm | Hacker News Reader

This often massively discounts the cost of reversing decisions. People often work to build things without any thought given to those who have to maintain it afterwards. Especially when it's not them.

I worked at a large, publicly-traded multinational where decades prior and they were still just a 4 man startup they decided the database server and all timestamps should be in the local timezone.

They are still using EST today even when they have global sharding of their customers/databases between US, EU, LATAM, SEA...

--- you're also assuming that the product roadmap will afford your engineers any time to build it the second, third, fourth, etc. time.

View on HN

There seems to be some general pattern here that you can find pretty often in "dev war stories" contexts:

(1) We're a small startup/new product team/etc, let's just build the MVP and keep everything simple!

(2) Now we're not small anymore and suddenly have all kinds of nonfunctional requirements we never imagined before! But our simple architecture from before is making everything a pain now!

The natural instinct is then to compromise on the "simpleness" of the first prototype and already try to anticipate all the scaling and nonfunctional requirements that might come later - but that rarely seems to work, as you can't really how (and if) the project will grow.

Seems to me, the real question here is why those teams are still using the "MVP" code even after being well inside the "scale up" phase. Shouldn't this be the point where you gradually migrate to a codebase that is more manageable at scale?

> The natural instinct is then to compromise on the "simpleness" of the first prototype and already try to anticipate all the scaling and nonfunctional requirements that might come later

This is a form of Second System Effect Brooks wrote about in 1975.

https://en.wikipedia.org/wiki/Second-system_effect?wprov=sft...

In addition to Second System Syndrome that has already been covered, I think there are several other things going on here that lead to the tug of war.

In another reply I already mentioned a flavor of Sunk Cost, which is resistance to any change to the code that 'made us successful' because they consider they have already won and taking it away now is some sort of revisionist history effort instead of just Progress. Hofstadter's Law, which tells you we still have plenty of time, despite three initiatives already having finished long past the point of maximum tolerable pain already.

But more damningly, I have worked with a lot of people who want to skip from Make it Work straight to Make it Fast without Making it Right in the middle. Those are the people who, among other things, reach for caching too early, and then declare the war for cost reductions over before they've even started, because there are no perf analysis tools that can see past and around the caching logic to find the issues that caching either didn't fix or made much much worse.

I'm not advocating for making hot decisions at every point along the way. I'm advocating for people having enough experience and foresight to understand which 20% of the code flow is Architecture from which is just feature factory work and making sure that 20% gets 80% of the design thinking and timely preventative maintenance.

On a couple projects I've just done all that myself. But it is a quick road to being a bottleneck and then to burnout. You have to aggressively recruit people to be bus numbers on all of those things. This is part of why I do so much mentoring.

> I'm advocating for people having enough experience and foresight to understand which 20% of the code flow is Architecture from which is just feature factory work and making sure that 20% gets 80% of the design thinking and timely preventative maintenance.

This is true until you're at a company that is a feature factory and nearly exclusively hires out of universities and struggles to work industry hires into their culture. And these tend to make up a glut of the companies a rung or two below the FAANGs/MANGO.

>> People often work to build things without any thought given to those who have to maintain it afterwards. Especially when it's not them

Because for maintenance there is a different team and budget. Same goes about the meetings. Recent example: it's okay to blow $1000 in man-hours to discuss and reject an increase of the compute capacity of my virtual workstation. I bet the cost associated with request was ~$25 of increase to the monthly bill. But it is from the other budget, so it was refused.

I've been doing this a long time and I've rarely seen a separate team and budget for maintenance.

Except for "all of the original team left and we're just here to keep the lights on."

I think it's important to distinguish between situations where the business intentionally made this decision from ones where it was forced upon them. Because the delusion of thinking we can still go on after we scared off all of the people who made this thing leads to a very different dynamic.

> This often massively discounts the cost of reversing decisions. People often work to build things without any thought given to those who have to maintain it afterwards. Especially when it's not them.

This is often because no thought was put into making decisions reversible. Version control, CI/CD, virtualization for example can make it easier to reverse decisions and make it faster and easier to replace bad designs.

When you first introduce the idea you'll find a bunch of people think decisions are Reversible which are not, as you say. And the flip side of Reversible decisions is the Last Responsible Moment, which runs afoul of Hofstadter's Law, and people wait until halfway past the last responsible moment.

The key to reversing a decision is getting over Sunk Cost, to start thinking of some code as scaffolding. Scaffolding allows you to get on to other work and then remove it after, because it's either not needed or the 'real' solution has been installed. People get defensive when you propose to rip out their code. Hey that code made us $250k at a time when we were about to miss payroll. Yes. It did. Thank you for your service. But now it's costing us $30k a month and that shit needs to go.

Getting people to figure out that if a decision is important, making it later is actually the sane thing to do, is a challenge. Because many people's intuition is that we should put energy into this now while it's fresh and we have abundant energy. We can 'solve' it and not have it dangling over our heads. But we don't know the right answer yet. We don't know the strength of our tools or the expertise of our coworkers.

The product roadmap is now and has ever been complete bullshit. Refactoring teaches that you amortize rework across all new stories. That's just how it goes.

(And everything should be in GMT unless you can literally point me to a several hundred page treatise on why another time zone is the correct one. Yeah I've worked west coast places that got bought by NY or Chicago companies and it's a clusterfuck if you both didn't use GMT)

Don't get me wrong, I'm a fan of the idea. I pretty much follow a lot of what you're saying without giving it names. I'm just used to people giving lip service to certain ideas as an excuse to move with less friction in their org and end up doing long-term organizational damage.

Thank you for the added detail.

> The product roadmap is now and has ever been complete bullshit. Refactoring teaches that you amortize rework across all new stories. That's just how it goes.

Also agree, but teams use sprints and "the roadmap" as a way to say no to fixing bottlenecks they've created for other teams and don't want to take the time and effort to resolve.

I sort of agree. But I kind of think it misses the point.

Bike shedding means no decision is made. That is the worst possible outcome as it provides absolutely zero value. Even a wrong decision provides at least some value even if it fails horribly. Maybe we paint the shed black and it burns down due to excessive heat, but it’s a bike shed so who actually cares if we need to rebuild the entire thing and paint it white this time.

And when it comes to bike shedding, getting it “wrong” is by definition already low stakes. It just doesn’t matter a whole lot.

Even the timezone example isn’t really that material IMO. Sure, it’s an extra annoyance to normalize timestamps to locale in every app/debugging query. But in the end it’s just not that big of a deal.

Worse would have been that early team arguing about it for 6mo without implementing anything. Then no one would be employed currently to curse those early developers to begin with.

Often times the only wrong decision is not making a decision. Very rarely are there decisions to be made that are not overtly obvious which are actually material. These exceedingly rare situations are where the winners from losers get sorted out - but that sort of decision in my experience so far happen a handful of times during an entire career.

As I moved up the ladder in my life I’ve often found my job is to basically flip a coin when a decision choice is presented to me. Very rarely is the topic worth me spending any material amount of mental energy on.

> Bike shedding means no decision is made. That is the worst possible outcome as it provides absolutely zero value.

And "bike shedding" is now a term that people hyperbolically apply to any discussion of something the labelling-participant doesn't really care about. A lot of times doing nothing is actually the best business outcome. Waiting until you have more, better information is great.

Lots of stakeholders ask for things that they think matter and then realize two weeks later doesn't and a ton of man hours have already been spent trying to build those skateboards as fast as possible.

> Even the timezone example isn’t really that material IMO. Sure, it’s an extra annoyance to normalize timestamps to locale in every app/debugging query. But in the end it’s just not that big of a deal.

Oh god are you underestimating the long term impact of data not being stored in UTC. For one, it will be a perfectly reasonable thing that your engineers forget this fact and build new systems in UTC, because that's the reasonable thing to do. Also as the years go on your company will make acquisitions of other companies and whatever timezones their data is stored in (hopefully UTC but obviously not always). To properly integrate this data you have to determine for every timestamp whether there was Daylight Savings Time happening at the time on every single object. You also have to figure out all kinds of edge cases around leap seconds.

Otherwise you have a ton of data that's out of chronology. Pity you if that's a thing that matters to you (it almost certainly does).

Also maybe you're lucky enough to be in a position to modify timestamps of all of your objects in all your datastores without significant engineering effort and coordination...but probably not. Obviously the reason the company I mentioned hasn't done it is because they're in that position...

But say you don't want to "fix it" and just want to do this "on every query"...now this process above is something you have to do across maybe thousands of services for potentially every request...

No big deal, he says...

Well you shifted the goal posts. Storing everything in EST vs UTC isn’t great, but isn’t some business ending level problem. It annoys engineers and causes inefficiency. Unlikely to even show up on a quarterly report if someone chose to write a mandate to fix it.

Once you get to the point of bolting on random systems that need to talk in whatever time zones is when you bite the bullet and change it. Yes, that is expensive.

The point I’m making is that if you spend a week dicking around with two junior devs while in your startup phase arguing about EST vs UTC you lose by default. Would much rather have my imaginary employee just choose EST in 14 seconds and move on at that stage. And I have dealt with systems that have done exactly this.

When it becomes a problem I can afford to fix it. Talking about it for more than 5 minutes at the bootstrap company stage is the definition of bike shedding to me. If those are discussions you are having in your weekly dev meetings you have already lost.

This will obviously depend on exactly what you are developing. Payment systems? Probably way more important to get this right vs. some social media app.

The ironic part to me is that this comment thread is basically bike shedding itself! Timestamps are an age old nerd snipe.

> It annoys engineers and causes inefficiency.

No, no, and no.

It causes unforced errors which damage your relationship with customers.

Burnout is a combination of wear and tear and lack of impact. Having giant footguns laying about facilitates the former. And I’ve been in arguments where people disagreed about time zone information and that is not good for interpersonal relationships. It’s not about efficiency it’s about turnover.

Bike shedding means no decision of consequence is made. People wanted to participate in something, and they sucked all of the oxygen out of the room for other decisions.

It's that important decisions were either not made or got insufficient scrutiny that's the problem. So aggressively shutting down bike shedding that is actually bike shedding by picking a solution and moving on or tabling the discussion to let a team of 2 solve it offline is what you're after.

Sometimes though, bike shedding can be a delay tactic. We have a decision we are not ready to make that someone is forcing a vote on, and there's a silent conspiracy to derail the conversation to avoid the vote for one more meeting.

And then there's people learning not to be responsible for anything, either as a career choice or due to previous trauma at this business. So they are asking for too much input so any consequences are not their fault alone. As you say, being a coin flipper in such situations becomes its own soft power. I've worked at a couple places where a handful of people clearly followed my advice solely because they knew I'd say it was my fault if things went wrong. They just wanted a Decider and I sounded like one so they did things the way I wanted them done. Their motivation was to avoid having to deal with being yelled at. I did my yelling in private, instead of turning it into an opportunity for public humiliation.

Reply on news.ycombinator.com