Hacker News Reader

Comment by noisy_boy | Hacker News Reader

I barely get a day or two to do some research on a new piece of work e.g. scope the tooling, mental model, checking how well the third-party components integrate etc before I need to provide an "update" at the status check meeting on "where we are". If you say that you are researching more than once, you are told "if you are facing difficulties, maybe xyz can have a look at it". It makes you feel that you are being labelled as slow/not quick enough and basically comes across almost like a threat of your work being assigned to somebody else who doesn't need to research as much.

View on HN

A lot of this comes from hard lessons learned at the business level. If you're ultimately responsible for the success of a project with any significant level of uncertainty, you very quickly learn that any significant level of uncertainty means that things are probably not going to work out well. This becomes a knee-jerk fear response whenever any report says that things are not going according to plan, where the immediate impulse is to give that work item to someone who promises to make it go according to plan.

Very quickly leads to a race to the bottom where executives settle on tried-and-true strategies for driving revenue: give Google & Facebook lots of money for paid leads, spend a lot on salespeople to convert potential customers, nickel & dime them for everything, work your engineers to the bone to deliver the features that salespeople promise, and flip the company before the accumulated technical, management, and oftentimes financial debt kills it. Ultimately, if you want durable success, you need to take some strategic risks and invest in projects where you don't know the outcome before starting. But that goes against nearly every emotional circuit in how we're wired, so few executives can do it.

I agree financial debt can kill a company. I'm not familiar with what management debt is.

But I don't know of any companies that were killed by technical debt.

> I agree financial debt can kill a company. I'm not familiar with what management debt is.

> But I don't know of any companies that were killed by technical debt.

I would imagine the failures of those companies are attributed to other causes. Is it hard to imagine a company with a low bus factor being in an unrecoverable state if a key person leaves? The time it takes to get a new person up to speed could cause the business to cede ground to a competitor, just as a for instance.

Companies with high technical debt don't die suddenly they are prevented from adding new features. What kills the company is a change in the world that the system cannot adapt to because the debt.

Well if the debt prevents you from acquiring new customers since you can't add more features then that would certainly cause most businesses to die.

Technical debt doesn't (usually) cause "quick death" to companies.

I think a good analogy is that technical debt is like having a poor diet - if you don't fix it you end up with diabetes and other issues. Sure they're treatable, but it becomes harder and harder to continue innovating.

When all is said and done, you don't know for sure if that poor diet was what lead to a companies demise - but there's no way it helped them.

It's not the prima facie cause but a lot of issues can be traced back to technical debt. Being out maneuvered by a competitor and just losing customers can be because it was too difficult to produce new features due to technical debt.

That’s because when technical debt kills a company, they attribute it to something irrelevant. Imagine Wikipedia, for example, and the great “Wikipedia has Cancer” post[1]. You might conclude, if Wikipedia fails, that it couldn’t secure enough funding, even if a significant amount of time and money was spent on dealing with operational issues that shouldn’t exist. In my experience with Mediawiki and scaling up a wiki, I feel that it suffers intolerably from technical debt, even being state of the art as far as wikis go.

Here’s a good example. Many modern websites have task queues to perform background tasks. So, too, does Mediawiki. You may assume it uses RabbitMQ or another queueing software, but nope, it stores the queue in MySQL (acceptable) and by default it runs cronjobs at the end of requests (less acceptable.) As your wiki grows, these jobs take longer and longer and happen more and more. Suddenly you are wondering why pages hang and you always run out of PHP FPM workers.

Then you look at the stack traces and it makes more sense... so you go to configure cronjobs. Except, there’s not really a great fix. You can tune the parameters so that the queue effectively never progresses, but then you have to run the jobs manually - which is fine, but there’s no task running daemon or anything. Actual cronjobs work but are catastrophic for responsiveness. So... one of the recommended solutions is a shell script that loops and runs the runJobs.php file repeatedly. This solution works, but from an operations standpoint it’s not a wonderful solution. You probably want at least some visibility into what’s going on, and bash scripting is hardly the best platform for that kind of thing.

Of course, this is just one microcosm of Mediawiki, you can run into one technical debt related issue per day and you’d not run out for years.

[1]: https://en.m.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost...

Management debt is when you create social or organizational policies that solve the immediate problem but will create morale, personnel, or legal issues later. The concept comes from an a16z essay:

https://a16z.com/2012/01/19/management-debt/

Other examples might include tolerating an aggressive status-based "brogrammer" culture, institutionalizing long working hours, or instituting gatekeepers that can block launches to avoid specific issues that have bitten you in the past.

I've worked at two startups that were killed by technical debt, and founded one. The way it usually manifests is that the startup fails to get to product/market fit before the complexity of its codebase prevents any further progress. These failures usually happen early in the startup's lifecycle (before many people have heard about them) and are often chalked up to failure to find a market, but the reality is often that if they could've iterated in days instead of months they would've found a winning product. Frequently a competitor started a few years later comes in with a simpler approach, leveraging building blocks that have already gotten mainstream vetting, and takes the market. (For more prominent examples, there's projects like General Magic, Apple Copland, Windows Longhorn, Chandler, and Friendster.)

Take a look at Knight Capital for a case study -- tl;dr, they deployed an update that accidentally triggered a blind spot in their trading system which had been inactive for almost a decade. By the time their engineers were able to stop it, Knight was on the hook for several billion dollars' worth of trades and had to close their positions for a loss of almost half a billion dollars. From my understanding they ended up merging with another firm to cover their losses so while they weren't entirely "killed", they were still severely kneecapped by tech debt.

A lot of this is work culture. You can push back by trying to formalize the research phase. Saying "I'm working on the architecture/design document, we have some known unknowns regarding [foo] I'm hammering out" shows other people in the meeting that you are working towards their problem in a meaningful and structured way.

Ultimately, if the people in the org want to push the work to xyz instead I would let them. Maybe xyz is as good as they think, but probably everyone in that room will enter a slog project/feature.

When someone makes a threat your default behavior shouldn't be to back down.

definitely needed this, thanks!

This is a good answer.

What managers dislike is a lack of transparency. "Research" is very vague. How can anyone tell what you were researching, or what the result of it was?

A very clear spec doc is a useful tool - almost like writing an essay - to demonstrate what your research uncovered.

Agreed on the structure part: structuring research is incredibly helpful for guiding the research itself, knowing what’s left to do, and the nice bonus of also making it easier to share the results, both incremental and final.

Research is in many ways naturally more unstructured than other dev tasks and people don’t get too much practice at doing or delegating it, so it can be fertile ground for organizational misunderstandings. Structure agreed upon up front can help all that.

I think they key word in my post wasn't "structured" but "shows". Even if the occasion calls for 100% unstructured research (let's spend some time in the sandbox to see what we can do with some new software) it's a critical skill to be able to communicate what you learned during that phase with other people, even people with different job titles and backgrounds.

I suggested formalizing and structure as words because new features/products/projects are a common enough occurrence that we have standard procedures for it.

Architecture decision records (ADR) are an excellent way to show managers that work is proceeding, and that research is necessary, structured, timeboxed, and transparent.

https://github.com/joelparkerhenderson/architecture_decision...

Speaking from the other side, my advice is to over-communicate, especially if you haven't built up a deep trust and reputation with your team and your manager.

Here's what your manager is worried about:

  1. Will the job be done well?
  2. How long will this take?
  3. Are you on-task, or are you stuck, overwhelmed, going rogue?

You can help your manager answer these questions by communicating these things:

  1. I have a plan, and here is the plan.
  2. Here is my progress in executing this plan.
  3. I have made these findings so far. I anticipate 
  these difficulties and challenges.

How this is communicated to the team, and your manager, depends on your team culture and workflow. But something typical might be:

  1. Two sentences in standup. Think through what you are 
  going to say, read a prepared statement from a sticky 
  note if necessary.
  2. A short email (<100 words) answering any questions 
  people have regarding your research. Use your own 
  judgment as to audience size and email frequency.
  3. A document or wiki page that is the final work 
  product of your research.

This is a wonderfully concise summary of the situation with concrete and helpful advice! This advice applies not only to research but equally to any longer-running project or task status update as well.

As someone who has felt frustrated in the past in both roles at play here, I expect I will find myself sharing this comment with others soon and often. Thank you!

I find it's best to think of and also communicate the research phase as stage of the project itself with a report at the end as a deliverable.

At the status meeting, talk about the stages you've accomplished so far ("I've completed scoping, looked at these three third-party components, etc").

When the research stage is done, write up a research report with what you did and the findings and present it to the team. This gives the rest of the team visibility into what you're doing. It's also very helpful months later when you question "why did we pick Foo?" because you can refer back and know exactly why you did.

This is helpful. Thanks for this.

I can't tell you how many times I've been on the team and decided on X, and years latter someone asks why not Y. I can remember many discussions and Y was part of it, there often were good reasons not Y - but I no longer remember them.

> think of and also communicate the research phase as stage of the project itself with a report at the end as a deliverable

Yes. This is the classic funded feasibility study / scoping study phase. The report deliverable can include a price estimate and also a risk estimate, e.g. expressed as the cost of additional rework, especially where there is a residual uncertainty (e.g. the use of a new component / framework, etc) at the end of the initial study. How companies manage their risk 'pots' is extremely telling.

In some situations there can be additional breakpoints, sometimes contractually, if third parties / subcontractors are involved.

‘Architecture Decision Records’ are one pattern of attempting to document these discussions in a lightweight manner.

http://thinkrelevance.com/blog/2011/11/15/documenting-archit...

I think a lot of famous programmers did good work because they had the freedom to explore things and iterate until they had something that was good. In my company you often have to explain yourself after a day or two so you feel constantly guilty because you didn’t produce anything.

But my best work was often done when me and others had weeks or months to play with things without interruption until we found an approach that really worked.

> If you say that you are researching more than once, you are told "if you are facing difficulties, maybe xyz can have a look at it".

This.

A thousand times, this.

I think a lot of times that's just a failure of communicating progress on research. Like others have said you can break 'research' down into sub tasks and even present a document or report at the end.

> present a document or report at the end.

Yep. Our research tasks either produce a document for the team to read and then discuss and/or user stories for the team to discuss, point and work on. Which happens depends on what the research is about.

Also, if someone is a manager/lead, make it a point to read, understand and ask questions about the produced document. Nothing demotivates more than producing something that isn't used.

When someone questions my work, effort or results from what I feel is an illegitimate place (ex: they're just trying to score easy political points) I find the easiest way to resolve the situation is to ask them to take on some non-trivial component of the work and share it with the audience at a future date. Nothing shuts down jerk disrupters like the potential of performing hard work.

can you give an example on how they are questioning you?

What if they offer to rewrite your components as well to ensure quality across the board. Could lose a project that way.

I strongly agree with this, and want to add one wrinkle.

Depending on how you count, I have somewhere around 30Y of paid coding experience. It took me until maybe 5 years ago to have discovered this for myself.

One thing I've noticed, though, is that if you create a culture around this, you'd must have guide rails in place. Otherwise, in the wrong company, or people will use the culture to present shoddy work with the expectation that other (senior or not) developers will challenge. These people exploit the culture by using shoddy implementations and challenges to offload hard parts of their projects. This is the mirror image of the illegitimate questions: illegitimate claims.

As a senior engineer, you should ALWAYS be willing to lead from the front with code deliverables, but your review-to-code ratio is so skewed that the number of times you can satisfy that kind of challenge is low compared to the number of times when you might offer (sometimes negative) input.

You need to better presenting your research. If you've done two days of research presumably you have 10+ hours of work to summarize.

"I talked to person A, who said we need featured X,Y, and Z. Then I talked to person B, who said they need A, B, and C, but feature C and Y aren't compatible with each other, so I'll need to reconcile that".

"We'll need to use third-party products A, B, and C. B has the API we need, but I'm still waiting to hear back from the C's vendor to see if they support API call foo."

>> I talked to person A, who said... Then I talked to person B, who said...

this has the added benefit of discouraging most developers from being willing to take over the task

> e.g. scope the tooling, mental model, checking how well the third-party components integrate etc

This is very insightful - what if you first created a formal plan with a bunch of checkboxes so that your status audience can see a steady moving progress bar. I've been there and agree if you keep saying the same thing people start to get nervous.

> If you say that you are researching more than once

Then maybe it's just that you aren't specific enough?

You aren't just researching, so tell them what you did, what you are doing and what you plan to do instead.

I had to implement SAML quickly on our software once. It's a subject that needs quite a bit of research and ideally not much implementation. So here a timeline of what I would have said if you asked me an update regularly (which is probably quite close to what happened because we needed that for a demo pretty quickly and they asked me quite a bit of update):

    T + 5m: I'm still trying to figure out what is SAML. 
    T + 1h: I found what is SAML, but I am trying to understand how it works technically. 
    T + 2h: I'm looking for libraries to help me implements it because it's too complex to implements in house and not worth it 
    T + 4h: I found libraries, but I lost a few hours because it needs to be implemented as a login-module, not a library.
    T + 5h: I've found one that has potential but our version of J2EE is too outdated and it may not works. I'll need to experiment with it.

At this point I gave 5 updates, I just did research, but never I said the word research. Each time what I said was justified and it doesn't sound unreasonable.

At my work we use what are called spikes to indicate that research is required and at the end of the spike you must have some documentation or a proof of concept produced so other people don't have to do the same research again. Unfortunately with your job it appears that the problem might be more that business relies on what they deem 10x programmers instead of creating a collaborative environment. They should be saying, "Hey if you're having difficulties maybe you and xyz can look at it together". If developer xyz comes to the same conclusion as you then business should can it at that point. If they don't then it could be that you were wrong and having another set of eyes on it helped out a lot, or it could be due to weird competitive culture where people want to take credit for things. I try to avoid this as much as possible by discouraging usage of the word I during any team discussions. It's almost always should be We. If one person fails, everyone fails. If one person does good, everyone does good.

As a PM, for any big project I always try to get a good developer a full two week sprint to do nothing but whatever research he or she feels is appropriate after I have a basic framework of the functionality built out.

I'm convinced that 100% of the time, this saves time. The problem is it's tough to quantify the time saved from mistakes or mid-course changes that you would have made without having this time, and it's not easy to get execs to give up two weeks of time from a senior engineer who will really do a good job at this.

Obviously anecdotes are everywhere, but at my work I never had any issue saying "we will need 2 weeks for research on how to use this technology", if you say you need 2 weeks to assess something then you need 2 weeks. What, my producer is going to tell me that I need less time? Based on what?

I think your issue is that you're doing open ended research - if you don't specify upfront how much time you need to investigate something then no wonder you're being asked for updates after 2 days.

Reply on news.ycombinator.com