Hacker News Reader

Comment by MattGaiser | Hacker News Reader

How much of the workload is just the dev shortage/insufficient staffing?

I am currently the only dev on my team because of departures (there should be 4). Now, it is not really all that in production yet so it hasn't required late nights or waking up at odd hours, but it does linger in my mind that this could be a big issue if it grew a lot and my team didn't. I would hate to support it alone.

A friend of mine is missing two people on his team at another company. It means that they had to shove the interns into the support rotation to have full coverage of their app. They are hiring too, but as things stand, interns are staying up in the night deal with production errors and they are waking each other up as well, they are interns and don't know what they are doing.

Software seems like something that needs a sufficiently sized team after a certain point and nearly every team I know of anywhere is short people at the moment, even at FAANGs.

View on HN

> How much of the workload is just the dev shortage/insufficient staffing?

ZERO PERCENT. Listen to me very closely. The quantity of people has ZERO IMPACT on your work quality. Well... maybe like 3%, but hear me out.

Your manager has a HUGH impact on your work quality. The work that is given to you, your PERCEPTION of the work, the urgency of the work, your development opportunities (and your response to them), the people you "hang out with at work" (note, this is about quality and nor quantity of work friends).

Of note, hiring more people can help with things like "We need 24 hr support", but that can also be helped by things like "well, actually, we only need 8 hr support, and the customer can pound sand" - that is where a manager comes in.

You're talking about quality, but what about quantity?

If you need to develop:

  - the back end for an application
  - the front end for an application
  - some external APIs for the application
  - a structured data storage layer, say, with an RDBMS
  - an object storage of some sort, for example, on top of S3
  - some sort of reporting functionality
  - mailing list functionality
  - etc.

Then no matter how good your manager is, there'll be more work for 1 person to reasonably manage.

Furthermore, the quality of your work can be stellar, but if there are 5 different problems, all of which need solving and you tell your clients to pound sand, you might find yourself with no clients.

Understaffing feels like a real problem.

Having many engineers causes changes to your engineering workflows and standards that make them a lot less productive, in order to cope with having so many engineers. Also, some of your many engineers will choose to spend their time creating internal platforms that are much worse than commercial and open-source alternatives, and since you are good collaborative synergistic team players you must always use the thing written internally, no matter how bad it is.

Creating a new backend endpoint requires a multi-week design review process and must be implemented in a straight-jacket of a programming language with 90% test coverage and tons of layers and extensive mocks of each layer.

Creating a new frontend requires the same multi-week design review process, and must use the Nth incarnation of our NIH Javascript framework. The developer who built it got promoted and then left; it's been handed over to India to maintain. So anytime anything goes wrong, post a question to the framework maintainer's slack channel and wait to hear back in the morning. Or don't. They're just as mystified by it as you are.

Creating new external APIs requires, ironically, no design review but you must use the shitty homegrown code generation tool to generate a PR to add your endpoints to the API gateway. The builds on this PR take 4 hours. Then after it's merged you can wait for the API gateway to be deployed later this week. You must repeat this process any time the schema changes.

Creating a new RDBMS table goes through a theoretically self-service workflow hosted by the RDBMS team but probably one step fails so you need to file a ticket and wait for them to fix it before your table is provisioned and ACLs dialed in.

In-house object storage implements an S3 compatible API but you must submit a ticket with a detailed description of your use case and wait a week or two to get provisioned.

If your email is static or contains only simple account properties, it's easy to create in the internal email platform's template builder UI. But if it needs complex data from your application, you'll need to land changes against the email platform to implement fetching of that data from your application. No, you can't push data into the template from the sending application, that would be too easy.

Compare to one or a few people working directly with Django/Rails, React, S3, Postmark, etc. and practicing the "write tests, not too many, mostly integration" thing.

> Creating a new backend endpoint requires a multi-week design review process and must be implemented in a straight-jacket of a programming language with 90% test coverage and tons of layers and extensive mocks of each layer.

Or perhaps an hour or so long meeting, in which the overall design is discussed so everyone is vaguely on the same page and the rest is handled when reviewing the merge request and/or as Slack messages in the appropriate channels, as necessary. Heck, shoot a message to the business analysts and maybe have a 5-10 minute conversation to clear up any of the stuff that you don't understand.

> Creating a new frontend requires the same multi-week design review process, and must use the Nth incarnation of our NIH Javascript framework. The developer who built it got promoted and then left; it's been handed over to India to maintain. So anytime anything goes wrong, post a question to the framework maintainer's slack channel and wait to hear back in the morning. Or don't. They're just as mystified by it as you are.

Or use the exact same approach as above, with whatever was used in the other components of the system, for familiarity's sake. Some vaguely recent version of Vue? Sure, go ahead! Maybe even copy some bits that don't feel like they should be shared code and could change in the future, but that'll be easy to reuse for now.

> Creating new external APIs requires, ironically, no design review but you must use the shitty homegrown code generation tool to generate a PR to add your endpoints to the API gateway. The builds on this PR take 4 hours. Then after it's merged you can wait for the API gateway to be deployed later this week. You must repeat this process any time the schema changes.

Quite on the contrary - it's just an API that perhaps needs a bit more attention from the security specialists and some additional scanning here and there. Knock it out, give it to others to test and fix any of the problems that they find. Builds might take around 5-15 minutes.

> Creating a new RDBMS table goes through a theoretically self-service workflow hosted by the RDBMS team but probably one step fails so you need to file a ticket and wait for them to fix it before your table is provisioned and ACLs dialed in.

Or, you know, are integrated as a migration file within the application, that will automatically get applied to the testing/staging/prod databases when the changes land in the environment. Maybe even use a local DB instance for breaking changes inside of a Docker container, so that you can discard any of the stuff that you break.

> In-house object storage implements an S3 compatible API but you must submit a ticket with a detailed description of your use case and wait a week or two to get provisioned.

Or just add whatever software you need to a Docker Swarm + Ansible deployment on your company's servers and have it be up there in a matter of minutes. Maybe grab a coffee, because rerunning parts of Ansible playbooks is sometimes a bit hard, but procrastinating for 10 minutes never hurt anyone. Then test whether everything works in your development environments and let the people responsible for accept testing, security and prod deployments handle the rest.

> If your email is static or contains only simple account properties, it's easy to create in the internal email platform's template builder UI. But if it needs complex data from your application, you'll need to land changes against the email platform to implement fetching of that data from your application. No, you can't push data into the template from the sending application, that would be too easy.

Or just add a new Java class and some templating logic to generate whatever you need in the application code.

> Compare to one or a few people working directly with Django/Rails, React, S3, Postmark, etc. and practicing the "write tests, not too many, mostly integration" thing.

And possibly end up with an insecure, faulty and otherwise unmaintainable solution because there wasn't even a modicum of oversight, or alternatively don't ship anything in a timely manner because the scope creep was unmanageable for the amount of people that you had. Or maybe just end up with those people using SaaS solutions which are more like SaaSS ( https://www.gnu.org/philosophy/who-does-that-server-really-s... ) which you were legally not allowed to do due to where the data ends up and then have bunches of fun dealing with the legal implications of that.

Bottom line: if most of your time is spent fighting corporate policies and bureaucratic apparatus, it doesn't really matter whether you have 1, 5 or 10 people on your team, because dysfunctional environments can happen at any scale. I'd say that what you describe is an environment that's actively hostile to getting anything done, which could stem from a lack of decent management and people doing whatever they can to make themselves look better for a performance review, or to justify their own existence. Having too many people would definitely contribute to that, but at this point i feel like this discussion has perhaps devolved into some parody of the Goldilocks story, where the corporation would struggle to find the right amount of people.

Furthermore, i guess it's a matter of attempting to centralize everything vs having a decentralized approach, seeing as different projects and teams might need different solutions, hopefully mostly open source ones. I do think that you're perhaps also bringing up the fact that larger corporations oftentimes attempt to standardize on this stuff, much to everyone's chagrin, but that's not necessarily caused by the team size alone, rather the point in the company's lifecycle.

I should know - i've seen both of the scenarios above and having few focused people can indeed work nicely! For example, i developed the homepage for the contact tracing application in my country alone, a page that saw hundreds of thousands of visitors: https://apturicovid.lv/#en Due to that trust placed in me and the creative freedom, i was able to deliver about 70 different deployments of the site as well over a few months.

Did it work nicely? Yes. Was there also a pretty big possibility of me messing up? Also yes. Would this approach work for startups, side hustles and other smaller projects? Sure. Would i want to do the same for your typical enterprise project where battling scope creep and shifting priorities is the everyday reality, whilst the concept of set sprints is a joke? Probably not, that's where having more people makes sense.

This is known as the "mythical man month." For some reason software engineering resists more programmers equating to more productivity. I believe this is because of the theoretical and personal nature of programming. We are constructing in our minds ideas about how a system should exist to solve the problems at hand. By adding more programmers, we are adding more ideas, that when ineffectively communicated (certainly human nature), compound into a smorgasbord of discord and bug tickets.

There is obviously some validity to your point. A 10 person startup could not run a mature company the size of Twitter. But that same 10 person startup will likely not see a double in productivity by hiring another 10 programmers. My experience has shown they will see a mysterious doubling in their miseries, and a many more "strategy meeting" to ameliorate the issues.

I hope that you would agree with me that it is feasible for a skilled programmer to accomplish everything in your list in some expedient amount of time, even though it would clearly go faster if he had a buddy to work with.

[0] https://en.wikipedia.org/wiki/The_Mythical_Man-Month

> ... But that same 10 person startup will likely not see a double in productivity by hiring another 10 programmers. ...

> I hope that you would agree with me that it is feasible for a skilled programmer to accomplish everything in your list in some expedient amount of time, even though it would clearly go faster if he had a buddy to work with.

Agreed. What you're describing above does sound like Brooks's law, which states that adding more manpower to a late project does make it even more late (at least as far as the mindset that you're hinting at goes): https://en.wikipedia.org/wiki/Brooks%27s_law

Or maybe it's simply diminishing returns in your example above, which also seems valid.

For the most part i've observed that in my own experience, however that's not exactly what i'm advocating for. Think more in the direction of Amdahl's law, but applied to the management of tasks within a project: https://en.wikipedia.org/wiki/Amdahl%27s_law

There will always be parts of a project which can be done in parallel and therefore should - that's not to say that it's possible to skip out on context for how they'd fit together with the greater solution, assume that all problems are interchangeable or run into any other problematic lines of thinking along the way, yet the possibilities should definitely be covered!

Here's perhaps a better example from my dayjob: There are problems with the Foo view in the application, which are costing X$/month. There are also problems with the Bar reporting functionality in the application, which are costing Y$/month. Furthermore, the system needs a new Baz module developed, which is projected to bring in additional Z$/month. Also, the metrics show that historically there have been ops related issues, so introducting containers into the mix could help save W$/month, as it has already been proven in other projects.

So essentially you have tasks that concern the following functionality: Foo, Bar, Baz and some DevOps stuff. So, if each task takes T amount of time, that'll be 4T in total.

If you have 1 engineer they can probably develop all of that eventually, but all of the tasks will probably be addressed in whatever order they get to them, so it'll therefore also take them 4T.

If you have 4 engineers AND the tasks have little overlap (this is where Amdahl's law makes a difference), they can each work on one of the tasks in parallel, so suddenly 4T becomes 1T.

I think it's clear, which would lead to the highest gain, i.e. X + Y + Z + W.

Of course, people always love to talk in absolutes and noone ever considers the particular circumstances of the projects. For example, in the 1M SLoC Java project that i'm currently working on, a single pizza team is pretty unlikely to step on each other's heels. Even entire features can be developed and delivered separately, without merge conflicts down the road. Whereas on another project that i'm also working on, a much smaller one, i get merge conflicts whenever i'm about to make a pull request to merge changes back into the master branch. Figuring out which project is which and what approaches and design patterns to use to minimize the pains of scaling, that's the hard part.

It is a fundamental truth that there is more work to do than time available to do it. That this pressure lands on developers is really a perpetual management failure.

> Your manager has a HUGH impact on your work quality.

Are you talking a Hugh Grant impact or a Hugh Jackman impact?

Sorry. Couldn't resist. Hugely.

https://pbs.twimg.com/media/EetzqDFUcAA818B?format=jpg&name=...

I used to work in a company with 400+ developers and our team barely had any new features to add. We took care of one screen in the app.

However getting anything done was an impossible task and required huge amounts of code because of the complexity of the architecture.

I left after 6 months and a couple weeks because I couldn't bear the stress.

Fred Brooks is still right in 2021. More people, more problems.

> More people, more problems.

My theory is that teams scale exponentially. That means two people working together are probably as productive as two people working on completely separate things. (By completely separate, I mean totally different goals -- your weekend projects vs. my weekend projects. Definitely not employees at the same company, who in theory have some sort of common goal.) But it means that 10,000 people working on the same overarching project are about as productive as 14 people working on their own thing.

This sounds bad, but if you treat it as a law of the universe, it's not so bad. If you want to do a project whose scope is the same as 14 unrelated one-person projects, you just need 10,000 people. It certainly seems to happen in other fields; there are no one-person teams building skyscrapers or new subway systems.

People's workarounds for this are trying to split teams apart so that they can be somewhere lower on that exponential where the actual ratio doesn't feel so bad (1/1 is better than 14/10000). Unfortunately, it doesn't really result in great products and you end up with the classic "big company" feel -- where the company has 3 different chat apps, 2 infrastructure teams, is a gigabit ISP but can't upload or download to its own products at 1Gbps, etc. The individuals working on those projects are slightly more productive, but the company as a whole fails.

So I guess if you work at a company of size N, and you feel like you're as 100%∗log2(N)/N as productive at work as you are on your personal projects outside of work, then you're doing great. The problem is that this feels super bad to software engineers, who probably got their start working alone in their spare time. But it's the law of the universe and there is nothing you can do to fight it, except hope that you can figure out a business whose scope is low enough where that log2(N)/N is pretty close to 1.

(2 may not be the right exponent, of course, maybe it's something like 1.8. You'll never get around the fact that communication paths goes up like the factorial. At a 2 person company, there is only 1 1:1 to have. At a 40 person company, there are 39∗38∗37...=39! 1:1s to have. That ain't great scalability.)

This is why FAANG pays so much. Imagine 1000+ engineers working on a single page in a web app. Your entire team is responsible for 1 widget on the page.

Are FAANG developers more productive and efficient?

Coming from a bit different of a situation, I am at a government systems center that runs off customer funding rather than tax money, so we take clients. My group is being run short of people just as you describe.

Almost every project I have been on has been understaffed. My project last year was one year of funding to research application of deep learning for a problem that had only some classical engineering algorithms applied before. Besides the r&d, we promised to deliver software that they could use to verify experiments and test models against their systems, basically everything up to but not including integration (with integration coming later if results were good). The team was a PM, domain expert, and me. That's one person to do all the research and experiments and provide basically two working and tested software deliverables. Good times.

My current project is one piece of a government effort run shared between two three letter orgs, a smaller contractor that's kind of PMing plus infrastructure, a large contractor handling modeling, and us handling correlation. We already had a correlator written in Java for portability--that is why we were brought on--so should be easy, right? A bit of data analysis, configure the software accordingly, pass off the jar and config and should be good. The effort was funded for about two people plus a PM but had had one full time and one partial for the past six months. Our part of the project ballooned to helm charts, four Kafka interconnected docker containers, security scans on all that plus a bunch more stuff required every release, lots of core rewrites to hit their scalability goals, constant data analysis because every week a performer comes in with a "hey why didn't this particular example get correlated as expected" question. Basically one person to cloudify + do data analysis on big data + rewrite Java software to scale better is just crazy. I am on it because the "half time person" left a year ago and wasn't replaced, and the full time person is now leaving also in August, so they're finally forced to restaff. I was told I'd do data analysis and algorithm work only, but over the first month I've only done security fixes and cloud infrastructure related tasking.

I bring this up to point out, yeah it's cool to learn new stuff, but the shortage of people also means double or triple role work that definitely accelerates burn out.

I worked for a government agency funded by clients as my last job. Same issues.

A big issue with understaffing is knowledge. Like in my prior job and you in your current one, you are spread across so much tech. I know that even in my current job I have certain work that is little more than Google and assume the tutorial does the job.

It might also be applicants recognize that its not a healthy place to work -- interns staying up all night to fix production errors is a red flag. Then the problem is a bit self re-enforcing, and the only solution is to reset expectations and deliverables to maintain a healthy pace of work. If that pace is not possible, and the company can't offer competitive compensation for the scenario (ex: "We're asking you to carry the company, so here's some extra meaningful equity") then its in a downard spiral and its merely people abandoning a sinking ship.

Well that's just it. Apparently WLB is great otherwise at the company, but because they can't hire people, it now has become horrible.

Anecdotally, a heck of a lot of teams are in this state.

How much of the dev shortage is due to excessive gatekeeping? The average interviewing process has become very burdensome these days. Not sure it translates in a higher quality of hires wrt the past.

using interns as this is unacceptable.

is there a dev shortage or is there an expectation that you can hire people for cheap and work them to death?

In economic terms, there are no shortages. In practical terms though, it means that there are a bunch of projects out there that aren't viable without heavily working people and employers are incentivized to extract as much as possible from people.

projects that are not viable should simply not get done.

it always amazes me that people put up with this insanity. The first thing I teach junior developers that join my team is 1) learn to set boundaries 2) your WLB and mental sanity is more important than any project (if you are struggling you’re gonna get fired, ie don’t kill yourself with work when you are … disposable)

Reply on news.ycombinator.com