Ask HN: What would you do for developer experience | Hacker News Reader

Ask HN: What would you do for developer experience

29 pointsAtotalnoob47 comments

Hi, I’m a principal engineer and I’ve been put in charge of developer experience at a company I’ve just joined.

Currently:

- this is a .NET, JavaScript, and python shop with around 100-200 engineers. I don’t know exactly, because they don’t even have a full list!!!

- CI/cd is manually configured by a centralized team.

- cloud is not meaningfully leveraged (it’s all VMs, including for things like queues (which was manually built in house)) and is also centrally and manually configured.

- little to no documentation

- no shared libraries

- code is siloed off by team

My plan is to

- create ci/cd templates, starting with a paved path into k8s

- create iac templates, starting with the most common patterns

- create a shared wiki and require teams to put access directions, openAPI specs, and each environment urls.

- scan all repos for secrets and make internal access the default

- require linting and security scans on all code.

- create a quarterly summary and survey of engineers.

- create standardized new app template.

What else should I do? What else should I start implementing?

View on HN

Before you go changing anything, I'd go talk to all the people who are there - understand where they came from, how the systems got to where they are, why they made their decisions, what they like and dislike, and what their biggest complaints are as well as what they love.

Then look at solving their actual problems. Because if you just come in and declare how you want it to be, you aren't improving their experience, you are just inflicting your own preferences on an existing ecosystem.

Dev experience is about people, not tech. You will use tech to improve it, but your devs are your customers - talk to them.

I’ve been in situations like this as one of the existing employees. It is beyond irritating.

Eventually there’s a presentation or meeting where the new person declares there is no documentation, or formal process or they’ve spoken to a former colleague who knows about X and thinks Y about our situation. And yet we have documented what they need, there’s a process written out and audited that we can go through and instead of asking your ex colleagues why not speak to your new colleague who’s been working on X for 3 years. I blame the manager to be honest more than the new person. As a new person though it is worth realising that this team who has been successful enough be able to offer you employment might know a thing or two.

Yes, I understand that. There is always a good reason as to why. I’m listening and measuring before we make changes.

It’s not documented and if it is, it’s not surfaced in an easy to consume way. It doesn’t matter how good your docs are if they aren’t easily accessible.

I totally agree with you, though. The new guy shouldn’t change anything for the first X time. And I won’t. I’m just developing a roadmap for the future which will change and adapt.

I think documentation is a great example - the problem is not "nothing is documented", or even the accessibility. The problem is "lack of documentation causes X". Once you establish the business pain caused by the technical shortcomings, you have a list of "Xs", and can prioritize based on the level of impact of each. Then you come across as someone really looking to help and improve the organization, and not as someone just pushing personal preferences.

100% this. we had a contracting company and in the first year changed out our source control system because they didn't like it.

We went from a mono-repo (which worked for us) to 50 separate repos because they couldn't figure out how to manage a mono-repo.

They changed our daily work style, forced sprinting on us, and then had to backtrack when execs told them to change what they were doing. They still don't quite know how to manage us, but they are sure trying to force their ways of working on to us

Excellent advice. Right now, many points on OP's list make it sound like OP wants to improve dev experience by having teams jump through extra hoops. (To be clear, some of OPs ideas are very good in the long term.)

I would start by realizing that often the desire to change things in a new environment stems from the need to regain control and/or to help with sensemaking. I think starting with some "ethnography" could be a more productive approach.

Lots of truth… but, there are two models for change.

One of them is making incremental improvements to what they are already (Say improve performance by x1.5) but making a big change (say x50) the existing mental models are probably part of the problem and not the solution. Not that you should ignore people, or leave them feeling like you don’t listen, in fact it is a considerable challenge to keep people engaged when looking for fundamental change which is possible with software, particularly in 2024.

Are you a principal eng? The principal engs I’ve known express things in terms of measurable business problems. You have a list of practices with no success criteria.

I'm also surprised that they're asking HN what should be improved -- I think HN is a random assortment of people working in or interested in tech, many of whom will happily give advice based on no expertise. I hope "ask on HN" is near the very end of their list, and they're just being thorough? ¯\_(ツ)_/¯

I’m just being thorough.

Most of the stuff people have mentioned I have done/will do.

This sounds like mistaking five minutes of googling as research. Sorry, for being so snarky. Please look around you and talk to your new customers as other posters already suggested. And talk to the management. Which perceived problems lead to the creation of the position for someone responsible for developer experience?

A lot depends on exactly this, and what types of applications or features the developer org is responsible for building.

IaC could be useful if you're constantly spinning up new standardized apps, but if you're looking to increase deployment velocity it's not going to do much.

My context has been startups with engineering teams of 15-60 engineers.

In general, I've found CI/CD templates, IAC templates and app templates to be an anti-pattern. It goes to the heart of scalable vs replicable.

> Put simply: something replicable can be copy-pasted (with variations as needed) to grow impact linearly in relation to effort and cost. Something scalable can create impact at a rate that increases faster than the rate at which your effort and costs increase.

In the contexts I've worked in, templates enabled us to rapidly grow and expand our business and technology in unmaintainable ways.

We scaled to 100 services, but our revenue and headcount didn't scale linearly to support the services over time.

---

In general, I lean towards solutions that facilitate distributing and updating templates over solutions that facilitate copying/pasting templates.

- Kubernetes this meant a common Helm service chart for the organization -- Now we could distribute and update kubernetes manifests to the org - CICD this meant building plugins that we distributed to teams - Application templates I've historically moved folks back into monoliths but the microservice chassis pattern looks like a solution here -- https://microservices.io/patterns/microservice-chassis.html

Your calculus might look different depending on your internal capabilities and resources. Good luck!

Perhaps I wasn’t clear in my post. Templates to me means distributing them in a usable fashion, and all templates should be able to be used via invoking some binary and providing inputs/flags.

Similar to helm.

Right now, we are using dotnet new templates, which is .net’a way of creating new apps, yeoman for npm, etc.

No copying and pasting. Copying and pasting isn’t DRY and is currently ramped at my org, since we didn’t have any shared templates

How often do developers have to make the same changes to their dotnet new templates? How often will they have to make changes as a result of the central team making an update to the central template that breaks their setup?

When somebody needs to use an escape hatch, how painful are you gonna make it to keep that escape hatch running?

So essentially in dotnet, when you create a new project, you run

Dotnet new templateName

Dotnet runs that template and out comes dotnet code.

That is what we have built and we consume internal libraries we created in order to keep people updated with changes.

But at the end of the day, it’s just dotnet code, don’t like what or how we did something? Go ahead and change it. It’s just regular code.

It’s similar to running “helm create {name}” or npm run create-react-app, but developed internally.

https://github.com/dotnet/templating/wiki/Available-template...

Templates to me means distributing them in a usable fashion, and all templates should be able to be used via invoking some binary and providing inputs/flags.

I am wondering why nobody talks about using version control tools for templates. Git lacks the ability to replace variables and handle flags, but it allows to rebase a small micro service on top of the latest version of the template.

These are version controlled. We use semver and they are released using the languages package manager :)

And how do you update a project started with template x version 1.2.3 to template 1.3.4?

What’s the deal with templates? I make new projects infrequently. Templates save me little time. I spend far more time struggling to fix broken code, fitting existing code to new requirements, etc

I'd basically first audit the entire infrastructure. Doing this will tell you about other problems (if any), so that you can take proper decisions.

Second I'd also talk to my customers - ie the developers on the challenges, pain points they are facing.

Next - you'd have two options. One is to build something (semi-automated or fully automated) or write scripts to achieve those tasks. Or you can consider buying an Internal Developer Platform. There are a lot of solutions out there that can solve for your potential needs. The answer depends on whether you have the internal expertise and bandwidth to do something like this. If not, buying an IDP would be a better idea.

> cloud is not meaningfully leveraged

I would very much warn about this. "Leveraging" cloud is a massive double-edged sword.

I've been in places that "leveraged" cloud. In practice this meant that every application had "gaps" filled in by cloud-only services that are impossible to run locally, can have their own state separate from application state and are a black box when things go wrong.

There is great value in being able to run your entire stack locally and having superuser access on all the parts. You may however not see that value until "spooky action at a distance" starts happening and the lack of access/local reproductibility impedes troubleshooting.

Yeah vendor lock-in sucks.

Everything we leverage by default should be able to run locally :)

>> cloud is not meaningfully leveraged

>I would very much warn about this. "Leveraging" cloud is a massive double-edged sword.

Absolutely, the "cloud" is a tool like many others. Don't use them if you don't have a need or even better a goal in mind.

Most of the developers and managers won't know what change needs to happen. If they did, they would have done it, and you wouldn't have been hired. They can give you insight into the lay of the land, but you'll get nowhere by considering everyone's prescription; the majority of them will be wrong and a minority will be right but you'll have little way to tell which is which.

Whether or not metrics will capture it, the efficiency of the organization can best be increased by centralizing the teams around one common toolkit to cover most use cases. You'll have to be judicious in deciding which cases can be exempted.

No matter what you provide for these teams, they will tend to stick to what they know unless they are forced to switch. Some can probably be persuaded, but your success will likely depend on having the backing to force them to switch one by one. The first ones will be the most difficult and then you can gain momentum.

Use your metrics cynically to this end. This all probably sounds rather dark, but leadership is tasked with leading, and inefficiency and disorder will rule the day until someone steps in and leads.

It depends a lot on company culture but ideally talk to a decent sample size of the employees across teams and find their perceived pain points. Then measure things: time to build locally, time for CI, time for tests to run, release frequency. Gather the combined data from measuring objective things and developers subjective experience and work it out from there.

If your role crosses over with SRE and platform then you need to get the data for these things to.

Make sure you understand why X is like it is for all X that you want to change. Know the history and the battles won and lost that got it like that.

Move slow and fix things!

Of course if the culture is all “quarterly goals, show progress etc.” then you may need to adapt for that too. Maybe get some quick win projects out that are no brainer things. Parallelize tests or project master branches etc.

By the way your list of changes might be perfect or awful. It is like a diet plan… it depends on the person and the reason for it.

This is a large task list, and many of these things will require a lot of time and effort to implement properly. Especially given the language spread, where a different solution is often required for each language, you'll need to prioritize effectively to make effective positive change (not easy, but important).

Those changes should ideally make developers' jobs easier, but it should also improve the security and quality of the systems. The priorities depend on not just what developers see as friction, but also what's important to the business.

You don't always get to make everyone happy, but one example at my company is that we implemented [OpenID Connect](https://docs.github.com/en/actions/deployment/security-harde...) with our GitHub Actions and Artifactory installation to enable passwordless auth for uploading build artifacts as part of a CI build. This let developers not have to deal with managing a password for Artifactory, and also made security happy because that's a lot less passwords floating around.

Another idea that's helped the comradery and quality of our teams is language-specific working groups. My company has a few thousand developers, so establishing a monthly meeting where you share ideas, and facilitating discussions of recommended paths is useful. Your job as developer advocate can be a facilitator of the objectives and tasks, and run the meeting. Again, it's a significant time commitment, but has been very valuable for us. To emphasize; you're paving paths, not building walls. This is key to avoid preventing teams from deviating as needed, but also attract those that are looking for a smoother way of getting where they're going.

Did you talk to the developers what they want to improve, their pain points?

> create a quarterly summary and survey of engineers.

Be very careful about creating performance, compliance, or progress metrics that can be followed in its wording but not in its spirit.

For example, if the developers do not like writing documents but you decide to somehow track a documentation metric per team or person, that might lead to grossly misleading "positive" progress.

Unless the developers see a value in what additional things they need to do, imposing metrics will not go well.

The summary is what my team has done for developers.

The survey is to measure the difference we are making.

First of the most important question: why? When proposing such changes, you need to carefully answer this.

Why do you want to implement those changes? Is it because as it stands currently it actually doesn’t work, or just because the status quo isn’t best practice?

> create a shared wiki and require teams to put access directions, openAPI specs, and each environment urls.

Imho this is a great change and pretty much universally applicable, but it’s not an easy one. Wikis quickly go out of date when there is no buy in from everybody involved. Also, you need to really think about the structure. The search function of some wiki tools (confluence) is famous for being awful, so make sure the wiki is well structured *and* remains that way.

> create ci/cd templates, starting with a paved path into k8s

CI/CD templates are great for some projects and awful for others. Depending on the standardization in your org in terms of tooling you may find it impossible to achieve any meaningful technology coverage for templates. My company has done a pretty good job in providing just enough scaffold to make building a CI/CD pipeline easy with libraries. We have libraries that make stuff like setting up internal package repositories, internal root CAs etc very easy and nothing more. People are quite happy with that approach.

However, a paved path into K8s and deeper into the cloud ecosystem of your choice (as implied by your status quo) is a kind of political and architectural decision I would not dare to touch in the pursuit of better developer experience.

K8s is a real beast and kind of overpowered for 95% if the instance is being used with. You need a real need for it to justify the investment required in terms of migration effort as well es engineering in maintenance (note that while managed K8s offers lift some off the maintenance burden off your shoulder, there is still a substantial amount of responsibility on your part).

> - create a quarterly summary and survey of engineers

Is this your entire job or is this some additional responsibility you plan on taking on. If the later, I’d suggest you stay away from this. I’ve seen this approach being tried several times and the result was always the same. After an initial spike in enthusiasm engagement is starting to die off while the effort required for upkeep keeps on rising.

> create a quarterly summary and survey of engineers

What is exactly is meant by that?

What's the testing situation like? Also, is this a body shop / consulting company?

Tests? We don’t got no stinking tests!

They have an okay manual testing practice, but adding automated testing is on my list. While it’s simpler to add unit tests off the bat, I am thinking we get E2E first. This is so we can make sure regressions of the input/output are not present while we make other changes. We can add unit and integration as we go.

This isn’t a body shop consulting company. It’s in insurance, but they are retooling the company to be engineering focused, which is why I joined them.

How long have you been at the company? Did you just join? I ask because no company is perfect. The hardest part which is what many comments seem to be saying is figuring out the "real" problems.

Let's take documentation. I tend to be the person who onboards people for source control. There is indeed a document buried in previous version of SharePoint that was not migrated to the new SharePoint. We only get new a couple new employees a year so it is just quicker for me to have a 30 minute session with new employees than to figure out how to migrate all the documents from the old SharePoint site. The 30 minute session is better than any documentation I will ever write. New employee gets to ask questions and so on.

Good luck. With 100+ engineers and practices that have been in place for years, it'll be very difficult for you to make major changes in a reasonable amount of time.

1. Talk to folks about what they actually want. People will find a way to resist/gum up things they don’t actually want to do. 2. Document the shit out of anything you change. Because you’re introducing a new standard on how to deploy, things will slow way the hell down as people have to figure out how to adopt these new patterns. 3. Then, dog food it. Spin up 5 new dummy services, and double down up improving the process, documentation, figure out where you can automate.

All of this is also to say, if a new principle engineer joined up after I had been there for 5-10 years, and suddenly told me we need to pivot to changes no one asked for, I’d either push back as hard as possible, or look for the door if I wasn’t specifically courted and take my tribal knowledge with me. Your job is to advocate FOR developers, not to impose.

> around 100-200 engineers. I don’t know exactly, because they don’t even have a full list!!!

Uh, perhaps start by making a full list?

talk with the devs, see what's the pain point, etc... and take it from there.

Have you been put in charge of dev ops or devex?

Developer experience. But isn’t devops part of developer experience?

Most of the metrics I’ve used for devex have been influenced partly by devops.

From https://github.blog/2023-06-08-developer-experience-what-is-...:

Deployment frequency (DF): how frequently an organization releases new software

Lead time for changes (LT): the time taken from when a change is requested or initiated to when it is deployed

Mean time to recovery (MTTR): the average time it takes to recover from a failure

Change failure rate (CFR): the percentage of changes that result in a failure

What’s the market sector?

As engineer in a .NET/javascript/python shop, I can tell you that literally ALL your suggestions would freak the shit out of me.

- k8 => why? I’ve never worked with k8, but I’ve heard enough to know it’s an overengineered shit show that solves a problem I don’t have. Probably most people think like me.

- iac template: don’t even know what that is.

- shared wiki: and this would be the 14th repo that is meant to contain all doc, and that would be totally incomplete and out of date before anyone reads anything in it. And if you had written any docs, you’d know wiki is the worst thing to ask your eng to work with. You want docs, write it yourself!

- scan repos for secrets, and make all access internal => how is that helping me exactly?

- linting and security scans on all code => well imposing linting is certainly going to help me reach that deadline. You want clean code? Clean it yourself, cause I surely won’t clean garbage code from a guy who left 10 years ago! Security scans? Meh.

- quarterly summary and survey: not my problem. Wait? Are you seriously asking me to fill in a painfully long form every quarter? Can’t wait for the “46% of you didn’t hand off your survey in time, you should do this over the weekend” email.

- standardized app templates: AH, at last something for me. Is that helping in any way? Probably not, cause I already have a standard way of doing things, so this again is some top-down imposition on how to do things. Unless your standardize way comes with a ton of stuff that would be painful to do in my way of doing things, this is more stuff preventing me to do my job efficiently.

—-

The problem is that you should ask yourself: what can _I_ do to help those guys? What can _management_ do to help those guys?

The best people I’ve worked with would take the pain points away. Like one of my boss would come in one morning and say: oh I’ve written the documentation for API x, and people would look at him and say “what? Thanks!”. Or one other came in and said “I hired some external database guy who’s going to go through all your sql queries and optimize them for you”.

“Ask not what your team can do for you, ask what you can do for your team” is a bit cheesy, but it’s basically the job you’ve taken. And no dev exp is not devops. At all.

You should really dive in the life of these people, take one-week rotations working very closely with all the various teams. See how they’d “onboard” you if you were a new employee. The first thing you want is a general overview of the place.

But you should really change your attitude towards your team. Comments you’ve posted here kind of show that you think you know better than the 200 guys who’ve been here before you. Don’t be that guy. We’re all engineers. We’re all smart. Things are the way they are because things happened. Learn humility.

I understand where you’re coming from “freaking out”. I didn’t want to write an overwhelming amount in the post. Perhaps that was a mistake.

I won’t make new work for people they didn’t sign up for. That isn’t fair.

I suggested a lot of changes and change can be scary. I’ve listened and already rotated through around 40% of development teams.

There isn’t a wiki or centralized documentation management system, so it’s not the 14th place to contain all docs. The docs that do exist haven’t been updated in years and are hidden away in random locations. Myself and my team will be writing a lot of the docs ourselves. In fact wejust hired 2 technical writers.

Scanning repos for secrets is needed because they have a LOT of secrets in their repos. That isn’t good, so I dug into it. The reason why they are there is because the average developer finds it “too difficult” to manage these in cloud. That’s a pain point that we are going to address BEFORE scanning and requiring the end developers to make changes.

Linting isn’t going to be turned on and you now have 5000 items to fix. First off, we plan on making essentially automated codemods to fix most of the easy issues. Secondly, we are setting the target state in conjunction with the tech leads on all the teams. Manual fixing of linting issues is nuts when you are talking about millions of loc.

The survey will be (max) 15 questions with multiple choice and 1-2 free text maximum. You don’t have to fill it out at all. I also will be hosting “office hours” and plan on being accessible as possible. I have to take some sort of measurement, otherwise how will I know how myself and my team are doing?

No one is making you use our templates, but we have already measured better velocity.

Everything is opt in and distributed using shared libraries. If you want to do things differently you can either not use my teams stuff, raise an issue and we will see if we can include it or open a PR with a change.

Devops is part of devex, I guess we will have to disagree. If people are having trouble with their pipelines and that’s a development pain point, then that is directly in my wheelhouse.

I’m not better than the 200 people that have come before me. I’m going to listen and adapt. I just wanted some outside feedback from other engineers.

I don’t look at myself as dictating changes, but rather advocating for developers and delivering them QoL changes.

Ok then it's just the way you presented things.

If you want to take anything off people's plate, please do!

If it's devops, I think people will be happy. Nobody wants to do devops. But engineers should not have to learn k8 because you chose to run things with k8. You choose k8, you take care of all aspects of k8.

You might as well take all shared services: permissioning, configuration management systems, logging, monitoring, environment management, email sending, message queues.

I’ve been at my currently company for almost 20 years. I’ve seen a lot of people come and go, most of them thought they were going to fix everything… none of them did.

The problem is everyone comes in with their own ideas of what things “should” be like, this comes from their past or what they read online last week, not based on the needs of the organization. Have you talked to those 100-200 engineers and listened to what their actual issues are? You could spend a lot of time trying to push them onto k8 where it doesn’t make sense, or when other very basic problems aren’t solve for yet. You could setup all these new things and no one will use them, because they don’t know how, education could be needed, and depending on where they are starting from, you may need to do this it steps. Taking them from crawling to sprinting could take years of realistic steps toward an ultimate goal. If you dump it on them all at once, they’ll just keep doing what they’re doing and wait for you to leave. You don’t know what that goal should be yet, because you just took the job. You may think you know, but you don’t.

I’ll give you a perfect example of this problem in action. I did the math last week and on average, in my years at the company, we change our CMDB every 3.8 years. This usually aligns with a new leader seeing asset management as a problem and decides a new tool is going to fix it. We spent a couple years re-writing everything we have to work with the new CMDB, almost get to a point of stability, and then that person leaves and someone else starts the process all over again. Our problem has never been with the tool, it has always been an issue with the data. It’s still the problem. However, none of the new people who come in ever take the time to listen to the people who work there, so they migrate an existing problem to a new tool, cause thousand of hours of needless work, and solve nothing. They assign someone to own the tool, but no one is ever accountable for the quality of the data. Good luck with automation when the data sucks… the lack of data and standards mean all our code is 10x longer than it needs to be to cover all the edge cases, and new edge cases show up all the time. This is one problem, I can give you a long list of other problems that follow a similar pattern, some are specially done under the umbrella of “developer experience” and all they’ve done is create chaos and confusion among developers who just want to focus on their app.

Don’t start implementing anything until you talk to people, and let their pain points and requirements influence the direction. Actually listen to them, don’t just go through the motions. This process will also build trust with those in the organizing. You’re new, they don’t know you, they have no reason to trust you, and you’re trying to change systems you don’t understand. That’s going to go poorly.

It sounds like step 1 is putting a system together to know who all the employees are and what their roles are. That can be used as a way to get to know who you’re actually dealing with, and can then facilitate the conversations to find out what you’re dealing with. Having visibility into who all the teams are, publicly, can also help facilitate teams talking to each other and breaking down silos.

Also remember, the technology isn’t the product. Your company has something they make and sell, and the customer doesn’t care if it’s on VMs, K8, or any of that. Make sure you don’t load people up with so much stuff that they don’t have time to work on the actual product being sold. The technology should help get things done, not be the blocker that prevents new features from going out the door. Keep the workload and timelines related to the idealistic state realistic and the priorities clear.

> Don’t start implementing anything until you talk to people, and let their pain points and requirements influence the direction.

This is especially important. People know what's wrong. You just need to ask them.

Reply on news.ycombinator.com