Ask HN: How do I feel less guilty about bugs?

You should step back and see the big picture.

1) they don't have to use your software. they could write their own, or choose another vendor, or do whatever your software automates manually. if they are using your software, they have chosen that option because, on balance, it meets their needs. that's not to say it's perfect, but it must do more right than it does wrong.

> For example they might spend an hour working on a task and then suddenly our code runs into an exception and their data is not saved properly.

Nobody should ever do this! Spending an hour working on unsaved work is not sensible. Either your software should save it along the way, or they should do their work in some other place (notepad?) that saves it incrementally, then copy and paste it in as a last step.

This is an example of where you need to step back and see the whole picture - it shouldn't be a case of doing unsaved work for 1 hour then saving with 100% reliability, it should save incrementally. Even if the save code were 100% reliable, they would still lose their work if they lose power, have a network disconnect, etc.

Unless you step back and understand why they are choosing your application, despite these problems, you won't be able to see the most important aspects of the business.

"Nobody gets fired for buying IBM"

> if they are using your software, they have chosen that option because, on balance, it meets their needs

This doesn't hold in any enterprise setting.

Coming to this case that OP describes, I think OP, you have high empathy and are passionate about your product. Since you're likely an entry-mid level engineer, sometimes others have more context of the business setting.

It would be good to implement ideas like: 1. Tracking crash reports 2. Coming up with a test framework that helps avoid regressions 3. Coming up with a script that can analyze errors, which reduces manual ops. 4. Setting up a job that compiles these errors and send it out over email every week, so managers and business automatically learn about what users are seeing.

This is a great point. Sometimes a more elegant solution will come to mind, if you only give yourself the free time to relax, explore the problem, and find it.

Guilt is probably not going to contribute to that.

The cure is using other software. It's almost all so fucking terrible—yes, even from FAANG—that before long you'll stop feeling bad about your own bugs.

That's not an excuse to get lazy about bugs, but it ought to help you not feel so bad about it.

Apple's way better about software quality in general, than most. Yet every day I use their alarmingly-janky multi-user login/switching screen. If I my M1 Air falls asleep while not connected to power, I'll have to juggle wifi networks to get it re-connected (if I don't want to wait the several minutes it takes it to sort things back out without that intervention). Last I checked opening a link in a new tab in the background while in a "tab group" in Safari still caused a flash of the real content before loading the blank-tab screen and losing the original URL, when you eventually switch to the backgrounded tab, basically making tab groups unusable. Even they ship bugs in big, user-facing ways, and leave them there for months or years.

Apple also ships less code than other companies, relies on 3rd-party apps to do the marketing research of what users want, has horrendous documentation, and tries as hard they can to block users from doing unsanctioned things with their machines. If it seems like they have less bugs, it's because they do as little as possible with their software as they can get away with.

Their hardware is great.

> If it seems like they have less bugs, it's because they do as little as possible with their software as they can get away with

This is tautological; the obvious way to ship fewer bugs is for your software to simply do less so there's a smaller surface area for bugs to happen in the first place. I see this as a good thing and I wish more software companies would take Apple's approach in this regard.

> If it seems like they have less bugs, it's because they do as little as possible with their software as they can get away with.

This does not square with my experience of their bundled software being very capable, very respectful of system resources, plenty featureful, quite stable, and usually sitting somewhere between "quite good" and "best there is".

That's a psychological trick. I got my first iPhone ever this week, so let me tell you about my experience.

One of the first text messages I ever sent via iMessage, it defaulted to texting the user's "home" phone number in the contact card, even though they had another number listed under "mobile" (and they have an iPhone themselves)!

The only reason I noticed was because the text message in iMessage was a green bubble, not blue. Then, I figured out that in iMessage you can't long-press on a message to see metadata about it (i.e. exact date/time sent, and to/from what number, like you can on Android).

The message silently failed in a stupid way, and I had basically no way of knowing anything save for the fact that I happened to be aware of green/blue bubble and knew what type of phone this person had.

Aside from that, I have noticed the little app bar on the iMessage keyboard will just disappear from time to time, inexplicably. The only way to get it back is to reopen iMessage.

I didn't want to enable Face ID, but I caved and did it because it's the only way to install free apps from the app store without requiring your Apple ID password every 15 minutes (my password is long, for security reasons... I figured I'd at least be able to just enter the phone PIN, but no!) Now I'm stuck with Face ID as a mechanism of authentication that I frankly don't want. Face ID isn't continuous, so if someone snatches your phone, points your camera at you, and runs, your phone is now unlocked for them to do whatever they will.

I don't think their software is actually that good, unless you stay precisely in the box they want you to stay in. The second you deviate from that, the experience is utter garbage. And if something DOES go wrong in their walled garden, good luck figuring it out. You literally have no ability to.

Your experience is a consequence of software that is largely inflexible and when stuff goes wrong, Apple just pretends it never happened.

I started my journey on DOS. I'd put more hours into probably five or six other desktop operating systems, plus Android on mobile, than I had anything from Apple, before I finally gave OS X a real shot (around 2011) and, a couple years later, after developing for both Android and iOS but being and Android user, switched to iOS for my personal devices.

I assure you, I'm not being "tricked", and I could name a bunch of shitty things about Apple software off the top of my head. They're not the best because they're perfect, it's just that everyone else is extremely bad so they look great by comparison.

> Our prospective customers tell us things like "we can't sign up until you support X number of features," and my boss and the founder believe that our number 1 indication of success is the downward trend of X.

I’m reading this as a pattern of prospective customers asking for more features, and the downward trend being that of the number of “missing” features.

As your installed base grows, the number of features supported and requested will tend to grow faster than the number of users grows. Users will always want more features. Even if you support 3 ways of doing the same thing, users will want more ways of doing it, ways that better fit their in-house work flow, or the work flow of some other software they have used elsewhere or in the past.

Your boss and the founder are in growth mode. They want more users and want more features to get more users, of course. But, at some point, the number of features and the number of interactions of those features will start to become a burden on the company, unless it is successful beyond the wildest dreams of avarice, and can afford to hire faster than the burden grows.

You, and your boss and the founder, should start thinking about a future time when you can feel that the product is nearly feature-complete, and thinking about what the natural limits of its feature set might be. If you start that now, you have a chance of having a coherent set of features in a supportable whole.

Stop feeling guilty and start writing/improving a test suite. Start using linters and set up a CI pipeline. Don't ask permission, just do it in between tasks. If management is at all competent they will see you as promotional material in the medium term.

There's a classic Joel on Software post where he talks about making workflow changes as a grunt. An important point is that you want to get your daily work done first:

https://www.joelonsoftware.com/2001/12/25/getting-things-don...

"… I also knew that making a good first impression was crucial. So I allocated the first seven hours of every day to just writing code, as was expected of me. There’s nothing like a flurry of checkins to make you look good to the rest of the development team. But I reserved another hour every afternoon before going home to improving the process. I used that time to fix things that made it hard to debug our product. I set up a daily build and a bug database. I fixed all the longstanding annoyances that made development difficult… Slowly, the process got better and better."

Sure you need to improve your software engineering practices. Take a look of your version control, deploy process and automatic tests. The increase of your QA team is probably a bad smell. Remember the old quote: "You can't test quality into a product".

But also pay attention to your company culture. A "blame culture" is one of the worst enemies of good software/product. If your management work by making people feel guilty, you can't have candid conversations about your problems. You'll never find what you need to fix, because you won't be able to discover the real problems.

I've worked in a company where the "blame game" mode were always on. It was terrible. People didn't want to fix a problem, just pass it to the next guy. You'd never find the original cause of problems or have your management invest to solve it.

If you have a culture problem, don't let your employer make you feel bad. Take care of your mental health. A job change may be the best option.

I love the idea behind this - as a software engineer, it's your job to build good processes, and not something to ask permission for.

But wow, some of this post is incredibly toxic and passive aggressive. Create your own personal bug tracker, then refuse to fix any bugs unless everyone else uses it? "Neutralize the bozos" by sending them incessant bug reports, until they're no longer productive? Can you imagine working on a team like that?

I totally agree that you should incrementally work towards hardening your systems. Inasmuch as it's something you can fit into your workday, I'd say that's just part of your job.

That he gets sarcastic at times is for entertainment purposes, not to hurt your feelings. Bozos do exist however and are not productive by definition. I don't think writing valid bugs should be a problem.

Memento mori. You are not your work, you are not your code, and you are not your bugs or lack thereof. You'll one day be on your deathbed, looking back to your life. I promise you you won't be thinking of a couple of bugs you introduced when you worked at some startup you worked for decades earlier.

>So it's probably an issue with myself that needs fixing, right?

You are feeling guilty because you care about your (your app) customers, this is not (IMHO) an issue, it is only you being a "good" person (like hopefully most people).

But 1% (if it is 1%) can (still IMHO) be a lot.

Let's say (for the sake of reasoning) that the app is about invoicing.

One of your customers used to be able to prepare 8-10 invoices per hour and now, thanks to your app, is able to make 16 instead, BUT every 6 hours or around 100 invoices one fails, the work done is lost and it takes "a few hours" to solve the problem.

Hardly an increase in productivity.

0.01% or 1 in 10,000 would be probably acceptable in this completely hypothetical scenario.

> You are feeling guilty because you care about your (your app) customers, this is not (IMHO) an issue, it is only you being a "good" person (like hopefully most people).

Yep. Perhaps an unpopular opinion but developers should care. They should feel bad about their bugs. At the same time, recognize that shit happens and you're not going to be perfect. But take that negative energy and use it at motivation to reflect on how you made that mistake and what steps you could take to avoid it in the future.

There are always bugs.

What keeps bugs from happening is processes, not programmer conscientiousness. Processes begin with a test suite, and enough time spent on maintaining the test suite. And generally, enough time spent on managing bugs.

The way for the software to have fewer bugs is for the organization to spend more resources on processes to reduce bugs. If they are choosing not to do that, perhaps thinking they won't gain/lose enough business over it to justify the investment... well, your boss has already told you that they think the number of bugs you have is appropriate for business requirements and there is no reason to invest in having fewer. So, you can accept that, and give up your feeling of responsibility to make things different (your boss has already said they don't want things to be different), or you can look for a different job, I guess!

I admire you for feeling guilty for putting out crap software, and don't want to tell you you are wrong. Too many of us get paid to put out crap software. I think we are responsible for making people's lives just a bit crappier having to deal with our crappy software. I don't want to tell you to become a soulless automaton who doesn't mind producing crap that makes people's lives crappier as long as you are getting paid.

(Alternatively, maybe it's not crap software? Even high quality software sometimes has bugs. Maybe you are being overly critical? But there is so much crap software out there, that's not what I'd assume).

But it sounds like there isn't a lot you can do about it at this job.

Or... is there? We can brainstorm. It's not spending your time doing things your boss has told you you should not spend time on, though, to try to save the company from itself. That won't be good for your career, or your personal stress level, and also won't help. Whatever it is will about trying to pitch your boss/team on new processes.

> What keeps bugs from happening is processes, not programmer conscientiousness

While maybe it's not sustainable, or maybe it doesn't work for large companies, if you're at a startup or small company, conscientiousness absolutely plays a large role. That's not to say that process cannot help reduce the mental burden of developing though by reducing the need to be conscientious.

I mean, sure, you can try to write good code or not give a shit, and if you do one or the other will certainly matter to the outcome, agreed.

But I (who has worked my entire decade+ career on very small teams/companies/projects) think that, past a certain point of not being just apathetic/irresponsible, "trying harder not to have bugs when I write the code the first time" has much much less effect on outcome than code review (including even code review by yourself, but giving yourself the time to do that, including some resting time in between when you write and review your own code), test suite, QA, error monitoring (with allowance to spend time diagnosing and fixing any errors reported; actually just having that allowance in general, otherwise bugs keep building on bugs), etc.

Plus if you think bugs are a result of "not trying hard enough" or "not focusing hard enough", you just stress yourself out. Even more so on a small (or especially solo) team. There are ways to create high-quality low-bug software without stressing yourself out. I don't think they include a model where bugs are your fault for not trying hard enough.

But sure, everyone should try to write good code, and if you don't care about what you are doing and don't even try, you will have more bugs, it's true.

Breathe. It's just software, we're not saving babies here. https://www.hanselman.com/blog/software-and-saving-babies

Some people write software that saves (or can take) lives. Don't take one of those jobs if you're stressed about bugs in your web app.

Yes. Having worked on systems that have the ability to directly or indirectly have a profound impact on real people's lives, it can be a big deal. The hardest part in those jobs was always the newer folks who are used to a more standard release early & often, iterate on your bugs, etc model.

"Now, to be clear, if you ARE saving babies or working on software that does, for crying out loud, don't breathe and make sure you've got unit tests!"

I would hope that there are end-to-end 'did it save the baby' tests!

" a customer recently complain about a bug, which we fixed in a few hours, but the next week they had to complain about the bug again because it was back."

Don't feel bad about the bug. Bugs happen. But the company should feel bad about the process that let it come back.

Can the process be fixed? That should be top priority.

We have blameless postmortems after major incidents like "the app is completely down." The action items aren't always followed, even easy ones like "new PRs should not use X function, they should instead use Y function." Two days later a PR will be approved and merged that uses X function. So then I'll make a meeting a month later saying "here are all the action items from the last couple postmortems that I don't think we're following" and everyone will be in full agreement again, but still nothing changed.

So given that, I don't feel very comfortable pushing hard for any process changes. I already feel like I'm bordering on annoying.

The action item isn't easy if everyone has to individually remember it going forward. An easy action item would be to add a lint rule to prevent the X function from getting checked in, even if the author never heard of this rule before.

Well, I guess your options are:

* Trying to figure out a way to improve processes at your current job. (There might be a way that isn't or doesn't feel like "pushing hard". OR, your current job might just be completely uninterested and this is never going to happen).

* Finding a new job. (The job market is as good for software engineers right now as it ever has been for anyone ever, pretty much)

* Deciding to live with it

Oh. OK, time for a new job with a company that can learn from mistakes.

(Alternatively. you can try to be the force that turns this ship around -- but I don't recommend it.)

Look for a different job with a better culture.

Blameless company culture can help on this a lot. FB had “move fast and break things” so people would not feel bad about breaking things and instead would focus on fixing them. Other companies may have improved versions of this culture.

Think of this in contrast to working at a place where, if there is a production bug the lead’s first instinct once finding said bug is to git blame who did it.

Leadership should be encouraging breakages as long as there is a review of the bug, remediation, and a fix put in place to prevent it from happening again. That should be rewarded in any performance reviews.

> The wise programmer is told about Tao and follows it. The average programmer is told about Tao and searches for it. The foolish programmer is told about Tao and laughs at it. If it were not for laughter, there would be no Tao. The highest sounds are hardest to hear. Going forward is a way to retreat. Great talent shows itself late in life. Even a perfect program still has bugs.

- The Tao of Programming: https://www.mit.edu/~xela/tao.html

Don't focus on making it bug-free. That will never happen. Structure things so that bugs don't have big impacts. Auto-save, local backups are a good band-aid, better if you can rearrange things so all the data isn't in one "basket". Think about failures outside of your control -- power outages, network outages, etc. Will people lose work? Don't make it hard to do destructive features, but instead make it easy to undo them. Often it's useful to store data in a redundant way -- sort of caching things to make things faster or easier, and these can be used to restore data when it's needed. Think very carefully about which data can be recreated and which is primary, and treat them differently.

People are going to complain about everything. If they don't complain, that means they are not actually using it. That's ok. Listen to them, listen for patterns, take them seriously but don't take it personally.

It sounds like you're doing the right thing -- when you find a bug, fix it and don't just put in a quick workaround, and learn from it and try to avoid that from happening again.

Many commentors have given you advice at higher level.

On a tactical level,

First work with your development team on setting up temporary stores(autosaves) workflows either at the browser/ app level and/or in the backend with redis/Kafka etc.

If your users can think they can safely refresh and un submitted forms would be there , then they won't worry that much.

Secondly from a devops point of view, there is a lot of things you can do improve tooling for your teams. Canarying deployments, CI/CD automation, performance monitoring (APM) robust traceability and observablity, better error tracking and deeper data captures(session /request replays), to help solve bugs faster, production like environments with good mock data, failure point simulations with network etc, isolated dev environments (virtual clusters) . Ability to run code in a PR etc. [1]

The idea is you catch bugs before it goes to users or before the user notices.

----

From a biz perspective your boss will not change his view, even with some bugs you are better than the market and decision makers in companies buy a product for features, not because it is bug free , they don't care because the buyer is not the user. Your boss will only start caring if you start losing customers over bugs.

There is no right answer, many times your boss is right, sometimes not fixing bugs can explode technical debt and slow your growth or kill the product too, very hard to know that balance.

---

[1] these are things devops could do without product engg involvement for the most part, writing units tests or fixing architecture is not a practical initiative for you to take up although it may pay off more .

I think this feeling goes away over time as you gain more experience and see how others are doing in the industry. There's a never ending tension between not breaking prod (having bugs) and shipping. In most places, you can't stop shipping, so you have to find how much stability you're willing to sacrifice for your feature work. Then you need to accept the casualties as being part of the battle you're waging. Of course it's better if there's no casualties, and it's better if you minimize the risk of casualties, but if you stick you're nose out there, you need to accept the risk and live with it.

With experience, you can learn ways to minimize the risk you take when you ship, by writing code in better ways, by organizing things with more backstops, handrails, fail-safe patterns, but it's always at the detriment of shipping. But the better you get, the lesser the impact of the safety tax will be.

The best thing you can do is to try and get your leadership to align with you on the customer pain. It sounds like you're feeling a ton of empathy for the client (good) but not your boss/leadership. One way of doing this is to get customer feedback, either by email, or by tickets. Then aggregate that into a document and show it to your org/boss/bosses boss.

If you can't convince them, please don't kill yourself over it. Remember that its just a job. Things break in business processes all the time. The best organizations have a good process to prioritize fixing such issues and ensuring that they happen less frequently. But single handedly trying to change your org is difficult if you don't have the backing of someone powerful.

I get where you're coming from. My first job out of Uni was in a start up, and I know I felt much the same as you describe here, with the added pressure of bug fixes being harder to distribute (since connectivity to the devices we worked on could be spotty).

The thing you have to do is just keep trying to improve your process. Beef up your test suite, set up automated regression testing, start reaching out to customers or combing through other forms of feedback. You can't stop bugs from creeping in, we're all human and it happens. It's totally fine! But you can put your best foot forward and give it your best shot.

You've got this bud.

The thing about enterprise software is that sales are the hardest thing. Even if you had the perfect product already, each new customer wants their specific customizations before they sign that big cheque. This will not change regardless of how many features you have or how many customers are currently happy with the product. What should change is that the effort of making customizations should become easier as they fall into categories that you've done before.

As you company grows, eventually you may have the luxury of core product/feature developers and support engineers that adapt the software filling in those customer requested features. Until then priorities have to be balanced. But it shouldn't be your responsibility. If a particular issue needs to be raised because it doesn't seem to be getting sufficient attention then you can raise it and make as clear an argument for it as you can.

This shouldn't be for every one, and it shouldn't take all your energy. Things like you shouldn't be in direct communication with customers. There should be a designated support person(s) with understanding of the product use but not its implementation to record the issues that can be triaged, grouped, prioritized by a PM, and scheduled to be worked on.

Beyond that, it's common that developers will do incremental refactors or bug fixes (as time pressures permit) along the way when working on areas of the product.

If developers are solely responsible for finding bugs, your company is doing it wrong. Don't feel bad, our company is also doing it wrong; not having a QA department is a rampant cost saving measure.

I'm not sure I would try to feel less guilty about having bugs, that guilt helps you be conscious about finding them before going to production. Keep in mind it should never be solely your responsibility. If you don't have a QA team, you should feel less guilty, because the biz doesn't think it's important to hire at least one QA person. If you don't do user beta testing in your cycle, you should feel less guilty, because the biz doesn't think it's important to establish that communication relationship.

I work as an internal dev environment now, but I have worked at several software companies that sold software to other companies, and they were pretty mission critical to the clients. We had a dedicated and talented QA department, we had automated testing, and we had a beta cycle with a few choice companies to help us find bugs. Bugs still happened but we took measures to make sure they were minimal if they ever got to a release.

Try to minimize the impact of a bug. In the example of the user losing an hour's worth of work, perhaps save periodically. Also, any bug you squash, particularly in production, you should make a script and make sure that gets tested every time. Unit tests are great for this.

Definitely keep being a perfectionist and honor your craft, it will pay in spades throughout your career, especially once you get older.

> I feel guilty when our customers have repeated bugginess in our app.

You might be feeling over-responsible for the bugginess in your app. If you did your job perfectly, which forces in the environment around you would still cause you to ship bugs?

In your shoes, I would still want to do everything in my power to limit the bugs that I ship---that's why I practise TDD, for example---but I look carefully for the boundary between what lies within my control/authority and what doesn't, then I refuse responsibility for what doesn't lie within my control/authority. And that was a more-than-a-decade-long change in my mindset.

Not everyone likes TDD; I don't mind. I have a system that helps catch my mistakes sooner, so that I limit the associated damage. I have a very effective system, which I trust, and which therefore allows me to work aggressively: if I can recover from failures inexpensively, then I don't need to agonize over failures. If I drive the cost of failure down close enough to 0, then I don't need as urgently to limit the probability of failure.

What can you do in your work that might help you limit the cost of failure? If you had that, maybe you'd feel less guilty about making mistakes, because you'd done significant work to limit the cost of those mistakes.

Good luck.

Polish is great, but it sounds like your customers - enterprise users with many hours in your app - are of the sort that would benefit from an ability to help themselves. You're not in B2C, you're in B2B. Add features (because that's what your boss is measuring) that give your users enough rope to hang themselves, as they say.

In the event of a crash, they are likely able to choose to either start over and drop the last 4 hours of work, or to load state from autosave-20220507.1300.xml and retrace their steps from the last 5 minutes. Or improve your logging until it's user-accessible, with the goal of giving them access to a file they can read and figure out how to reproduce the crash (and avoid those circumstances in the future) or send to you to help debugging.

Absolutely give them an API: Your customers are literally offering to help you do your job for you! If you can swing it so their plugins run serverside and you have a license to the macros they're building, you'll soon see exactly what they need and be able to anticipate those features for other customers. If your product requires special features and you're being told by your customers that they can't convince management to buy it until it supports feature Z, give them the tools to build feature Z so the answer to "Can you integrate with our ERP?" is not "we could add that feature after we finish integrating with our other customer's ERP, talk to you again in 3 weeks" but "Does that ERP support XML? JSON? CSV? SQL? Sockets? Regardless, we can help you write a simple plugin to accommodate whatever specific needs you might have."

> We have even had a customer recently complain about a bug, which we fixed in a few hours, but the next week they had to complain about the bug again because it was back.

An interaction like this was the genesis of me spending a lot more time trying to understand code via the commit history instead of just looking at what's in front of me. That in turn made me highly opinionated about bad commit hygiene practices.

There's a class of regressions that is caused by interaction between two features. As long as one exists, the other does not work. Until someone sits down with a three way merge of sorts, and finds a new solution that allows both features to work at the same time, you're just going to ping-pong between two open bug reports. The only way I know to get past this is to figure out what both developers were trying to do, for which the commit history is often the cheapest option.

So we do things like require bug numbers in commit messages, pester people to learn to do move operations the 'right way', push back on squash operations, separate formatting changes from functional changes, use revert operations to remove bad code, and so forth. Like any safety procedure, they seem like a waste of time and energy until something bad happens.

> you're just going to ping-pong between two open bug reports.

I guess you don't have an automated test suite that would prevent you from committing code that causes a previously fixed bug to come back? (Ie, the real meaning of "regression")

Peer reviews really help develop a sense that "we're all in this together." The work is then the product of the team collectively and depersonalizes any problems that arise. It's similar to a QA team having your back ("...if it got through a whole QA team then it must have been a tough one..."), but even better because defects are caught earlier and everyone learns from reading each other's code.

You're a developer? The best you can do is to write code with as few bugs as possible. Don't stress about the other stuff.

Product-wise: bugs are a part of life, but they shouldn't destroy any data. Whoever is in charge of the roadmap should ensure that this is priority number one. You're not responsible for the roadmap, though. Raise it to your boss once, any more emotional investment in these things is a waste.

There is no software without bugs.

I had a Google spreadsheet that I put together two weeks ago. Used some custom code in "App Script", a few very simple functions that made some of the spreadsheet formulas much easier.

A week later they stopped working. Google app script was just not working for a lot of people. It could have been embarrasing because I had shared this spreadsheet with several hundred people to communicate some real-time information, and now it was not working. But, that's software. It always lets you down eventually, and never at a convenient time.

Here's the specific issue, still unresolved after many days. This is Google. Google. If their software is buggy, what hope does your small startup have?

https://issuetracker.google.com/issues/222342097

It's part of the reason that I hope to as much as possible evict computers, phones, and the internet from my life when I retire. I feel like Charlie Brown kicking the football over and over, thinking that this time she won't pull the ball away.

Why is it something that you feel needs fixing? Caring deeply about something you care about is a good thing, it shows pride in your work/craft. Sure, nothing and nobody is ever 100% perfect 100% of the time but none the less I can assure you that people in other fields often feel the same way about the same things (bugs, faults, design errors, something missed in an audit etc).

In my opinion you are showing the attributes of a great employee and with your attitude I would hire someone like yourself in a heartbeat.

I have come across both those that care deeply and those that shrug off errors as 'meh, shit happens' and I can honestly state that the good ones cared while the ones that had the 'shrug - so what' attitudes were mediocre (at best). In one of my more uncharitable and judgemental moments it struck me that their attitude of not caring was because they had so much prior practice at dealing with shit-that-went-wrong issues that it was nothing new.

The reason I think it needs fixing (and the reason I created this thread) is because _I know_ every product has bugs.

Suppose I invited you to a party and I gave you directions but they were so bad you were lost for an hour. I would feel bad about that, even if you enjoyed the party. But that's just one person. If I discover a bug that causes one user each month to lose an hour of work, I'd feel bad and try to get the team to fix it. And then I find out in a couple months that because we have more users it's now affecting 100 people? I can't keep feeling guilty forever.

So I'm sort of asking how do I make peace with that and maybe even feel happy about the work even though it occasionally pisses people off, and as our user base grows the number of people who are pissed off will increase.

But remember (actually remember to your founder/CTO/whatever) that the moment enough people get pissed off, growth will stall, particularly in a niche market, word of mouth is relevant.

Before being allowed to have (too many) pissed off customers you (your app) needs to be unreplaceable (or have much worse or no competitors).

If on the other hand you would expect to have no pissed off customers, you will need to reduce your expectations.

Quality assurance is a craft in it self which takes experts years to master. As a devops your expertise lies elsewhere. When a bug of yours makes it to production that is a failure of the QA process but not yours personally. If you are expected to do your own QA then just know that QA is not your expertise and you are bound to make mistakes.

If this keeps happening perhaps you can try to persuade your bosses to hire a qualified QA expert. If they don’t want to then perhaps quality is not as much of a priority for upper management as it is for you, and you can free your self of the blame. Sacrificing quality is sometimes a reasonable choice for a company with limited resources, and the bugs that you put to production might be a result of that choice by the upper management.

I will give my personal experience as an experienced engineer in FAANG - while trying to be very pragmatic from resources pov.

Do this in the given order: 1. Add monitoring wherever you can. Not just the exceptions, but latencies, handled success/fail statuses, user flows from clicking on x to saving data on y etc. Create meaningful dashboards & alerts on them. 2. Add release canary process. If you have sufficient traffic, do an A/B test roll out. Have load balancer send 10% of traffic to new binary, 10% to old binary (and make it user sticky - so same user doesn't get bounced between different binaries) . You'd need to be mindful of caching (so version static assets etc). 3. Add tests to critical workflows. This part is usually the hardest to do and pay off is usually the longest but it's critical. However, only do this if your user flows are more or less settled and changes are limited.

Also - before doing 3 - it'd be useful to identify the kind of bugs you see - is it client side bugs or server side bugs? Client side bugs are a pain in the ass overall as UX changes can render tests useless, but try to test critical components using a realistic user flow. Also - as each bug shows up, definitely add a regression test for it (which would have caught some of your bugs that you said "repeat").

Remember - you need to adopt "swiss cheese model"[A] for catching bugs. Each layer will catch something with (1) being a catch all, and the more layers you have the better - but only over time.

The guilt goes away once you start catching bugs before client notices them, or as soon as the bug happens. And as you do (3) and (2) your confidence will improve.

Bugs are fact of human life - no human is perfect, and software is complex. The more complex it is from user interaction pov, the more likelyhood of a bug. Don't sweat, keep doing, keep improving over mid-term and long term, and they'll happen less.

[A] https://en.wikipedia.org/wiki/Swiss_cheese_model

Everyone makes mistakes. If you feel guilty about writing bugs, you will feel guilty all the time.

It's a good thing to want to develop a good product, and it's good to take pride in your work. But if your goal is 0 bugs, it's just not possible.

If a bug happens because you didn't test, you should feel guilty.

Do everything you're supposed to do as a software engineer; then you won't feel guilty about the remaining bugs that slip through.

Also, don't feel too guilty if you break something which someone else developed that wasn't (1) documented and (2) covered by regression tests to catch the breakage.

When users report a bug, don't feel guilty if they are relying on some undocumented behavior, and it's not working they way they imagine. (Of course, security, reliability and robustness to bad inputs shouldn't have to be documented; that's not what my remark here is about.)

Lots to unpack, here.

How old is the company? There's certainly a stage where features are critical, but there's also one where you have 80% coverage, and it's time to harden them and clarify the product and feature set?

What does customer churn look like? That'll tell you how bad the problem really is.

The alerting situation sounds bad. Sometimes it can be that they're mostly false alarms, in which case people will ignore them. The fix there is spending a few days tuning the thresholds and cleaning up unmeaningful alerts. This is also an opportunity to drive this. If people are ignoring real alerts, that's bad, and it's harder to fix. What would happen in an actual outage?

You're at a startup. Have you talked with anyone on the sales side about what they're hearing? Is it hurting sales or retention? Look for people with titles like account manager, account executive, solutions engineer, etc.

How small is the startup? If it's less than 150 people, the signs actually point to it being a somewhat real issue, and you don't desperately need the job, you can book a meeting with someone high up. CEOs at startups have open door policies for things like this that slip through the cracks. Just do your homework beforehand, only do this rarely, and be ready for it to possibly backfire (but if it does, you don't want to work there, anyway). Keep in mind that all codebases are shit, especially the one you're currently working on.

If things are bad enough that it hurts sales or retention, and there's no push from sales or drive in engineering to fix things. I'd leave. Remember that part of your compensation is equity, and the company's success depends on the success of the customers.

Another way of framing issues only happening 1% of the time is you only get beat up with a baseball bat on your way to the office twice per year. One place I worked had a known issue we were very unhappy with where the p95 latency was .5s, p99 was 1s, and p999 was 10s. People feel p99+ more, and it's usually better to track these metrics on the high-end.

And the obligatory "it's a feature, not a bug."

From a human perspective, you are caring for your customers, which are the people who care the most about your product - hey, they use it all day. I think you are amazing for caring so deeply, and I admire it.

From a business perspective, your company has finite resources. It has to decide to invest in getting new customers or in retaining existing ones. This decision would depend on several things such as churn rate of current customers, potential customers, value of locking in more customers up front.

Your concerns are valid. Keep empathizing with your customers. But also be aware of the big picture of the company from a business perspective.

I don't feel guilty about bugs but I feel super guilty about not having tests.

I think this is ultimately the healthiest balance. Bugs will happen, you shouldn't feel guilty unless you didn't even try your best to avoid them by at least having automated tests.

That was the first thing I thought of too! Sadly, I had to read to the bottom of the comments to find yours. It should be the top one for sure.

Bugs are a part of life and I think it's safe to say that we don't intentionally put Bugs on our code. Therefore, while your guilt is justified, I don't think you should let yourself be overwhelmed from it. The best thing you can do is to double, and triple check that you don't lose data because even if users face down-times, they'll be mostly fine. But the same thing can't be said for data loss.

Apart from ensuring that all parts of the code are covered via appropriate tests, and being responsive to the customers' complaints, there isn't much you can do.

Developer test. I always put a task to developer test my changes. This is beyond any Unit Tests I might write. This allows me to go through each COA and make sure I have requirements complete. Work with the understanding that QA are not there to find bugs. They're their to Assure Quality. Once you change your mindset, and your development practices, you won't have as many bugs and the ones you have, you won't feel bad about because you know you tried.

Libraries. Keep libraries of code that you have used in the past that you know work and have been code reviewed.

Doing the best you can has a ceiling, my friend!

You sound like a very thoughtful and caring person. These are qualities that can make you an excellent leader and an excellent ambassador for the good of your customers. Trust in yourself .

Maybe consider starting your own business if you feel you’re not able to utilize your skill set best for the customers. That might not be best though, I don’t know your situation. It sounds like a confidence issue, and you sound like a younger professional getting started. Keep at it, and take good care of yourself.

Feeling guilty about bugs sounds appropriate.

In the sense that you care about the product.

Some people don't, they're there for the paycheck and don't uphold their end of the employment bargain. Or justify it by saying they're "too busy".

Some people might not litter on purpose, but occasionally something falls out of their car or bag, and they say, "Oops. Can't be bothered. It goes against the principle of leaving things better than how you found it.

It helps me to think of my code/systems as livestock, not pets. I don't get emotionally invested in my work, or tie it to my self worth. Highly recommend.

So some of what you describe is just the way of the things, but others are organizational failings.

First, while "features first" may be the right mindset when in growth mode (I can't say for certain), retention matters too, and the org needs a plan to address that. Retention tends to give better ROI than acquisition, but is very easy to overlook. I worked on a product once that only focused on acquisition, not retention; that product is no more. I currently work on a mature product; I had to set draw some lines in the sand, as it were, so product didn't keep my team 100% focused on features, but instead left us room for bug fixes and tech debt. There will always be pressure to do features > all else, and organizations need to have ways of counterbalancing that. So, this is an issue that is 100% outside of you, and outside of your control, though you can advocate for it (but you are not responsible for convincing people).

Second, in the comments here I saw you mention a regression. Generally speaking, regressions shouldn't happen - work to change the culture so that if a bug is fixed, a test is written. You know this is a way the app can break, you know that it has been broken this way before; it is worth asserting that it behaves correctly for future builds. This is an issue with dev culture that you can advocate for, and likely also adopt personally.

Third, recognize what you can control and what you can't. Do a "good job" within the confines of the former, and learn to distance yourself from the latter. This is a purely personal issue, and will be an area to work on regardless of the job as there will always be things that 'should' be better. There is nothing wrong with wanting to feel proud of your work, of your company, etc. However, there is, pragmatically, an issue with tying your personal worth and sense of happiness to things you can't control, and it's worth working on divorcing yourself from that.

Fourth, and as a counter balance to the first, work to get more business experience. You'll come to recognize that non-optimal customer experiences are sometimes the cost of doing business. Finding the right balance of what to fix is non-trivial, but ultimately with a finite amount of dev capacity, you still have to prioritize and tackle things in some kind of order. So, yes, learning to accept that that will leave things being undone for a time is worthwhile for you as well.

1. Accept the fact that no software is perfect and everything has big/small bugs in it.

2. Not every bug is critical, so there will be bug which don't need to be fixed and never will be fixed and that's okay.

3. Have a process you trust and improve which is targeting those critical bugs. For example, put a couple of bugs on every sprint. This way you can be sure sooner or later you will be handling those so you can sleep better :)

You sound like a really smart, engaged employee. I would absolutely not try to 'fix' this in yourself.

Advocating-for customers, fellow employees, technology to solve specific business problems—is the quickest way to progress to senior or staff level engineering positions. Sure you need the engineering chops, but my experience has been that applying those skills in the context of business goals is the quickest way to climb the ladder.

In your case it sounds like you need to be willing to engage with the founders on the field they care about, which is business goals. Their metric of success (downard trend of x) is a good focus for early stage startups. Over time, there will be other metrics that will become important and one of those is going to be customer acquisition channels.

The holy grail of acquisition channels is customer referrals. Do you have NPS surveys at your company (or some other proxy of 'I would love to refer you to other customers')? If you do, I would take a look at what those scores look like, which will help you engage your boss with something like:

- Our NPS score for key customer X is in the shitter, do you think that means we should focus more on bugs so that we don't get a bad reputation? How do we think about NPS vs feature count in a situation like this?

Or if NPS is still good, you have grounds to feel better about the bug situation. Sure it's not good, but you can feel good that you are spending your time in other impactful ways.

If NPS is not a thing yet, advocate for it. One way to do this might be to engage with your boss / founder. I would suggest simply asking how they know when you have met your goal of supporting x number of features. Or if they don't have an answer to that, how they think through when other metrics might become more important. If they go into customer acquisition, marketing, etc, it would be a natural and good time to advocate for NPS or some other form of engagement with your customers so you can discover how to harness them for future growth.

Sorry for the novel. This is basically a long winded way of saying it's not something that needs fixing in you. Business is a big chess game of strategy. If you want to feel better about it, the best way is to continue to engage your boss/founders on the business goals and to help them think more clearly about the tradeoffs you are making as a tech team.

Even at the startup you’re a cog in the machine. Your boss decides how you spend your time. If some of that time is on preventing bugs, and there are things you can do to get better at preventing bugs? Do those things. The rest is not in your control.

Unfortunately the extent to which we can take professional pride in our work is limited by priorities above our pay grade.

It sounds like the company needs to spend some time engineering for failure. One of the first considerations I usually make is what to do when inevitably I bring it all crashing down. In your case it is probably as "simple" as engineering some features so your customers don't lose hours of work, maybe only a few minutes.

What do you mean, guilty? Bugs are unintentional (I hope). People make mistakes, no one is perfect. If employers don't want bugs they need to hire enough people to be able to put all the processes in place to ensure 100% bug free software. Of course, that would never be economical, so here we are.

You caring and feeling bad means you take pride in your work, you are paying the cost of giving a shit but I hope you continue to do so. If you want to grow try and identify recurrent sources of bugs and design processes to mitigate. Again - taking pride in your work is admirable but this is the cost.

1) don't learn from your mistakes; learn not to make mistakes. Look at what others have done; there are a lot of blogs about what when wrong with similar products; make sure you don't repeat other people's mistakes. Take a hard and deep look at the failures to identify and fix the structural reasons they happen

2) increase the effort you spend on tests; write less code; implement simpler features; code review every change. Don't listen to anyone who tells you it is ok to test less; do what will make the company the most money long term; it is better to work somewhere else than to work for someone who doesn't care about product quality. Alpha/Beta test more.

3) implement a checkpoint/rollback system (using git can save you effort) Save config changes to a temp location so it is never lost. Allow test configs to be validated and then allow the user to roll out the new config gradually. Save the old config so users can rollback easily. Implement a monitoring system that can auto detect failed rollouts and trigger rollbacks.

At the first startup I worked for, we focused on making a high quality product, understanding the people who would use our software, solving their problems - users loved us, I loved the company. I worked there for 9 months even after the money ran out, simply because I was so happy, but dev team enthusiasm couldn’t save a lack of sales.

The second startup I worked for, we focused on sales. The salespeople would lie about what features we had, make impossible promises, then stall the customer for a week while the devteam would work overtime to build a flimsy fascade of the desired feature. We focussed on delivering not what our users need, but on what our users’ managers think the users need. Management formally reprimanded me for fixing a bug on a tuesday, because only mondays were for bug fixes (Management didn’t actually want us fixing _any_ bugs for existing customers, just build new facades to trick new customers - but the dev team were so miserable at the state of our codebase - things like “it takes 45 minutes to reload the webapp after each code change” - bugfix mondays were a compromise). I ended up in hospital from the stress of working there. Seems they churn through recent compsci graduates, giving people burnout by age 25 and replacing them with fresh, optimisitic, cheap students. That company is still going strong.

I myself, bailed out of the startup scene, and now work fairly comfortably doing SRE work at a FAANG - it pays the bills, and I have permission to fix the bugs. The only downside is people on HN always blaming me for the downfall of western civilisation :)

Welcome to capitalism, I guess? ¯\_(ツ)_/¯

> They feel that as a startup of our size, our number 1 priority is making sure we have enough features.

Thank you, I'm going to ask about this in interviews from now on. If they say # of features is more important than quality, I'm not taking the gig.

Personally idgaf, it's not my company and it's on them to provide the necessary QA resources. It's a dying field however due to MBA style management only cutting costs and never investing in technical long term sustainability.

How about this: customers, as you put it yourself, begged their bosses to sign up for an expensive plan. That tells me that your software is still better than whatever they had before, even with the (occasional) bugs.

long term, customers vote with their time/money - there will always be bugs, but in general if the software is good enough to warrant the time/money they will keep using it

short term, I used to feel this way as well and would try to figure out how to fix all the things, just to realize it cannot be done. Not to pass the buck but it's the person above you whom probably has more information about whether or not these bugs are going to sink the ship, if the company needs to create less bugs someone will need to prioritize ways to prevent them

Reopens implies you didn't write unit tests for that bug. That's bad and will slow dev not speed it up. If you are ion a feature drive you need to be sure not to break existing features.

What is your code review process? Is every piece of code that gets committed reviewed? If not, that's the lowest hanging fruit IMO.

It sounds like your managers aren't prioritizing stability.

Why should you feel guilty when the CEO or founders don't feel guilty?

Work is just that: work. Live your life to the fullest. Be proud of what you do at work but don't let work consumes you.

Believe it or not the language you use influences the amount of bugs in the system.

Also your programming technique and style of programming influences the amount of bugs.

Generally to reduce bugs, go with a functional programming style and make sure the language has robust type checking. Functional has a bad connotation in some circles so another way to think about it is to use immutable variables as much as possible. Avoid mutation wherever you can.

This is the first step. The next steps are testing and QA, but most teams have that side covered.

This is true, but not nearly so much as FP advocates make it out to be. Once you reach a halfway decent level of competence in any programming language, most of the bugs that you write will end up being ones where you didn't understand the "requirements" correctly, or where you failed to predict some sequence of actions and account for it correctly. Type systems and immutability provide exactly zero help in situations like this.

It's better to focus on clear and simple code styles and architectures. The goal is that when you're debugging the code or explaining it to someone else, you don't want any situations where things don't make sense, or something happens in a weird order for no apparent reason. Immutability can help here, for sure, but FP is a double edged sword. Sometimes it's fantastic but at others it makes your code into an unintelligible mess.

So yeah: Don't go rushing to rewrite your perfectly functional Java/Ruby/Python app across to Haskell or Clojure. It won't help you much at all. Rather, think about how you might use some of the ideas like immutability and function composition, to make your Java/Ruby/Python app easier to work with.

Agreed, but I would go further to say that the type system and immutability helps in the area you describe as well: "or where you failed to predict some sequence of actions and account for it correctly."

Basically immutability serves to create more invariants and reduce complexity of the program such that it is much more predictable.

I'm advocating exactly your conclusion hence the reason why I specified that immutability is the keyword, not monads and other advanced concepts associated with functional programming.

Dog food your own product.

You can uncover annoyances, bugs etc. that you might otherwise have missed or understand the pain of.

Software that has no bugs is no software at all.

To feel less guilty about them: fix them. Everyone makes mistakes, but that's just part of the iterative process. If you fix your bugs as soon as they come up, there is no issue. QA is there fore finding bugs, and if a bug leads to a catastrophic failure, it's certainly not just your fault. Typically a whole chain of safeguards failed.

That said, it is a good idea to look into your bugs, and figure out why they happened in the first place. With this you might find a way that allows you to avoid them. Good luck!

I'm responding _before_ reading the comments, because this is something I feel like I have some fairly strong opinions on.

I see two discrete issues here. I'll post one here, and one in another comment because I'm apparently long-winded today :).

First, as a developer, you _will_ break things. There is absolutely no getting around that - it will happen. You can't choose not to break things. You _can_ choose to learn from it.

I've broken production systems many times, and will break them in the future. I've been writing software for sixteen years now; I break things less often now than I did when I was more junior, but I would say "frequency" is the biggest difference that comes with experience. Instead, I tend not to break things in the same ways.

When I review my work (and the work of teammates) I do so with the critical eye of someone who's seen how things are likely to break. I also look for ways that things are likely to go wrong and pre-plan - to the extent that's possible - how they can be fixed.

Finally, I do everything I can to make it easy to troubleshoot and recover when things (invariably) go wrong. I have a very thorough and "intentional" plan for when things are on fire, and always pause before taking a remediation step.

For example, I once deleted a major project on a scientific publishing platform. The company was very new, and while we had backups, the recovery plan was not exercised and significant work would have been lost if I'd used it. In this case, I was intending to update a single nested attribute on a MongoDB object, but used the wrong operator and replaced the entire document with the single attribute I wanted to update. As soon as I saw what happened, I took a deep breath, and made sure my next action was intentional. My first instinct was to call over the founder and tell them what happened. I estimated that it would take about 5-10 minutes to explain the issue, so I considered my other options. Restoring from backup would take too long, but mounting a backup on another DB was something I could trigger in a few minutes. I sat that aside for the moment as well, but remembered that I should initiate that if I thought I might need it later. Then I looked at what I had in my shell. I scrolled back and found that I had read the document in the Mongo shell a few minutes before performing the update... and the entire document was right there in the output! I took a picture of the screen with my phone. Yes, that's silly, but I wanted to make sure that I didn't misclick or something and completely lose the output. Then I took a screenshot on my MBP, and then I copied the output from the terminal and pasted it into a new text document in a new terminal. I saved that, then opened it in a third shell and verified that it was in fact there. Back to another shell, I built the command I would use to restore the object in Vim. It looked good. I connected to my local database, pasted the command, and executed. It worked! I verified that the data looked like it should, and it did. I then pasted the command into production, visually verified that it was correct, and executed. It worked!

In the above, the Total time from "Oh crap!" to "Whew!" was less than five minutes. As I mentioned, it would have taken me at least that long to call over a more senior person and bring them up to speed. I stood by my decision not to bring others into the emergency immediately for that reason, but that doesn't mean that I was "hiding" anything. Once I saw that it had been fixed, I sent them a Slack message to the effect of "I just broke Project X by overwriting the document in MongoDB. I believe I have resolved it, but please verify it thoroughly ASAP. I'm writing a summary of what happened right now." I then wrote the summary and sent it to him. That summary ended up being the basis for a presentation that I gave at the following all-hands meeting.

The second issue I hear is that you care about your customers, and I think you feel like your company's culture isn't correctly prioritizing quality. There are definitely some steps you can take to improve processing and make these sorts of things less common, but they won't have as big an impact as they could unless everyone "buys in" to them.

Let's talk about the cultural aspect first. You said:

    I have casually asked my boss how they feel about all the random bugs in our product. They feel that as a startup of our size, our number 1 priority is making sure we have enough features. Our prospective customers tell us things like "we can't sign up until you support X number of features," and my boss and the founder believe that our number 1 indication of success is the downward trend of X. I understand what they mean (it doesn't matter how bug-free your code is if you run out of funding and go out of business) but I still feel guilty.

I think you're on the right track here. You're recognizing that the people "above" you in the organization have different motivations, and allowing the possibility that you may just be wrong in how you view this because you are operating with a different perspective on the system as a whole.

That said... "we need to prioritize features over stability" is a huge red flag to me. In my experience it generally means that the people in control of the company are measuring the wrong things; they're prioritizing increasing the effectiveness of your sales funnel by trying to satisfy everyone when they should be either focusing on ARR (annual recurring revenue) by making your existing customers happy or changing their customer acquisition strategy to get potential customers with the "right" problem into the funnel in the first place.

Basically, de-emphasizing stability in order to grow will quickly lead to problems with retention. It's a totally valid choice to do it for short periods, but if you go too far it's very hard to come back.

My boss also is correct when they say that these issues happen less than 1% of the time. I don't feel like I have buy-in to make any changes to this process. Alerts are ignored, and some people don't even have PagerDuty set up, their alerts go nowhere.

    I don't feel like I have buy-in to make any changes to this process. Alerts are ignored, and some people don't even have PagerDuty set up, their alerts go nowhere.

My gut says this is the heart of the issue here. I think you care deeply about the people who use your product, and that you feel like the people around you don't.

If I were you, I would start preparing to leave the company if necessary. Once I was confident that I had a place to go, I'd stretch myself and try to influence the people around me to improve quality and change the focus to sustainable growth. It might work. If not, while it's possible they'd just fire me because of the pushback, it's much more likely that I'd just get to the point where I realized that this wasn't the place for me.