Ask HN: Feeling grossly incompetent working in opensource. What do I do? | Hacker News Reader

Ask HN: Feeling grossly incompetent working in opensource. What do I do?

18 pointsbluedays24 comments

While in college I worked three times as hard as any other student and I felt more prepared than any other student in the school. I built toy languages, and compilers while others were working on understanding basic data types. Now I am here working on an OpenSource project and I have come to realize that the multiple layers of abstraction that exists has made it impossible to change even one line of code without it effecting the entire codebase. I have spent the last three weeks attempting to understand all the moving parts, and feel just as lost as when I was brought into the project. No matter how much documentation and how much code I read I come no closer to making any changes in the codebase whatsoever. I still have two more years of college left, and I am starting to wonder if I have what it takes to become a programmer. Any suggestions would be helpful. thanks.

View on HN

You are doing everything right. Assuming you enjoy the project or product you are working on, just keep at it.

You are lucky enough to be experiencing what it’s like to be part of a real software project. It is not at all unusual to be in this position for weeks or even months depending on the complexity of the code base.

If you somehow manage to continue until graduation, you will have invaluable experience that virtually no other job candidates at your level will have. It will be a massive advantage over your competitors, and when you are asked about your experience you’ll have specific and credible stories to draw on when you talk about how you solved problems.

At the very least, congratulations on sticking to this for as long as you have. This stuff is hard. That’s why they pay us a lot of money for it.

I had a job working on a piece of software that contains bits and pieces written in the 1980s. When I complained about how horribly unproductive I felt, the senior dev on the team just said, "it will take a few years to become productive." It took me a long time to internalize that and appreciate what it meant.

I think it's like becoming an architect spending your career building houses. Houses are small enough that you can reasonably expect to understand every detail of one. And when all you work on is houses, you begin to expect that you'll soon understand the totality of each product you work on.

But then you get a job working on skyscrapers. It's impossible for any single person to understand the entire structure, and you don't appreciate that fact because the concept is still foreign to you. I think a lot of companies have "skyscraper" code bases, and it's not always obvious when you're looking at one. A good hint is probably, any software that's been around for decades and enjoys widespread use is likely to be a skyscraper.

Large codebases can do this to you, but it’s all about finding manageable tasks for you to start understanding the codebase. As the saying goes: change it to understand it.

I like that. Change it to understand it.

> multiple layers of abstraction that exists has made it impossible to change even one line of code without it effecting the entire codebase

Either you have misunderstood the meaning of "abstraction", or the code base has what people call "leaky" abstractions.

The point of abstraction is to create stable internal APIs, irrespective of implementation details. The entire point of abstraction is to create the separation that your code base seems to be lacking.

> No matter how much documentation and how much code I read I come no closer to making any changes in the codebase whatsoever

This approach is necessary, but you won't really understand the code until you start debugging. Can you tell us more about the stack (and, ideally, the project itself) so that we can give you better tips? Sometimes understanding code is a lot easier with the right tools (IDE, debugger, etc.)

> I am starting to wonder if I have what it takes to become a programmer

I'm sure you can do it, but it's also important to figure out if you want to. If you work on multiple code bases, including your own, and you don't enjoy it, or the frustration you're feeling now is not healthy for you, then this might not be the right career for you.

A lot of being a programmer is banging your head against a wall for many hours longer than a typical person would be willing to, until you finally get past an obstacle. Sometimes the obstacle is bad documentation, often it's bad configuration, other times it's a bug.

If the fun parts are not worth the not-fun parts for you, then that sounds like an unpleasant career path.

> The entire point of abstraction is to create the separation that your code base seems to be lacking.

I think this appears to be the problem. The codebase appears to be very kludgey.

> This approach is necessary, but you won't really understand the code until you start debugging. Can you tell us more about the stack (and, ideally, the project itself) so that we can give you better tips? Sometimes understanding code is a lot easier with the right tools (IDE, debugger, etc.)

I think the architecture is essentially one of the reasons I am having so many problems. I'm not familiar with docker/apache solr/their custom database, etc. However, I think that's more of a tangential issue. It's more of a feeling of imposter syndrome as another poster said. For everyone else in the field it seems like this is effortless.

> I'm sure you can do it, but it's also important to figure out if you want to. If you work on multiple code bases, including your own, and you don't enjoy it, or the frustration you're feeling now is not healthy for you, then this might not be the right career for you.

It's not really frustration. So much more as feeling like I will never be good enough at this to be successful in this career.

> A lot of being a programmer is banging your head against a wall for many hours longer than a typical person would be willing to, until you finally get past an obstacle.

This is exactly my personality type, and my wife makes fun of me for you. Once I get my teeth into something I basically don't stop until I got it. It's actually created problems for me in other career fields, but it's seen as an asset for programmers. So I know on a personality level I have what it takes, but yeah... Just get the impression I'm not smart enough for this.

If you know that you don't know something, then you're already one step ahead of the game, because many programmers dive in thinking that they do understand something and don't find out until it's too late that they never did.

Deciphering large codebases takes a LONG time. Months often, to get up to speed. One thing that has always helped me is to take notes. Especially if you start getting lost in a deep set of function calls. Write all that down. Form hypotheses on what parts of the code are doing and try to come up with ways to test those hypotheses.

Soon it will start to become less murky. It just takes a lot of time and perseverence, even for veterans.

> For everyone else in the field it seems like this is effortless

It gets easier the more you experience. I’ve been in the game for around 20 years. I still get caught on sharp edges in code bases. Things still take longer than I originally estimate. Recently, I lost hours on a dumb docker thing because I don’t understand it as much as I would like. One of the code bases I work in from time to time still takes me sometimes days to figure out something that “should be easy,” and I’ve been in that code off and on for literally years. I think you are doing just fine. There are people in this industry who can’t code themselves out of a wet paper bag. You sound like you are already beyond what some professional programmers can do.

Seek help from other contributors to the project. It could just be a shitshow of code that takes hand holding to understand.

Usually understanding a new, large code base comes in at least three phases.

1. At first, they always look like overcomplicated incomprehensible crap. Every little thing that you would have done in way X but was done in way Y initially looks like that.

2. But much later on, after you struggled a lot with making changes to the code base, things start to click. You understand why things were made as they were, what the constraints and requirements were. When you want to make changes, you know where they are supposed to go.

3. Then after much more time, it will look like overcomplicated crap again as you understand that people should have said "no" to the more bizarre of the requests, and that much easier solutions for some of the other things exist.

The original programmers started out somewhere between two and three, and as long as they stay involved, they can always guide people, tell them where their changes go or whether they are ill advised. They know which parts are essential to the architecture and which were never meant as more than temporary crap.

You get problems when none of the original programmers are still involved, and you only get people who have to climb the ladder by themselves. Or only have some people in step 1 left. Then the project is dead.

As others have said, a complex code base can remain opaque for a lot more than three weeks (especially if the documentation is less than flawless).

Also, you said that you have two more years of college yet to go. You are placing a completely unrealistic expectation on yourself. You are very unlikely to be able to, after your sophomore year of college, jump into a big code base and understand it in three weeks, even if you're better than others in your class.

As for impostor syndrome, I'm stealing this from someone, but how about Meta Imposter Syndrome: "All those other people have imposter syndrome, and yet I see how much they're able to do. So they're not like me, the real imposter."

But, in your situation, all those people who are able to get things done in that code base are not like you. I don't know what the code base is, but I strongly suspect that they are out of school, have several years working professionally, and have several years in that code base.

Your problem isn't that you're not smart enough. Your problem is that your expectations for yourself are very unrealistic.

You wrote “This is exactly my personality type, and my wife makes fun of me for you. Once I get my teeth into something I basically don't stop until I got it. It's actually created problems for me in other career fields, but it's seen as an asset for programmers. So I know on a personality level I have what it takes” in another comment.

This is a virtue and a fault. After some successes and some failures, you are likely to learn to be more wise when choosing which problem to deep dive into. The wrong deep dive will damage you.

It also takes time to learn that cowboy duct-tape solutions are sometimes correct: this is best learnt by being in a team with someone who is productive in that manner (the downsides of it are obvious to engineer types; learning the positives of hacks is harder for engineers; judging when to apply fast solutions needs a good engineer). If you can only do anal retentive engineering, then your skills are still super valuable, but perhaps try to veer towards projects where beauty and correctness are valued.

For many engineers, I think it is worthwhile to learn the dark side: how to deliver a quick and dirty compromise that meets a need e.g. most web development. I personally find this difficult: google “yak shaving”. Some engineers never learn how to make good compromises, and they sometimes end up stuck in a dead end.

“but yeah... Just get the impression I'm not smart enough for this.“

Keep your humility, use it to connect with clients and users. They feel it 10x.

Finally I’ll add that the best developers I’ve ever worked with have been technically mediocre. Their other skills have made them great - abilities to concentrate on the right things and not worry about being technical goddesses or gods. I have worked with technically great people, that just do a shit job because they can’t get on with others or they produce something that only they can understand.

Running `git log --reverse` on any overwhelming mature project will give you commits in reverse from "Initial commit" to "Updated README.md" to "Add utils".

Do `git show COMMIT_HASH` for the first ones and you'll see the genesis.

You're looking at the result of thousands of commits, each one having required countless hours of thinking and coding. This can be overwhelming.

Here[0] are some tips that help getting to know a codebase. Some may be irrelevant.

[0]: https://news.ycombinator.com/item?id=19924100

This is extremely helpful, thank you

As everyone has already mentioned it's natural to feel like this and it's just a case of practice.

A concrete suggestion that came to mind is isolate the simplest unit of functionality the project implements and try to implement it yourself fromsscratch using the existing code as a reference; stripping out all the error handling, edge cases etc.

The feasibility of this depends on the project but it can be a useful way to understand both the basics and the rest of the code that seems like spaghetti which is probably due to error handling, edge cases, etc. As an example I implemented part (a simple passwordless connection and select statement) of a database connection library from scratch in c# and it really helped me understand the existing solution better.

Other people have already given a lot of great advice, so I'll just add this bit:

Working on someone else's codebase—especially if it's an open source project with many different contributors over a long period of time—is its own distinct thing that has to be learned. Everyone has to learn how to do it, regardless of where their abilities as a programmer are. If you are adept (as it sounds like you are) at building your own software, then it can feel like the rug has been pulled out from under you ("Am I actually any good at this?") when you try to contribute and simply can't grok what's going on, but if you persist, you will pick it up. Don't take your struggles now as a reflection of your abilities.

Ask for some mentoring from the other participants. Try to focus on one objective that you feel you can own. Now communicate your ambition, progress and problems to others. Collaboration is the key.

If the project is large and old (I hope you're not picking up LLVM or, god help you, GCC?), it can take months to years to feel comfortable within (parts of) its codebase.

The best way to tell whether its more about you or the project is to try several comparable ones, some of the biggest and oldest open source projects are practically impenetrable to outside contributors without years or decades of familiarity, this is a problem since the future of such projects is either to be abandoned as their core retires or become entirely dominated by corporations who sponsor the time for people to get on board.

Everything is an abstraction. Everything is a protocol.

Keep expanding your knowledge.

Read the new pragmatic programmer.

It took me about 5 years to get over my imposter syndrome.

Dont get hung up over one project. I have written a lot of bad code and a lot of clever code. Both in the same project.

Being a good dev is a lot like being a good doctor. Study and practice, sometimes you will get it right, sometimes you won't.

Pain is weakness leaving the body, Feeling stupid is ignorance leaving the mind.

Embrace it. School is designed to give you set pieces that can be understood in isolation, the outside world rairly has that luxury. You won't truly improve without discomfort, though the particulars will be personal and multidimensional.

The end skill is solving problems in the wider world.

Read up on the imposter syndrome. The fact that you're actively trying to work on open source means you're already ahead of the pack. I'd say keep trying and engage with others that are active on the same projects and you'll eventually be able to fill in the gaps.

tldr; don't give up.

Three weeks is not enough time. I've worked on code bases that took me 3 - 6 months to feel like I had enough understanding to start doing substantial work. You should give yourself permission to take the time you need to get a handle on everything.

If you're working on a code base that is this sensitive to change, I recommend you find another project to spend your time on. Choose your battles wisely. Not every hill is worth dying on.

Thank you for to everyone who commented on here. The response was more than I anticipated.

Reply on news.ycombinator.com