Hacker News Reader

I hate to depart from the group think, but there's another explanation aside from a smart engineer getting carried away with their own brilliance:

    They remarked how they spent a couple of days looking over the system which was complicated and creaky, they couldn't figure it out

Now it may be the case that they didn't really try to fix it and are just using that as an excuse, but OTOH, they may actually have been stumped despite best efforts.

If the system really did defeat this smart engineer's days of study, then in my opinion, and the opinion of at least one discussion I've seen on HN, the software should have been rewritten. (IMO) this really needs more clarification before scorn is passed.

Indeed. If i cannot understand a system (in my field) in a couple of days (depending on the size), and have been asked to work on it, chance is i'll ask for a rewrite, with documentation and readable code.

If your system is not readable and undocumented, don't blame the guy who will come and decide to rewrite it.

He didn't say this system was unreadable or undocumented, only "complicated and creaky," One particular programmer who was regarded as "smart" couldn't figure out what it did, but it may just have been too complex for him.

So you have a legacy system that is being used by other systems, you're the new super genius and can't understand the system, so you rewrite? Discounting the potential release iterations and bug fixes that are in the legacy system? Your "improved" version can't include that knowledge since you didn't understand the legacy system. Also discounting the company wide knowledge of the legacy system, warts and all that exists? Also having two identical systems isn't fantastic. Rewriting a system because someone can't understand it is rarely the correct solution. It most likely means the rewriter doesn't understand the problems that are being solved by the legacy system either.

And chance is, when you meet the 62k rows of code authentication system that is way too complicated and does way too much, that you will not be permitted to rewrite it in a more sane and modular way.

It is difficult to agree or disagree on the basis of a few paragraphs of anecdote. For example, I have often heard programmers say they don’t understand some code. When I probe, what I discover is that they have figured out what it does, but they disagree with why it is built the way it is built.

So they may explain that making such-and-such a change will take two days instead of a few hours, thanks to all the FactoryFactoryFactoryAPIXMLDescriptorSchemas they have to write. That is obviously a hugely different thing to having no idea how the code works and what they need to do to add or change features.

Or for another example, say instead of understanding a system in a day, it takes a week. The system sucks. But if it is relatively stable, even six days of lost time is trivial compared to the time needed for a rewrite and the risks involved.

I am not saying that this particular rewrite of Google’s XYZ is correct or incorrect, just saying that sometimes we might be right to leave well enough alone. Development time is not free, and in addition to figuring out the ROI of writing a new system, we should consider the opportuinity cost of having Mr. Brilliant rewriting a working but terribly written system vs. inventing something else that might confer a greater benefit to his employer.

I think that if we are going to live in snarky-genius-programmer-land for a little while, it's best for management to set the "guideline of griping" if you will-

- if a programmer who thinks of him/herself and/or is thought of by others as a genius comes to you and bluntly/opaquely describes their confusion over some code, just call them out as an idiot.

Make them realize that if they are going to come and describe a problem they're having, and they're going to spend/waste your time doing it, they better have their ducks lined up and already have details ready. "Oh I understand what the code does, I just don't understand why the coder did it that particular way" is an admission that the previous statement wasn't a qualified one. It's an admission that the programmer had no intention of approaching management with a rational conversation on the topic, just a, "Hey boss I'm going to re-do this whole thing". No one wants conversations like that in a professional environment, people are too busy with actual real problems for that kind of thing.

It takes a more technical manager than most IMHO to properly adjust for that kind of behavior, however. Many managers don't understand the technology as much as those they manage (perhaps it's different in strictly programming environments), and at some point "have to trust" what someone is telling them in shorthand. That is bad in this business. It may work in other industries, but it won't work here. A technical manager needs to be able to challenge a technical idea on it's merits, not on the aura of the challenger.

It doesn't mean coders have to approach managers with a four-page paper rationalizing their need to tie their shoes differently, and as the team gets used to the increased communication and personalities, shorthand understandings relevant to the team will become used. But it should always be placed on the foundation of proof. Trust but verify.

Treat your coders like adults, hold them to that higher burden of proof standard, and hold yourself as a manager to a higher technical standard to understand it, and the entire team can avoid this kind of junk. I come from network administration, and on medium to larger sized teams I've found that at first the smarter guys hate this method because they're used to being left alone to save the day. Being challenged ruffles their feathers in the beginning. After a little while, however, they realize that they're going to earn more respect because everyone around them, or at least their boss, is actually going to understand their brilliance.

The problem is not that the system was rewritten; it may well have needed it. The problem is that the engineer did not replace the existing system. Now they have two systems that do nearly the same thing.

For clarity, the above is what I considered the 'problem' and was what I learned folks who invent new stuff and about interviewing those folks on how they got to the point where they needed to come up with something from whole cloth. I am sure there are horror stories on the other side as well, with systems which haven't been changed but need to be because no one is willing to risk it.

My issue wasn't that they re-implemented the subsystem, it was that they did so without understanding all of what it originally did and so they were guaranteed to burden users of that system with a huge migration task. Folks who were previous users of the old system had to figure out what he had implemented and where their features had gone, and a series of iterative steps as folks educated this person on their particular requirements, and a series of changes, the new system was eventually a nearly complete replacement.

The opportunity cost of those developers who were rewriting their code to talk to the replacement wasn't considered, there wasn't really a solid metric for 'goodness', it had none of the old bugs but it certainly had its share of new ones, it increased the technical debt of the source code base.

So here was a brilliant engineer who did write a crap ton of code and got it working, but the path they took caused a bunch of other engineers a lot of work as well. They did not acknowledge, in any way that I could see, the burden they had imposed.

The converse, fixing a large number of problems with precise surgical changes to the code base, were not things that were being held up as the model for success. That was a sad outcome of that culture, but one I value highly both in myself and in others.

So when someone tells me they re-wrote something from scratch, I'm always curious to see how they got to that conclusion. There are many good reasons to rewrite something, whether it was a prototype that ended up being shipped and kept alive with a hodge-podge of fixes, or circumstance changes where the total code lines would be less re-doing it than amending it. I enjoy figuring out these sort of 'coral reef' architectures, emergent things that were born of the bones of a million quick fixes over the years, and I have blown up a few in my time. Sometimes the reasoning is from external sources (like a system that had inadvertently become tainted with proprietary information).

It is often not possible for the rewritter to change all the uses of the old system.

The things which weren't said explicitly but which we can assume are that the original implementer wasn't around any more, and that nobody else understood the system either.

With these pieces of information, the decision to re-implement seems like a reasonable one.

If just looking at it doesn't help you figure it out, you start writing tests and use test coverage analysis to ensure completeness of your tests. Then you can refactor safely.

I once read the following story about a famous hacker. If I remember right it was about Woz, and recounted in the book Founders At Work.

At some point Apple was taken over by beancounters who wanted reports from programmers saying how many lines of code they'd written. Woz spent a day or two refactoring something, removed 5000 lines of code, and thoughtfully submitted a form with LOC marked "-5000".

They exempted him from having to submit any more forms.

Bill Atkinson: http://www.folklore.org/StoryView.py?story=Negative_2000_Lin...

You wrote that reasonably smart people couldn't figure out the old system, even with a couple of days' investigation. How is this a good system? If people can't figure it out, how are they supposed to use it?

Then you wrote that, when the new system appeared, "nearly everyone" moved to it. Doesn't this suggest that the new system was actually better? Why would most of the old system's clients, for example, pay the price to move to the new system unless they believed it was improved enough to be worth the migration cost?

I'm not saying that your conclusions are wrong, just that from reading your anecdote, it's not clear how to reconcile your claim that the rewrite was an mistake with the evidence that the result of that mistake seems to be preferred by nearly everyone who had the choice to keep using the old system.

I had a working mapreduce job that was 300 lines of fairly high level java code. Compact, terse, expressive, everything you don't see in most Java. I came back over a the weekend and my boss/coworker made it 'configurable' which means he added 800 lines of code.

I fucking cried.

He was like, "I hoisted out some stuff that should have been behind interfaces." ... I didn't need an interface because I only had ONE and specifically made the API have a low surface area so this could occur IF IT NEEDED TO.

I sucked it up and took it in stride.

Jumping in and solving a problem is usually the last thing you want to do, but that character trait(flaw) is what people think of as a good leader. Someone who will take you over a cliff with conviction.

>Compact, terse, expressive

Does 'Compact' mean that you originally did:

if( foo ) while( foo ){ do something }

all on one line and the boss split them up into multiple lines?

I ask because I worked a job where statements were combined on single lines for ease or reading and it was a nightmare.

By compact I mean the domain logic was all in one place and not diluted like a fine mist across the code. You could read it like english and it made sense. Consistency in abstraction levels and not too much genericism.

"Adding interfaces" to improve is when i start to freak out too, also splitting a well-defined piece of code into multiple classes. I call that symptom "knowing object oriented programming principles too well".

'Well-defined' piece of code or SRP-violating?

I've never met a class I didn't want to inherit.

The only time I was ever fired from a job, it was for doing this. I rewrote an entire web crawler stack using parallel event driven programming, yielding a crawler that could crawl the same number of sites they currently needed 10+ EC2 instances for on my laptop without breaking a sweat. I did it in a one-week + weekend maniac code binge.

I was a little peeved, but looking back not so much. I was receiving pressure and what I felt were conflicting orders from the CTO and CEO, and any company that lets someone go for showing initiative is not a place I want to work. I found a more fitting job doing high-speed creative product prototyping (where they want that kind of thing) afterwords, and for more money.

(It was a startup, and I now see it as having had some funny politics going on. I also think that one of the co-founders wanted me gone since I didn't graduate from one of the "right schools." He was a huge school-snob.)

a startup fired you for showing initiative and doing more with less? messed up.

IMHO, I think they really fired me because I one of the co-founders didn't like me because graduated from the University of Cincinnati instead of MIT, Stanford, Harvard, etc.

I figured this out later when I checked their blog and he had prominently mentioned how they like to make sure their candidates are the "right people" from the "right schools." It was a very explicit statement, essentially that being from the "right school" is part of being the "right person."

I encountered this a lot in Boston actually, to the point that I developed a complex about how I couldn't really do anything because I hadn't gone to Harvard. Then I moved away and realized that the rest of the world outside this bubble doesn't care about that.

Whoa, I'm in Cincinnati. Are you still in the area? What are your thoughts on UC nixing the ComSci program?

I did the exact same thing a while back. The company I worked with used 4 fully-loaded servers driving Java spiders while I used 2 y/o laptop and 100 lines of Erlang.

Showed the code to my manager, he left the room saying I should do some "real" sysadmin work and stop playing around.

Just for a test, I started spidering big local news sites with I think 10000 threads and left it running for about 15 minutes.

Two things happened: 1. I consumed all bandwidth, rendering Internet unusable for the whole company, and 2. got about 10% of data other 4 Java servers could gather in a day

This stunt didn't get me fired, but my manager insisted I delete the program I wrote. Nobody was interested in results I got so nothing happened.

But it was fun watching bandwidth usage RRD graphs go through the roof :)

Interesting, the link to your blog in your profile goes to a parked domain (just FYI).

I hear these sorts of tales occasionally and I recognize a manager who is out of their league when comes to keeping someone with your skill set engaged. If crawling the big local news sites was something useful for the company, moving your application to a hosted machine where it had access to more bandwidth would have been a start, of course a demonstration is just that, a demonstration. The next step which is a bit harder might be to crunch the data into some sort of data structure for later analysis.

If this is the kind of problem you enjoy working on and are in the SF Bay area we should talk :-)

Yeah. In general, nobody knows how to write network code. I have seen tons of network apps that perform horribly and use utterly absurd amounts of resources for trivial tasks.

Used to work in a fortune 500 company that would send email notifications to customers telling them their monthly statements were available. Unfortunately, the monkeys who coded the email sending didn't understand the protocol, obviously had not read the RFC. The code would blindly send smtp commands without waiting for a response. One day the smtp server decided to change its behavior due to a dns misconfiguration and many emails were lost.

Similar story here. The company I work for had a system that worked just fine, but apparently lots of things weren't done the right way, documentation was lacking etc... They decided to rewrite the whole thing, incorporate processes and after nearly 18 months of hard work they relaunch the app. That was the last time anything worked properly. Now we are left with a beautiful system that just doesn't work.

Not understanding the system is rarely a good reason to rewrite it. Let me guess, the system they rewrote was Google Groups? Because I remember back in 2006 it was working perfectly. Now their search of Usenet posts can hardly find anything.

Anyhow, I was working on a system once which had a very crappy implementation. Thousands lines of code in each source file. Spaghetti code everywhere. I spent a lot of time fixing bugs in that system. Clearly it wasn't productive. I decided to completely rewrite it. Spent a couple of months working on it, but when it was done, it had a cleaner implementation and far fewer bugs. I noticed that I hardly spend any time on that system any more - an indirect indication that the decision to rewrite was the right one.

Fast forward two years. I moved on to other projects. A new guy comes along (senior than me actually). His task is to make the old system work with new requirements. He comes with an idea - a complete rewrite. Since I worked on that system before, I get to review his design. He explains to me how his new implementation is going to have various "new" features. Then I ask him if he knows that the existing implementation does all that already and more. He had to admit that he have no idea how the existing system works, so he doesn't want to touch it - thence the desire to rewrite.

I gave him my advice but he proceeded with the rewrite anyway. The old code obviously got scrapped. Now, looking back I can't see where his new code ended up. Most likely his rewrite failed, so the new implementation never saw the light of day.

The moral of the story: even if you don't like the old system, you have to understand how it works in order to create something better.

Rewriting might still be faster then understanding an existing implementation.

Nope, because you have to understand in depth the existing work in order to rewrite it properly. Then only good rewrite is when you understand the existing stuff enough to see what is wrong with it. You can't rewrite something if you don't know what it does.

If you don't understand it, how do you know what to write?

You know the know and understand the goals, but you don't understand the internals.

But look at the example cited: Google ended up with TWO systems, because he didn't fully understand what the program did.

Wow. So when you rewrite something, it's perfectly justified. When someone else decides to do the same thing, you twist the argument around on its head. Sounds completely hypocritical, mate.

Rewrite without deep understanding is bad. alexeiz claimed that the original implementation was bad, but still tried to work on it and decided to rewrite it, in the end, to something he considers clean, and as such, a good work base. The next guy did not even try to work with it, even though alexeiz seemed to have clear ideas on how to implement it on his system (which is, two years later, quite a good indicator of the sanity of the thing). You did not read that carefully, did you ?

That's a pretty dim view. Sometimes systems are too complicated, or have swaths of dead code that you spend hours figuring out only to find out they're never referenced anymore.

I'm currently working on a project which was written somewhat poorly with little technical oversight, the result being a minefield of code which is slightly brittle and full of dead codepaths. The old system (which works, is complicated and creaky, and I spent some days figuring out out things worked) needs rewritten and if we weren't pressed for time I'd rewrite it in a heartbeat.

Given a goal, an API spec, expected behavior one can rewrite a system which one doesn't truly understand and still do it better.

Certainly, something built by one person is rarely the best. That being said, what's wrong with occasionally taking out the trash?

In my experience, when you have a system that works, is creaky, and you have to spend days to understand how thing work, there is at best a significant gap between actual and specified behavior. Most of the times, there is actually no spec at all, it is mostly implementation defined. And you rarely have the same team as the one who did the previous team (people left, have to work on other projects, etc...).

That's one of the fundamental issue with complete rewritea: the systems you could most easily rewrite are often the one you don't need to.

Personal confession, I was stuck with the NFS lock manager in SunOS 4.0. I inherited it after it had been written by a mathematician who understood the theory of how it would work, but unfortunately could not write code, and it was then given to a person who could make anything work, even if they didn't understand how it was supposed to work.

The result was exactly as you describe, functions which had clear logical errors but they were never called so it didn't matter, data structures where a third of the members were never populated. It took me a couple of months of back and forth before I felt I could even begin to attack it. And after implementing the kernel half and starting in on the daemon proper, Sun discovered 'divisions' and I found myself documenting like crazy so that I could 'hand off' something that wasn't a 'networking' problem.

I particularly liked the n-level deadlock detection algorithm, that was the experience that taught me that a data structure can represent a large set of potential computations all at once.

> My experience has been that things built by two or three people, working as peers, are often much 'better' (meet the needs better, cover more cases, have a more consistent model) than things built by one person

I concur, but for a different reason than you seem to imply (which seems to be "being conservative with changes and goal-oriented" -- nothing wrong with that on its own though). I noticed that my own coding improved dramatically after I started working with others. This is due to several things: (1) competition and trying to be better than others, (2) once you work with several people, you take extra time to document what you are doing, which occasionally leads to you rethinking how you should be doing what you are doing in the first place, (3) don't know about you, but I am personally far more prone to taking "shortcuts" when working on my own (no one to embarrass myself in front of), (4) last but not least, when working with others, you can learn directly from others.

Exactly the same I think.

I have found that unless I can explain something to someone else, then I must not completely understand it. And working with another forces you to be able to communicate and explain, and that gives you both a better understanding. The result is you find things you missed.

I hired a guy who did that once. Twice, actually, ten years apart. One was right out of the military and one was right out of the university. It would have been the biggest mistake of my life not to have hired each of them. They each created something awesome with no help from anyone except me. They each literally changed the world.

Marvellous anecdote and some incredibly sage advice about interpreteting someone’s evidence of how smart they are and how much they get done. Thank you.

Your asking us to assume that an unspoken goal in that particular software development endeavor was to not grow the total amount of code or libraries or systems, unless absolutely necessary. But was it? What if minimal system growth was a goal, but weighted at 0.5 and just get something running was also a goal, weighted at 0.99, because there was lot of money to be made with a working system.

Actually I don't think I was asking that at all, if you read it that way then I didn't communicate it clearly.

The topic of conversation here is interviewing, and the importance of trying to develop as qualified an understanding of a potential candidate as possible in an interview. Reginald posited the 'mother of all interview questions' around the question of invention and perhaps more subtly the question of building it yourself versus leveraging the work of others.

Its a great question because it starts a lot of interesting discussions around how someone evaluates tasks they are given and allows them to talk about what tools they can bring to bear on those problems.

I related that anecdote from Google because I gave more importance in my mind to 'invention' rather than 'discovery' before I went to work at Google. Google is, in my opinion, the technical equivalent of Mecca for invention. One of the things I learned from my time there was that you could go overboard on invention. Not simply 'not invented here' syndrome, but literally lowering the barrier to creating new systems so far, that the net result is a negative rather than a positive. (Google doesn't see it that way, which I understand too, this is simply an opinion after all).

I practical terms you are absolutely correct that getting something running will have a weight. maintainability will have a weight, and maybe continuity will have a weight too. And if someone ponders those weights, gets some feedback to test their understanding of the relative importance of them, and executes based on that feedback, that is a very good thing indeed.

The person I was listening too at Google either didn't do that, or didn't understand the implications of their choices with respect to the impact on the company, or perhaps the simpler answer was they just didn't think about it. Had they thought it through, I'm sure they would have mentioned it in their talk as the talk was supposed to be about best practices and that would have been a great point to get across.

I think Chuck isn't making the point well in his reply to you, so I'd like to summarize it as I see it.

Ultimately, all software advice cashes out as 'something running right'. (Or 'make money', if you prefer. The following is true regardless of what you see as the ultimate goal.) However, we do not have perfect information about whether arbitrary changes will increase that. Our decisions and estimates are going to be approximate and heuristic.

Chuck has provided an example of how decisions can be suboptimal because employees/smart-people are rewarded for looking good, not for actually increasing the amount of things running right.

One way to correct this bias towards overwriting is to include in the decisions a fudge factor against writing new code - expressed as an anecdote summed up in 'Sometimes they are too smart and do too much'.

Make no mistake about it. Being too quick to rewrite a system is bad. It causes unnecessary work for others and can throw a schedule behind drastically.

That said, I think it's important not to go to the opposite extreme. Personally, I'm in the camp of being too quick to rewrite things myself. And though I've learned to control my gut instinct in this regard, I've also learned that sometimes it's right.

I can name several instances where everyone would have come out better just by rewriting the damn thing up front rather than trying to fix or maintain the old system.

TL;DR - Sometimes discarding your peers' work is the best way to respect them.

I'm surprised no one has suggested refactoring yet. Joel Spolsky wrote most eloquently on resisting the urge to rewrite quite some years ago. Often the best response to a convoluted brittle looking system is to just attack it with a series of small refactorings that clearly preserve the current behavior, while eliminating unnecessary confusion and complexity until you can finally see clearly the bigger picture.

'Working Effectively With Legacy Code' by Mike Feathers is an excellent read for tackling systems like this. Basically a series of down-in-the-trenches strategies for getting legacy code under unit tests - even if you're not sure what it's supposed to do - so you can start refactoring with a safety net.

There is an opposite extreme seen on government software efforts...imagine systems that have been around 15,20,25 years. Interfaces frozen from the days of mainframe, millions upon millions of lines of code, written not necessarily by software engineers, but other engineers.

There's a point where it is really really smart to throw away pieces of that system and regenerate them. You produce a lot of cruft of 20 years.

It's just good engineering practice to ask yourself periodically, why are we building this new component? If the answer ever ceases to be convincing, yell, scream and bitch until your team (btw, you should probably spearhead this effort) can convince you otherwise.

That's really the hardest part of working by yourself; it is very hard to strike the right balance between self-criticism and unwavering focus required to create something, be it code or a work of art.

It seems to primarily be an ego issue. This person must of thought that since he/she is so smart any system that isn't understood obviously shouldn't exist.

One of my new favorite interview questions is: When researching how to integrate a feature, you come across a chunk of code that is atrocious, but unrelated to your feature - what do you do? The answer can be very telling.

My first counter question would be "Do I have access to the original coder?" ;)

I almost fit this category, except when I replace a system, I make it backwards compatible, and put [Obsolete("Please use NewClass.Method instead")] on everything...

I wonder if you would hire me...

one person work is not advisable I have come out with challenging projects with just two people coming with good work, we dont need even three at times..

i like to hit the delete key by 'accident' all the time.