How Experienced Developers Can Handicap a Lean Startup | Hacker News Reader

How Experienced Developers Can Handicap a Lean Startup

kevindewalt.com

76 points_pius54 comments

In my past life as an "enterprise" programmer, most people at best paid lip service to doing the "Right Thing." Most code I saw was incredibly bad, written by programmers who knew they would switch groups or companies, or go into management, before they had to seriously maintain it. Programmers I met who worked in an "enterprise" were not into their work. Few of them knew of anything outside of their prescribed Java or sometimes C++, or followed anything new or interesting in their field. I could mention something like The Mythical Man-Month, or talk about looking up an answer in Knuth, and get blank stares. Explaining closures to all but a couple was like talking to a wall. This was at a top-tier company, in one of the most prestigious groups in its IT organization.

I think real experience and talent as a programmer shines in the ability to find a solution which is both correct and takes the same amount of time to write as garbage. This is not always possible, of course, but I would caution against coding without thinking about this trade-off.

In the case of the original article, the author cites the foreign key debate. Truth is, because reasonable and good programmers disagree on which is better, this is not even a good example of quick-and-dirty versus the-right-thing. If anything, it's an example of difficulties in storing data in traditional relational databases.

I guess an even more experienced developer knows when to use his best practices, and when to leave them be for the sake of the project...

Unfortunately I found that foreign keys in my Rails migrations were a constant source of headaches. Moreover, I decided to migrate to Heroku during the project and had to re-write the foreign keys for Postgres instead of MySQL. What did I learn about customers in this process? Nothing.

While I agree with Mr. Dewalt's basic message, identifying integrity constraints as something a "lean startup" shouldn't worry about for a production application containing his customers data undermines his implicit claim of data management competency - experienced database developers know how to design and organize their schema so that details like dbms-specific foreign key implementation are trivial. When foreign keys cause you serious pain relative to your other data management problems it means you're doing something wrong.

What he didn't learn about his customers in the hour at most it should have taken him to get the keys right (thus helping ensure his customers data is not corrupted by application, framework or provider infrastructure bugs) pales in comparison to the lesson he will learn if their data is silently corrupted.

The initial success of MySQL is evidence against your assertion.

When MySQL came out it was the first widely used database that explicitly said that speed of development and operation matters more than correctness. Experienced DBAs were shocked and horrified. Over time the database has matured, however its initial success stands as a testament to the principle that there are a lot of problems (particularly in the web area) where people care that the site does what they want and is responsive, but are tolerant of occasional problems.

Yes, I am aware of all of the possible shortcomings. However requiring people with good ideas to understand more theory before they can get things done does not help motivated newcomers to get interesting things done. And lots of newcomers getting interesting things done leads to successful startups. Which is how a crappy but convenient database (MySQL) got traction in the first place.

For a classic essay on the tradeoff between convenience and correctness, see Worse is Better at http://www.jwz.org/doc/worse-is-better.html.

When MySQL came out it [...] said that speed of development and operation matters more than correctness. [...I]ts initial success stands as a testament to the principle that there are a lot of problems (particularly in the web area) where people [...] are tolerant of occasional problems.

I wonder if this would still be true today? Things are very different from 1994, when MySQL was born.

Back then, the Web was so small it was almost possible to visit every site. Site designers, site developers, and customers were all trying new things constantly and learning what worked. There weren't many expectations because everything was so new.

Today, my guess is that people have a much higher standard for e-commerce. But that's just a guess.

People want to believe that their money is safe. But people are still pretty tolerant if something like twitter has an outage.

What really baffled me about that anecdote is that I wonder what he was "learning about his customers" by migrating hosting providers and database systems. And yet, somehow, foreign keys get all the blame.

Yep. Sure, to survive in the long term, you've got to survive in the short term first. But taking a shortcut that ultimately results in losing a customer's data (or indeed, losing track of their money or their stuff) is a bigger risk than getting your product out of the door a day late(r than you wanted).

It seems like there are two kinds of experience.

One is learning to do the same thing over-and-over again, getting somewhat better at it. This often results in being more rigid.

The other is learning the purpose and strategy behind what you do and being able to alter your behavior accordingly. The often results in being more flexible.

An experienced developer probably has a combination of these two ingredients. Thus there's no guarantee that an old timer will be more flexible or less flexible than a beginner. It's more a question of whether a developer has kept their learning and understanding to be at a high level.

I mean, it's important to realize that the purpose of a full ACID database is to maintain the integrity of enterprise data - key data that is used and re-used by multiple departments. Of course, in a start-up, the ACID requirement can be modified depending on whatever decreased need there is for the properties of ACID. An experienced developer looking from a high level doesn't just recite ACID requirement or glibly toss them out the window but looks at the actual requirements and modifies things accordingly.

I agree.

Good developers work fast and well. They produce less bugs and better code. That's just why you pay them more.

If your experienced developer is slower at doing better, he just doesn't have the proper amount of seniority.

Good developers are computing scientists or software engineers who think before they write code because if you don't understand the process or the concepts or the abstractions involved, your code is going to be shit.

Remember that TDD advocate who tried to write a Sudoku solver using TDD methods. He went up against Peter Norvig but that is irrelevant. Norvig knew how to think about the problem but the advocate did not. So the problem isn't how many lines of some random programming language that you can bang out and how quickly you can fix bugs in it, the problem is do you understand the problem and the solution you are creating?

You seem to be missing the point of the article: producing less bugs and better code can be wrong if writing that code slows down finding out what the problem is. Speed-of-writing-good-code is an orthogonal parameter, here.

No, it's not orthogonal. I can see it in things like the complaints about how foreign keys "caused problems" in the migrations. Generally, that means one of two things: The foreign keys were actually preventing bugs in your migrations (which even in a startup context is a Good Thing(TM), as "buggy migration" is effectively slower than "correct migration" no matter how fast the buggy migration is), or the developer was not experienced enough with foreign-keyed databases to really know how to migrate this.

Bear in mind that if you screw up hard enough there's no guarantee you can recover. Anybody hanging around this site for long enough has read at least a couple of stories about the technical error that brought the whole startup down because there weren't enough resources to recover, and "botched database upgrade" (combined with "failure to backup", which of course occurred because there was no time, we're going too fast and we're too awesome!!! to do the backup) is a prime candidate for that sort of event.

I say the latter because I've been there. Setting up foreign keys is one skill. Learning how to migrate them is another. I've screwed it up before. It does take a bit of learning. But once you learn it, you're still better off using them properly in the first place. There's also some skill in learning how to set up your database so it can be migrated; I'm not even sure how to begin putting it into words, but there's definitely some skill involved there. (I'm not guaranteeing I've got it all figured out, but I've certainly improved, which is enough to establish that it is a skill.)

They're not equivalent either, certainly, but when you say "orthogonal", you're making the claim that they are not just different (which is true), but utterly unrelated, and that claim is unsupportable. I've been deliberately spending my last couple of years learning how to build things both fast and right, and I've gotten to the point where I can spank anybody just "hacking around" after about a two-month window, after which it's all profit. In fact I'm building a system replacing just such an effort right now... well, after I stop posting this anyhow. My system isn't a glorious paragon either because as an experienced developer I know where to cut corners to get it out, but it's a lot stronger than the hack job it's replacing (and I'm doing it in less time, too).

(Part of the reason I started this learning is that I was at a startup where I did in fact make this exact error. In the end, this is not what killed it (what killed it was the management insisting that there was always just one more feature we needed before release, and then once we had that, there was just one more, repeat until out of money with no customers), but I've made this mistake and I won't make it again. But the alternative is not hackhackhackhackhack, the alternative is being more familiar with the cost/benefits ratios and picking the winners, some of which will still be "do it right", and some will not. Hardcoding the answer in either direction is wrong, as it pretty much always is.)

Let's see if you agree with this restatement of your post:

Fast, mediocre coders may be able to iterate really fast in order to get something out in front of users. This means that when you don't know what the problem or the solution is (as discussed in the article), they can be effective at identifying the problem that requires the solution.

But if their crummy code gets used as the nucleus of the solution (once the problem has been identified), the startup can often ensure its own doom.

This means that in successful startups, speed-of-writing-good-code and speed-of-problem-identification are strongly correlated, and therefore cannot be called orthogonal.

I agree with this & think it's a useful refinement to my thinking. Thanks.

Yes. That's actually an important point that I only alluded to. There's a crossover period during which a hackhackhack coder will indeed have "something" out the door faster than me. If your startup fits entirely into that horizon, then, well, don't hire me. Or make it very clear to me that that's what the situation is. Some "startups" do fit into that time period, though as my scare quotes indicate I'm not sure that's really the right word for such an endeavor.

However, I've gotten my crossover period down to about two months on most projects. Some projects even less; a hackhackhack coder who goes for PHP with no frameworks is going to have trouble keeping up with me working in Django. (In this case, I don't intend this as a criticism of PHP, as the sentence indicates.) Of course an experienced developer will reach for some appropriate framework in PHP, but the entire point is that we're comparing experienced v. inexperienced.

(Separately, I don't want to make a separate post for this point, but: There is a tautological element to my argument here, which is that if a putatively experienced developer makes a bad choice on the do-it-right/do-it-fast scale, then by that very fact they have shown they were not an experienced developer. There's a bit of "defining myself the win" there, but I think there's still a point to it. And like I said, I've made that error and am trying not to make it again.)

what does orthogonal mean in this context

Two vectors are orthogonal if the addition of one of them does not affect the "amount" of the other, when using them as basis vectors. In this case, "(1,0)" and "(0,1)" are orthogonal; no matter how far in X you go, the amount of Y-ness doesn't change. (1,1) and (1,-1) are also orthogonal. In the metrics most people learn, this means the vectors are at right angles. (Forgive me this simplification, mathematicians, going further will only obscure things.)

Programmers tend to abuse this term to mean "the two things are totally unrelated", even though the term really only applies to certain mathematical constructs like vectors. Claiming that type-safety is orthogonal to garbage collection would mean the two are entirely separate concerns and that you can freely have none, one, or both, and that the two issues never matter to each other. Now, technically this isn't quite true, which is why I say we "abuse" the term, because the full meaning of "orthogonal" is that you will literally never be considering the effect of one while examining the other, and this is not true; type systems do some impact on GC, for instance. However, in many cases this is "true enough" for someone to say they are "orthogonal".

In this case, I'm not arguing merely that the precise definition is not met, because it pretty much never is, but that the spirit in which programmers use the term is also not applicable here, as the questions of "code quality", "speed", and "developer experience" are indeed tightly interrelated and so answers like "you always have to sacrifice quality for speed" are too simplified.

In simple English, orthogonal means "at right angles". People often say that two things are orthogonal when they mean is that they are different dimensions that are unconnected with each other.

To pick a silly example, the number of coffee cups I have is orthogonal to the number of jars of jelly beans I have.

This is where I disagree.

Trying to be industrial strength for a proof of concept or an experiment is typically a junior's mistake.

Believing that your proof of concept or prototype won't end up in production is another junior's mistake.

I agree this mistake is typical, but I would not be surprised to see a senior make it.

A related mistake is, of course, to keep a pilot project in production (usually as the core of the production project, with a variety of both functional and structural additions to keep it sort of working). This mistake is usually a result of a management decision more than development (but could be the same person).

Why is writing good code slower than writing bad code? I'd rather write good code once, now, rather than bad code now and good code later.

According to my definitions, good code must perform correctly or else fail gracefully. It must also be robust, debuggable, and maintainable. Those extra conditions mean thinking more, and writing more code. That takes extra time over hacking out the quick-and-dirty version.

For example, right now I'm helping a friend out with a project to send email. In about 10 minutes I could whip something out that does this using backticks (i.e., shell escapes), no error checking, no logging, etc. It would be in front of the customer and suitable for feedback very quickly. It would also be short, easy to understand, and would do its job correctly 99% of the time. All of these are good things, but it would still be bad code, because the other 1% of the time it's going to break horribly and someone is going to have to come in and fix it.

Yeah, an experienced startup developer can acknowledge and articulate the tradeoffs between time to market and the perfect design, sometimes delaying some of the design (e.g. perhaps deferring scaling tasks until it's needed) to get things done faster, but at a minimum, making it functional and maintainable.

It's like the army corp of engineers. They can make a bridge for the general public, but they also know how to design something quick (and hopefully sturdy) for troops and supplies during a conflict. It may not be pretty, but it gets the job done when it's needed most.

Articles such as this are what fuck up the computing industry.

If you want software engineers to be recognized as actual engineers they have to conform to a code of ethics and other standards. When the employer says "let's just cut this corner", the engineer says "NO we cannot!" instead of trying to justify and rationalise it.

I dislike the term software engineer and I think computing science would be better and more applicable. A scientist doesn't shy away from his responsibilities and he sometimes tells the truths that hurt.

We already always ship buggy code. That part is done for you. What this article advocates is going further and questioning almost everything that can lead to a proper design.

So basically the author is saying that experience says foreign keys are good, but in practice foreign keys make development and switching data providers harder.

Seems like an experienced developer would have written a data access layer that wasn't so concerned about how the underlying data store managed foreign keys. Almost all senior level developers know that foreign keys are a performance inhibitor and plans for eventual denormalization of data.

I'm wondering if this person is actually an experience developer or not. He starts out saying one of the problems he solved would be to “figure out a way to speed up the data processing”, yet he doesn't understand how foreign keys play into this and had to work around normalization?

I also wonder exactly why it took so long to rewrite the application because of foreign keys. I've never migrated from mysql to postgres but if the process was so time consuming to warrant a blog post like this, I really question the amount of experience this individual really has.

If the purpose of your foreign key is to ensure data integrity, then no amount of denormalizing can help. The point of FKs is to enforce business rules in a central location (e.g. don't ship a part to a non-existent customer, if a customer's address changes then there's only one place to update, etc).

The alternative to FKs is to write your own "middleware" or "business rules engine" and guess what, that doesn't get you to market any quicker either.

My point was only that it seems the author of the post has limited experience with foreign keys.

I am having an experience that is in some ways strikingly similar to the one described.

I want to iterate on my game's level design abilities to choreograph various enemy AIs with detailed scripts - things like "this group will spawn over here and take this path over there, shooting at these intervals."

But figuring out what the best paradigm, the best data format and tooling to help me iterate on the choreography is - is a guess. I can make a bells + whistles solution, but it might not be the right one. But neither is it something that can just wait and becomes obvious at some later date - the technical solutions and the design solutions are intermingled and have to be resolved together. So even though I'm my own customer, I'm still in the "lean" stage of this feature.

So I'm trying to stick hard and fast to "stupidly simple" means of tackling each problem I encounter. Instead of making one all-powerful choreographer system I've started with the stupidest, simplest one I could think of, hedging my bet by allowing multiple systems to coexist, while making the key pieces of data I already know I need, like spawn positions and trigger targets, sit in a different layer where they're easy to edit and interchange.

Essentially, I want to ride on the "80% solution" for a while and then gradually accumulate the rest, and I think the key to making that work is to know where you are uncertain and start fencing off the most suspicious areas so that you can build the eventual shift into the architecture - not modularity so much as "this code will someday become legacy code, so treat as if it already is."

I'm really sleepy right now so this might seem more profound to me than it is.

It's always the developer's fault eh?

Marketing: We need these features now to close $1 mill deal Dev: but... but... we can't implement them in 2 days... we need more time Marketing: Go figure out something...

Marketing: Remember that Y features we asked last month? well, sorry, the customers decided to go with X... Dev: Uh... Uh...

Perhaps once in a while we should switch side: hey management/marketing/sales/CEO, do whatever you need, without modifying our software, to close out deals and to grab huge market share... go figure out something...

And believe me, if your sales and marketing team are any good they'll be sweating blood trying to do that. But it may not be possible. In fact, pounds to pennies it isn't.

(Won't it be a founder selling anyway? I reckon it's a tremendous advantage if your biz dev person/people know one end of Textmate from the other and can do prototype-building spadework, too. It's hard to really get how a job feels if you've never done it. Marketers gotta code and coders gotta market...)

Big, slow thinking sessions ignore this: in an early stage startup you don't know what you're selling. You don't know until you sell it. So you need to find out by showing people endless variations on your core theme, and an iota of effort put into making one which doesn't sell is wasted effort, and wasted effort is death.

You are going to kill nearly all your features. That's code which gets discarded, but it's served it's purpose - working out what the business really is.

I think the title should be "How Enterprise Experienced developers can Handicap a Lean Startup". This is really just limited experience. I once saw a very simple project list used by 20 people turn into an enterprise nightmare with 29 tables and a 3-month development cycle.

Wouldn't it seem to be a mismatch from the very beginning?

In most enterprises, 29 tables and a 3-month development cycle ARE quick hacks.

"The problem with efficiency is that it has nothing to do with whether or not what you are currently doing is the right thing to do."

http://blogrium.wordpress.com/2009/11/05/why-efficiency-is-n...

Hacking together a solution may be more efficient in the short term, but it may also distract you from discovering an effective solution. Diving into a solution before understanding the problem is often a recipe for wasted time. Regardless of whether your solution is a hack or a gem, you're committed to seeing it through. Take time to frame the problem before you commit to taking a stab at it.

Measure twice, cut once. Teak or balsa.

Solving problems TRW (the right way) almost always takes longer than quickly hacking together a solution.

I would rephrase this to:

Thinking about solving problems TRW (the right way) almost always takes longer than thinking about quickly hacking together a solution.

Or graphically, with "t" as "minutes thinking" and "d" as "minutes doing":

  The quick way:

  ttttttdddddddddddddddddddddddddddddddddddddddddd

  The right way:

  ttttttttttttttttttttttttttttttdddddddd

If you want to do something right the first time, thinking about it almost always takes longer. But actually doing it can take less time, often much less time.

When trying to convince my customers to do things the right way, I often tell them, "It's not how soon we get started, it's how soon we finish."

Sure -- but I don't think this is the point of the article. Let's say your ratios are correct, and you want the "t" to "d" ratio to be high in most cases, for shorter overall "t+d".

The article is saying that you often want to get through each of the following lines as fast as possible:

    ttd (release)

    tttdd (release different feature set)

    td (release totally changed and simplified app)

    tttttttttttttdddd (now we understand the problem, do a good job)

Don't forget that some of those final t's are because you already invested a lot of other t's and d's into doing it the wrong way.

Sure, but

ttttttttttttttttttttttttttttttdddddddd

Startups have areal risk of running out of money, just before they complete enough d. However with

ttd,

I have a shot at cashflow with that d, which I can invest back into more (nt +nd)

Established companies dont have that risk so they can do

ttttttttttttttdddddddddddd

While for startups it makes sense to do

ttddttddttddttddttddttdd... etc.

For a minute there I thought you were going to do

  tddtddtdd

I'm not sure what you mean -- that if you hadn't done the first three steps, you wouldn't have needed so many t's in the last?

you already invested a lot of other t's and d's into doing it the wrong way

But the first three steps were an incredibly valuable investment -- now you know the right problem to solve!

(Maybe you mean something like this, just not sure from your comment.)

You're thinking about it from a business development standpoint in addition to a technical stand point.

I'm just talking about the technical approach. You've gained no new insight into what your business is supposed to be, you've just learned a hella wrong way of solving a technical problem. Now your thoughts are cluttered with the effort and you had to spend time unlearning the previous approach.

This is even more true if you make the d stand for debugging.

Agreed - I understand the point of the article and that my post does not directly address that point. But it does challenge OP's statement (which I italicized) that he uses as a basis for his conclusions.

I fully agree that iterative analysis and development is usually quicker than the waterfall method. I'm just adding that a little thought each step of the process can make everything quicker. If you think things through enough (which more experienced people often do better), you can have your cake and eat it too.

When trying to convince my customers to do things the right way, I often tell them, "It's not how soon we get started, it's how soon we finish."

I'm stealing this. :)

In my experience, hacking things is only faster because you rarely try to do things the right way and are more familiar with hacking your way out of problems than you are planning for them. Hacking should be a last resort, when timelines are up and it is down to that or shipping nothing.

> Bugs in the application resulted in garbage in the database that wasn’t discovered until a couple years and terabytes later. One of my colleagues had the joy of spending 6 months as “data garbageman” cleaning up the database.

Cleaning up should be faster.

It's strange that thing A is optional but without doing thing A up front you can get yourself in a mess such hard to clean up. If thing A is optional cleaning up the mess resulting in not doing thing A and having bad luck should be much easier. Db guys tend to overlook interesting problem how to get out of the mess. They just have pie in the sky rules and if you don't use them all and get yourself into trouble then it's your fault so suffer!

I think the difference between the best practices for "lean startups" vs "enterprise" is overstated. The benefits of fast feedback still apply to large projects - the only real difference is that big companies won't necessarily die from inefficiencies like over engineering. And the fact that, over anything but a very short timeframe, the right way is also the fastest way still applies to startups. Sometimes a shortcut is in order - but you can't sustain that for as long as it will usually take to become established.

Real experience teaches you how to choose the right balance.

Warning: unscalable databases and spaghetti code can make your startup fail.

If Ruby on Rails causes nothing but headaches for those who insist on silly frilly things like data integrity, I will theorize that perhaps the problem lies with RoR, not foreign keys.

You probably don't actually need a database at all , given that the integrity of data stored is not paramount. You don't get any points for "We have this data. We have no way of assuring it's correct or is what we want it to be - but we have it!"

On the other hand, given that most start-ups don't bother with things like a business plan (at least one which doesn't end with "Step 3. ????? Step 4. Get bought by Google"), perhaps it's only appropriate that such start-ups do away with other fundamentals of reality, such as data integrity.

The problem here is that the devils are in the details.

Should you use small variable and function names because they are easier to type? Throw away all error-checking because without customers errors don't matter? Or as this article says, perhaps toss out FKs because they might be difficult to refactor?

Trying to make the argument that it's one way or the other is really taking a very complex decision and boiling it down to comic-book level. You need to do enough of The Right Thing in order to make it to your first value inflection point, not just whatever it takes to score a couple of dozen freebie users (although every user is awesome!) Without some idea of where you are in your startup process and what your next value point is, you can't make those kinds of decisions. They're weighted choices.

Look at it this way: assume that you hacked something together doing almost the "wrong" thing every time and ended up 2 months later with a couple of hundred freebie users. Perhaps a dozen or so are even paying for your code, say ten bucks a year. But guess what? Your income won't support the amount of maintenance you have to do on your sloppy code, and customer growth is inadequate to pay for anybody to help you out. Somehow the bills have to be paid, and instead of sales and growth you're spending a large portion of your time as "database garbage man"

So you're stuck with crappy code and not enough resources to maintain the customer base you currently have. It's called "getting upside down" in a project. Not a happy spot to be in. (And it can happen without the lean code problem)

The best scenario is where you make the conscious decision to do a few quality items even at the expense of customer growth for a few days or a week or two and then consciously make the decision not to do a bunch of other things because the hit is too much.

It's complicated.

It's not that complicated. Quality takes time.

As Greenspun said once, if we want software engineers to be treated with respect, we need to force employers to respect a certain minimum standard of quality.

Regarding your example...by releasing something badly designed/programmed into the wild and taking money from customers, you are committing fraud. It needs to be seen this way otherwise we will never be forced to take the time to create quality software.

So if I wrote a really buggy program but you could still use it to make a million bucks in the next week, I would be committing fraud?

Could somebody please defraud me like that? Please?

You're being too literal. What I'm saying is that, at the moment, there is no harmful consequence for buggy programs. Buggy programs that people buy means $$ from the initial sale of the product and your company continues to live and you get paid in $$ to maintain and fix the buggy program.

As long as the software can run, we're rewarded for creating it, no matter its quality.

In our own minds, we have to treat that as fraud and need to feel awful about it.

What I'm saying is that people judge the value of the program on just that -- it's value to them. The quality of construction can certainly affect perceived value but a perfectly-constructed program which provides no value is useless. No, it's worse than useless, because it's an example of craftsmanship-over-function.

All the quality in the world isn't going to make a pointless program worth anything. But a crappily-written program that provides lots of value is a great thing no matter how horribly it was hacked together.

Reply on news.ycombinator.com