Technical debt and tacking into the wind | Hacker News Reader

Technical debt and tacking into the wind

tedunangst.com

118 pointsGarethX34 comments

Looking back, maybe Fog Creek should have built itself a boat instead of walking up the beach. Except rule number one was never, ever rewrite code.

That is the crux of the problem. Sometime requirements change enough that a total rewrite is the better and cheaper option. Having to support an entirely different operating system is certainly a change of that magnitude.

Obviously, second guessing is easy and shipping is hard, but can anyone tell me coherently why Joel didn't push for a rewrite when he received the requirement to support Unix servers? My perceptions may be inaccurate here, but it seems like the reasonable thing to do when confronted with having to support Unix servers in addition to Windows is to rewrite the code in a cross platform way, and perhaps add shims to support the differences between platforms. I don't think I know anybody personally, whose response would be, "I know, let's write a compiler to translate our VBScript into something that runs on Unix." And in fact, lots of other successful companies (Twitter, Reddit to name two offhand) have successfully embarked on full rewrites, with much less justification than having to support a brand new OS.

It seems to me that Joel was so committed to his "don't rewrite" principle that he was willing to take on a massive amount of technical debt to avoid comprimising on it. Which... well, it worked out well in this case, but I have a hard time seeing it as good general advice to follow when building software.

Well, the task of writing the compiler was given to an intern, so I would guess it was an experiment that ended up working well enough that became their strategy for supporting Unix.

It seems to me, as an outsider, that Joel's reason for avoiding a rewrite at any cost is because so many rewrites have failed. Perhaps it's better to prefer steady, step-by-step progress over a giant leap that might be a leap backward in the end. What is often forgotten in these discussions is that, because Fog Creek had talented programmers, writing Thistle and eventually Wasabi was just another series of steps on the road, not a giant leap as a rewrite would be.

I don't think I know anybody personally, whose response would be, "I know, let's write a compiler to translate our VBScript into something that runs on Unix."

Yeah it was an odd strategy, I agree. But on the other hand, translating one scripting language to another doesn't sound sound that unreasonable either.

From the article:

The super obvious answer is to tell unix customers to install Chilisoft ASP. Boom. Done. Except it costs more than FogBugz does.

I remember hearing of Chilisoft ASP. Never used it, and I have no idea what it cost. I'd be curious if they attempted to negotiate any kind of discount for Chilisoft ASP, maybe locked to FogBugz somehow. Compared to running FogBugz on ASP, which I'm guessing would have required at minimum a Windows Server and a SQL Server license, you'd think some kind of deal could have been arranged.

The other thing I'm curious about is how many Unix installations of FogBugz were sold? From what I've seen, customers who are willing to actually pay for a bug tracker would not blink about buying a Windows server to run it, if that's what it required.

Having listened to nearly all of Spolsky's SO/SE podcasts, I think the other factor is that Spolsky's strategy has been to control critical dependencies by keeping them in house. StackExchange runs on its own servers rather than in the cloud so that problems fall into StackExchange's queue not in with other people's at a cloud computing provider. If FogCreek had made Fogbugz depend on ChilliSoft, then infrastructure problems with Unix integration would have been fixed on ChilliSoft's timeline if they were fixed at all. Writing Wasabi meant that Fogbugz had a more predictable development schedule controlled and instrumented by FogCreek.

> can anyone tell me coherently why Joel didn't push for a rewrite when he received the requirement to support Unix servers?

I probably can: "How much money are we really going to make from Unix servers?"

He almost certainly had people complaining about lack of Unix support. But the question was: "If we expend all this energy supporting Unix, will they pay us enough for it to be worthwhile?"

I suspect that the answer was: "Not really, but we can't bid on enterprise without that support"

So, they had to keep a single, unified codebase to keep the cost to support Unix down while only adding some controlled cost support increase to Windows.

The thing is, for real "enterprise" customers who wanted unix you would just bundle Chilisoft ASP into the quote. They would also probably want to use Oracle for the RDBMS which would dwarf the cost of Chillisoft.

But you still have to do the development and debugging.

I've used lots of closed-source "substitute, but just as good for porting" packages over the years. They NEVER work--sure they get the easy 70%, but they all miss the 30% of edge cases that code somehow always manages to trigger. I am not interested in debugging your porting library so you can make more money, thanks--especially if it is likely to piss off a customer. I will control my customer experience as that is my money.

The only exception to this is if you release your code open-source. Then I might adopt it since I can fix a showstopping bug myself if I need to for a customer.

Here's my litmus test if your closed-source package on Unix might be acceptable: Do you support FreeBSD? My team is smart enough to abstract code in such a way that it can run on Linux/FreeBSD/Solaris without killing ourselves. If your team can't do that, it's just going to get in our way.

Hm, your tone is a bit puzzling. Chilisoft ASP was mentioned in the original article as a "Boom. Done." solution. By all accounts I've seen, it was a complete implementation of ASP. ASP was tiny. There were six objects: Request, Response, Server, Application, Session, and Error. Not a lot of room for undetected edge cases, and if there are, likely not harder to work around than the ones in your own porting library.

Anyway the point is, if you are selling to "enterprise" especially at that time, open-source is a scary, uncertain proposition. Enterprise buyers want vendors with phone numbers and help desks and support agreements and maintenance plans. Your sales guy just says, "Sure, unix, fine, no problem" and bundles Chilisoft into the quote.

> Request, Response, Server, Application, Session, and Error. Not a lot of room for undetected edge cases

Famous last words, son. Like everything, seems simple enough up front, but since it's closed source, you have _no idea_ what's going on in there.

Ayup. "Boom. Done." It just exploded, and your company is now done.

There is no such thing as "Boom. Done." when I have to answer the call when it goes "Boom." Enterprise customers won't allow me to say "Call Chilisoft"--they will yell at ME to fix things. If I can't fix it, I lose the customer.

If it was so easy and small, why doesn't there seem to be an open source version of it? If it was so easy and small, why did Chilisoft sell for almost $100 million?

Open-source wasn't scary at all to enterprise vendors as they didn't even know it was there if the software used a BSD-like license. Only the GPL causes enterprise legal panic like syphilis in a whorehouse.

    > I don't think I know anybody personally, whose response
    > would be, "I know, let's write a compiler to translate
    > our VBScript into something that runs on Unix."

That's not how these things happen in the real world, though.

They happen when somebody says "You know, VBScript looks a lot like PHP. I wonder how much of our code would run with some simple regexes transforms" and gets 40% of the codebase "compiling" with some helper functions. And then someone realizes that regexes don't quite cut it, and actually look here's this VBScript grammar I found online, and it's only two days work to convert the AST to PHP, and 80% of the codebase compiles... And a little while later, you realize that you have a commercially viable alternative to a Big Bang Rewrite.

Isn't the story of Wasabi that changing OS's and dev platforms is manifestly not a good reason for a rewrite?

People seem to be mistaking the big story about Wasabi today. Wasabi worked. Yes, they're migrating away from it --- ten years after introducing it. They defered to 2015 what people on HN are telling them they should have done in 2005.

Not only that, but the "port" to C# was mechanical. They didn't rewrite to get away from Wasabi. Because they put the time in to build Wasabi, all their code was instrumented, and they were able to write a program to do the final port. They deferred the port for a decade, and also mechanized it.

I can't help but think you've taken exactly the opposite message away from the posts today that the authors intended.

There's nothing new here, a company can have a crap tech stack and still build a successful product if it has a competent team and good leadership.

This was I imagine a bit of a PR problem for FogCreek, because at the end of the day they were using Visual Basic to build a Windows-only website, not exactly the most awesome stack in today's world (or yesterday's for that matter).

It's normal that they want to communicate to the world the fact that you won't have to learn their custom version of Visual Basic in order to work there. It's also normal that people aren't giving them a pass that easily.

I think you may have commented on the wrong thread. I can't see what your comment has to do with mine.

I really like the metaphor of walking up the coast because you're getting closer to London. So many projects are like that, some sort of fitness function that tells you that you are getting incrementally closer to your goal, but nothing to tell you that you are fundamentally ill equipped to actually make it to your goal, or even that your entire strategy which is "working" is fundamentally wrong.

They made it to London, though.

Those who don’t learn from history are doomed to repeat it. Any day now, I suspect some fools are going to want to write web apps, but they won’t want to use raw javascript and they’ll create some ridiculous custom language to javascript compiler. I can only hope they have the good sense not to tell anybody about it.

We wrote a custom language, only it's javascript. We built a very simple module/class system that compiles to javascript, in javascript. It consists of functions called 'Module', 'Class', 'Static' and such, that generate javascript code. This has had so much value for us. It allowed us to abstract some patterns into 'compiler extensions' for things like scoping, timeouts and cancellation, etc. And more recently, it allowed us to add typechecking by outputting typescript instead of javascript. I think it was a great idea and would do it again, but it's only been 6 years, so perhaps ask me again in 4 years :)

I was 50/50 on whether or not he was just trying to defend poor decision-making with this post, up until this line:

> It’s worth repeating that what’s generally overlooked is that each decision was made by the team and a variety of alternatives were considered. Moving forward with Wasabi .NET was only done after considering a lot of alternatives, including porting to python.

That right there is the crucial point for me. I think the real danger (and concern) with these types of technical debt situations is that they're often forced on engineering by upline management (i.e. non-coders). But if the folks who will have to maintain any mess they create are the ones making these decisions, the decisions are likely the correct ones-- at least at that time.

well worth reading for the insight into behind the scenes at FogBugz - some depressingly clear headed and awkward discussions undoubtedly had. and one suspects another lesson - don't have your whole company dependant on one product.

I still adhere to the idea that LOC is debt and LOTests is capital.

So, what about the path between "complete rewrite" and "invent a proprietary intermediate language": doing what they did at the end (running a permanent transpilation and committing the result) at the beginning? They could have just moved from straight ASP to straight PHP, and then later done it again to move to straight C#. No rewrite necessary, no technical debt necessary.

Path dependency. At the time, PHP was not as easily deployed on Windows as ASP, and so the original design was set up to give them PHP on *ix and ASP on Windows.

    > A much hipper programmer might say something like “you
    > ain’t gonna need it.”

This is a little bit like saying "a much hipper programmer might describe this approach as being 'AJAX'", a term that is at least 4 years older than YAGNI :-)

FogCreek seems to be winning the game. So I agree with the sentiment that it was productive debt.

Winning what game? I can't find solid numbers about bug-tracker marketshare, but if I type "FogBugz vs" into Google, it throws up Jira as the first result. If I type "Jira vs", FogBuz doesn't show up -- I get Rally, Trello, TFS, Redmine, and Asana first.

FogBugz definitely used to have a lot of mindshare, but...

You have a valid point. But just as an aside, you know who makes Trello, right?

Yes, that would be Trello, Inc

https://trello.com/about

The fact that FC managed to come up with a successful (but not necessarily profitable) product idea and spin it off into its own company doesn't really say much about the validity of their decisions on a completely different product.

Nor is it evidence of Fog Creek "winning the game" (whatever that might mean)

Yeah - Trello Inc. :)

Was Trello written in Wasabi, though?

Is there anything else like Kiln Harmony on the market? The ability for our developers to use Git or Hg (with a couple of caveats...) on any of our repositories is a great recruiting point.

i really dont think anyone would base their choice on whether they used hg or git (or both, or either for that matter). It's a very small, and rather unimportant in the grand scheme of things.

Yes, this doesn't look like winning: https://www.google.com/trends/explore#q=fogbugz%2C%20atlassi...

So maybe Fog Creek was correct to take whichever path cost the fewest resources. If it took fewer hours to write and maintain a small transpiler than to re-write the entire application, then that would help profit margins on a product that is not taking over the world anyway.

Reply on news.ycombinator.com