Where there’s muck, there’s brass (2007) | Hacker News Reader

Where there’s muck, there’s brass (2007)

joelonsoftware.com

27 pointsTomte15 comments

In the case of FogBugz, the muck is the problem they decided to create for themselves (lots of targets to run on) that justifies solving the non-core business problem (creating Wasabi[1]). Or does it? [2] That kind of thing runs contrary to what I usually recommend and see recommended. It would be interesting to get a recent Spolsky take on it.

[1] https://www.joelonsoftware.com/2006/09/01/wasabi/

[2] https://jacob.jkrall.net/killing-off-wasabi-part-1

Supporting (Windows|Mac|Linux) on (MySql|MS SQL|Access) was a small but meaningful portion of our revenue stream for over a decade.

Having our own compiler meant we could migrate easily to a new platform (.NET/Mono, in our case) with two engineers working full time for only a few months. Should we have spent that time rewriting the app in C# by hand instead? How would we ship new code in the meantime?

This has aged pretty poorly. Turns out there really are a lot of dollar bills lying in the street if you can code at all. Turns out you can make a huge amount of money by doing something easy and just not fucking it up (e.g. Slack). A lot of the time you look at the existing solutions and assume there must be some reason why they're so awful - and a lot of the time there isn't one, the makers really have just been dumb.

Sort of.

The premise "rewards shall pass unto those who solve difficult problems" isn't neutralized by "rewards shall also pass unto the barely competent who can wing an interview in the correct trendy tech stack", or by the fact that solutions to solve everyday things like messaging are sadly so widely pretty mediocre.

Maybe it's wishful thinking but I'd like to think rewards more munificent in nature shall still pass unto those getting stuck right into the super tricky stuff...

Meh. IMO we have bigger problems due to the protestant work ethic and a kind of performative (almost macho) desire to do things the hard way than we have from a lack of raw talent. I hope those with the courage to do something easy and get mocked for it ("three kids and an iguana" indeed) are rewarded.

Google offer(ed?) an in-house search appliance that you could host yourself - they solved many of the configuration problems by selling you the hardware it runs on so they still had control of the whole stack.

Other providers solve (well, limit) the configuration problems by selling an in-house solution with very tightly locked down environment specs and they won't support anything else. If you want it in-house you have to follow their lead.

Solving a gnarly problem doesn't have to mean doing all of the work yourself. Very often a customer will put some effort in to meet you in the middle.

>the whole company is three kids and an iguana

I missed this gem the first time I read this essay. Legitimately lol'ed when I reread just now.

While a fun read Joel on Software always seemed like a what-not-to-do blog to me.

The complexity for the sake of complexity!

No, you don't need to write your own programming language and compiler to deploy a webapp across multiple platforms.

No, you don't.

Whatever the problem is.

Yes, I understand it's complex but really, you don't.

(Even in 2006 when you couldn't just stuff your whole support stack into a container.)

Trust me.

"What I cannot create, I do not understand", as Feynman said. I for one defend people's right to do unusual things. You do not need to write your programming language, but when you do that, you end up with something that changes you, even if the idea of writing your programming language was stupid in hindsight, perhaps especially if the idea of doing whatever you did was stupid in hindsight.

Two thoughts for you to ponder: exercising will is doing precisely what you don't need to do. Also, I feel that programming is a lot like writing. When you read something as a reader, you just read it, but when you read it as a writer, you're thinking about what makes the text flow, or "tick", and the ultimate test of that is whether you can write a similar thing yourself, or perhaps a slight variation.

How would you deploy an ASP web application to Linux hosts in 2005?

Linux and Mac made up a small but meaningful portion of our revenue for over a decade. Would you have foregone that, all because you didn’t want to write a compiler? It took one intern three months to do the first version of Thistle.

You could have chosen any of the cross platform frameworks of the day? PHP or Python would have been an easy choice at the time.

In 2006, the app was already seven years old. We couldn’t go back in time and decide to write it in a different language.

Could have just shipped the customer a server, especially as Joel's article suggests that most of your on-prem customers went with dedicated servers for the software anyway.

Find it hard to imagine that a significant percentage of your customers were 100% linux/mac to the extent that they'd refuse any win boxes in their shop.

I was in IT at the time and catered to a number of 100% mac shops (mostly ad agencies/gfx design studios) and there were always one or two win boxes around, usually a couple for fileserver/backup and one for the sysadmin to do sysadmin things with.

>We couldn’t go back in time and decide to write it in a different language.

You had an intern write a program that rewrote it in a different language for you, time after time after time.

You could have just rewritten it once :)

We didn’t have the margins to ship physical servers to customers. User seat licenses were $100 or so, and most customers were SMBs - 5-15 users, so we would have lost money on shipping any customer a Dell with Windows Server on it. We didn’t have a sales channel for real enterprise customers until quite late in the lifecycle of the product, so even our biggest customers weren’t that big.

Joel said PHP and Python were non-starters on Windows. We could have used the compiler to do a one-time cutover to C# in 2008, but the CodeDom generated code was fairly icky, so we decided to keep the Wasabi code and just ship the machine-generated DLLs with friendly Wasabi debugging symbols. We were only able to cut over once I rewrote the generator (I think that was the fifth generator, and there was also a Wasabi interpreter for compile-time evaluation… PHP, JS, ASP, .NET DLL via CodeDom, plain C#) using Roslyn. That took me just over a month, in two phases: first, use Roslyn to generate duplicate code to the CodeDom backend (2-3 weeks). Then, I deleted the CodeDom backend and started making syntactic improvements (and generate comments and white space!) until I hit the point of diminishing returns (1-2 weeks). Finally we checked in the C# source code, removed the Wasabi build step, and let ReSharper go to town. (2 days)

Each step we took was low-risk, because it didn’t create any new technical debt or accumulate as inventory. Each step was small and low-cost. None of them were the optimal step, but they also didn’t risk the entire business on stopping-the-world.

There were plenty of other disasters in that codebase. For me, the FogBugz Plug-in Architecture was the worst of all of them. Adding all those layers of indirection, plugin hooks was a huge performance cost, and maintenance burden because we couldn’t ever change function signatures once they were shipped. I wish we had just added web hooks. That would have gotten us 95% of the extensibility with 10% of the performance cost, and 0% hosting other peoples’ code in our FogBugz On Demand web server processes. I think I once murdered a fledgling startup by mentioning we were thinking of removing it.

Thanks for the context, I appreciate it. At the time I remember the concept of running PHP on Windows/IIS did have this air of 'hmmmmmmmmmmmno' associated with it.

In my experience it ran fine, but for obvious reasons when shipping software that might make or break their company I can understand Joel preferring the familiar territories of ASP/IIS/Win or LAMP.

Reply on news.ycombinator.com