Why We Shut Down Charm on the Eve of Public Launch, at $48k/Year and Growing | Hacker News Reader

Why We Shut Down Charm on the Eve of Public Launch, at $48k/Year and Growing

unicornfree.com

88 pointssethev35 comments

The "don't do it if it's going to make you miserable" stance is a nice switch from the usual "startups are suffering so go read some Marcus Aurelius" attitude we get around here.

As somebody who's read Marcus Aurelius, I feel like if you read it, it ought to make you realize how pointless pointless suffering is.

I am not sure if I buy the "it's nobody's fault" explanation. From what I gather, what they built simply did not work, even though it was clearly feasible. They had a failure of engineering and could not afford to rebuild the entire app. I know nothing about the specifics of the architecture, but I do know first-hand that if using Rails as the basis of an efficient and high-capacity SaaS backend, it is the wrong decision.

I was going to write a sarcastic reply saying "Whoops! Better tell all [major companies using Rails] that they made the wrong decision" but honestly, is it worth my breath? Haha.

Funny, but I disagree. Rails scales, if you know how... We serve millions of customers each month and have implemented a reliable and stable architecture.

I think you're replying to protonfish, not me.

Anything scales if you know how, but why choose to use a buggy, bloated mess if you don't have to?

What would you recommend?

Great question. In my career so far I have deployed Web apps in ASP Classic, PHP, ASP.NET and Ruby on Rails. It appears that RoR can only handle a small fraction of activity compared to the other technologies before maxing out system resources. There are some lighter weight Ruby frameworks out there (Rack, Sinatra) that are probably more appropriate for large-scale back ends. Still, I have not worked with any Web technology that is performant enough to confidently endorse.

Indeed! The only thing that doesn't scale in my experience is people's sense of humor on Hacker News.

> but I do know first-hand that if using Rails as the basis of an efficient and high-capacity SaaS backend, it is the wrong decision.

Ok, I'll tell 37signals that it's time to shut down Basecamp. You get on the phone with Shopify and break the bad news.

Even they - 37signals - have agreed http://37signals.com/svn/posts/1509-mr-moore-gets-to-punt-on... they solve their scaling problems throwing bigger and faster machines to them.

I think the problem here is that they choose a technology that is difficult - but not impossible - to scale, and they didn't have either the in-house know-how to solve their problems neither the money to hire someone with the knowledge to do it.

I think RoR could have been a good decision a few years ago, when the web framework landscape what atrocious, but nowadays there're a lot of good options based on other programming languages that have already proved could be scaled without finding untraceable bugs. Heck, just a few days ago an acquaintance showed me a webapp running over a perl - Amazon, IMDB, Slashdot... use perl heavily - async framework 20% faster than node.js.

It would be a terrible shame if you tanked a viable business because of availability concerns, because these sorts of problems are well solved. Highly available systems are designed in such a way that any individual component may fail without bringing the overall system down. (ie. "Kernel issues? Detect, remotely reboot server." Yes! There's free code for that. No. Many run of the mill sysadmins probably aren't familiar with it.)

Asked with all sincerity… did you read my article?

"A kernel issue" was not "the problem." There was no "the problem". There were a slew of problems of which that was merely the icing on the nasty problem cake.

And considering how many times I lauded our sys admin as an expert and top talent, not sure why you would think he was "run of the mill." Nope, we hired somebody who had one of the best possible track records in the RoR world. We had a redundant architecture from day 1. My demand was that we have a system ready to work at scale so we wouldn't be growing and 8 or 9 months in, have to migrate servers, complete with customers. Not my cup of tea. Unfortunately, that caution didn't help (and maybe it hurt).

Why not sell the web app? Surely some developer would be happy to keep it running and live off it.

Hi, y'all. Amy here. If you've got a real question, I'd be glad to answer it.

EDIT: As of 4pm eastern I'm off to meet my interior designer and then attend a joinery demo at a wood shop so it may be a bit but I will absolutely read & respond to your questions later or tomorrow. My goal is to share. And maybe mock people a little, if they can't think up an original insult when they attack me.

You mentioned the logistical, technical and financial problems involved. Was part of the problem that you simply lost interest in the project?

Good question. Not at all. We were exhausted, but we love the project. I still hope that if we come across the right hire(s), we'll revive it. But I'm not going to spend a year of my life desperately hustling to hire (which still may not work) instead of working on the profitable product we already have.

No questions, but hopefully the naysayers and trolls won't bother you this time around. I've certainly got to the point where I DGAF about other peoples opinions - at least people on the internet. Heck, I prefer Java for developing applications, but around here a large number of people would consider me an idiot for just thinking about Java, let alone actually using it.

Thanks for the sentiment. They may show up, but they never bother me!

Have you guys considered open-sourcing the code and/or providing some detailed info on the architecture you've built?

We won't open source, for a variety of reasons. One reason is that if we serendipitously meet the right dev/hiring manager, we will hire them and revive the project. Secondly, as all bigcos who randomly outsourced a dead project can tell you, OSSing a huge project solves no real problems. Thirdly, neither of us will run an OSS project for it, and fourthly, it's a bit of a beast. You can't just pop Charm on heroku and expect it to run at speed. So, no OSSing.

As for the architecture, it was a Rails app with fully-live Backbone/CoffeeScript UI, a tuned Ruby cron job for scraping mail (POP, IMAP), a Postgres database and I forget which full-text search engine because we must have tried all of them before we settled on one. Nothing terribly fancy.

I understand your hope that you can resurrect the project. I was more interested in OSS for the opportunity to learn through reading the code and looking through your server setup.

Do you think bootstrappers should focus on smaller products (that are less complex)?

Absofrigginglutely. Until (and unless) you have the people to work on scale.

I don't mean to dwell on the technical issues, but I'd be very curious to hear more about the performance issues you encountered that the Rails Core members couldn't resolve. Was there ever any resolution apart from downgrading to 2.x, and are there certain things that should be avoided in Rails 3.x?

Whatever it was, it is gone now. This was Rails 3 early days… very early days. The problem was in the development environment only. It was a crazy memory leak — 50MB every page load or so. As you can imagine it because so unbearably slow we couldn't develop in it. Bizarrely it didn't manifest in production (even if we just flipped the bit to run mongrel — or was it nginx at the time ? — in production on the same dev computer/user/PATH, etc). You'd think that would make it easier to track down, but weirdly, it didn't.

I don't mind at all answering technical questions, ask away. What I object to is people who spot a jargon term or two they understand and fail to read all the surrounding paragraphs, and then lambast me for something they know nothing about. Which you clearly didn't do :)

Being the sole person responsible for a system that customers rely on 24x7 to make them money is just about the most stressful thing I've ever experienced as a developer, so I can definitely relate to that weight on your shoulders.

If it's any small consolation, I'll bet you beat yourselves up about the downtime issues way more than your customers ever would.

Actually… I didn't really beat myself up about the downtime. I am totally over beating myself up for things… it just makes it harder to fix them and it's not as if my guilt would help anyone else feel better!

It was upsetting, though, on an intellectual level. I wanted to do it better.

Hi Amy,

Thanks for sharing your experiences in such an open manner. It certainly takes much courage and bravery to shutting down your service and even more so, writing honestly about it.

Could I ask you a few questions:

- Is it true to say that if this product required less maintenance (in terms of uptime, server administration, future migration), you would not have abandoned it?

- Secondly, assuming that you are not against raising external funding (which you are not in favour of, as mentioned in your blog post) and hiring the right team for it, would it be fair to say that you would be willing to continue with Charm?

- Lastly, and hypothetically, if Freckle and your other opportunities were not around, would you have continued with Charm? Would it be fair to propose that because of the high maintenance required by Charm upon you and your team, the opportunity cost is simply too high, relative to the other options that you have?

Thank you, and hope to hear from you. :)

1. I think we gotta talk about the word "abandoned," which to me involves letting something languish without attention, which is exactly what we didn't do. But aside from disagreeing with that word choice…

Would we not have carefully shut it down if it wasn't for the running-of-it issues… hard to say. Aside from the server problems, I described our staffing / resource problems. We didn't have enough staff to keep building Charm, adding features our users deserved. This is a probelm.

2. Not sure what the funding part has to do with willing to work on Charm. I will never, ever, ever take funding, unless it's to pull a 37signals or a Github — on completely my terms for a tiny sliver of the company — to get a personal advisor or a big "in." But that I doubt (doubt we'd ever get big enough for that to be attractive to an investor). So, assuming I wasn't against funding is too big an assumption to make.

3. Good question. If Charm were our only baby, it would have been much easier to keep going… though frankly I don't think we ever would have attempted Charm as a first product (that bit of sense we lost after the success of Freckle). The opportunity cost, as you say, was huge.

I'm curious whether you looked at "selling" it to another team? Something like "run this product as your startup, we'll keep 20%" or similar. When I see folks shut down products or pivot to something else, I wonder if this approach is workable. WDYT?

Pure flame-bait for HN readers, sysadmins, developers, founders. Not worth reading.

Agreed

Totally. Nobody on HN likes to read post-mortems or understand what goes wrong on a large, revenue-generating product, or in partnerships, or growth. Or hear opinions from somebody who's bootstrapped a biz to $750,000/yr gross. Nope. Definitely not of interest.

Reply on news.ycombinator.com