Charm shutting down because of unfixable kernel panics with Rails on Ubuntu
raw.github.com
raw.github.com
I call BS. It's either a smoke screen for some other issue, or their app was built back asswards. Either way, this post and the circumstances are indicative that Amy and/or the Charm leadership has no idea what they are doing.
And they were not even launched? Beta or something? The comment about opening to the general public implies that they weren't getting that much usage. These days it's hard to build something with a modern framework and decent hardware that does not at least work for a closed-beta period. Add 100k to the mix and honestly what the hell.
Amy/Tom run Freckle and do some other cool shit so why are they dropping the ball on this? It's like getting a paper cut on your finger and going home for the day because it's just too much to bear. Cry me a fucking river.
Why does this keep happening?! People are throwing cash into the dumbest shit these days.
Reading this must make founders striving to get investments furious..
I'm sure shutting down a perfectly good and potentially very profitable product will make lots of wannabes and aspiring entrepreneurs jealous. Luckily I don't make my life decisions based on what other people would do in my situation.
When you run a real business, with real customers, you quickly discover that what sounds good, isn't, and what looks bad, is often the absolutely right thing to do. Luckily, our early access customers for Charm heard & understood & appreciated my email.
FWIW, it deserve respect to shut down something you built with your own money.
In hindsight, I think you would have gotten an entirely different response had you said "We can't run Charm with the team we currently have, there are technical challenges we can't solve". Understandable and believable. But for all the people dealing with problems like the one you blamed on a regular basis, saying what you did just sounds like:
"We spent or life-savings building this great house! But it keeps running out of toilet-paper, so we burned that s"#¤ down!!"
BTW — you got all the facts wrong. Just in case that wasn't clear. And nobody was asking for your sympathy.
Good to know you couldn't master the most popular Linux distro though! Hope the VCs can read this.
You probably noticed Charm had some nasty downtime a couple weeks ago.
Service quality is very important to us. If we didn't think we could do better, we wouldn't do it at all.
We've spent very generously on sysadmin services and infrastructure (nearly $100k of investment on sysadmin services/infrastructure alone). We hired the best possible, and we splurged on a redundant, powerful, and expensive server configuration from the beginning.
Now we've discovered that there's some kind of base incompatibility with Ubuntu, which is giving us kernel panics which nobody can track down. Charm has been plagued by mystery technical problems from the beginning, when we had to backport from Rails 3.x to 2.x because of massive performance slowdowns which even Rails Core members couldn't identify.
What this has really shown us is that, if we open Charm to the general public, we won't be able to provide you with the kind of service you deserve. We are a tiny team, and so far, we've had zero luck in our attempts to grow by hiring developers. Problems which are small now will only get bigger.
There are a lot of things I'm willing to take risks with, but not with your ability to provide support to customers for your business.
And so it is with a very heavy heart that we will cease operating Charm from Dec 15, for the foreseeable future.
You won't be billed again, and we'll refund your last payments.
We will gladly help you migrate your data out of Charm. Please contact us directly (support@charmhq.com) for help.
Thank you so much for taking a chance on us, and sharing our dream for a superior email interface.
I'm truly sorry to disappoint you.
Best wishes,
Amy
1) Bought a ton of Heroku dynos with every single addon enabled. That would have given them any version of Rails they wanted with Ruby 1.8, 1.9 or even 2.0; Postgres, MySQL, Redis, MongoDB, two different kinds of Memcache and three different kinds of asynchronous processing.
2) Gotten fully managed servers at Rackspace with pro support for pretty much any setup they wanted.
How is this even possible? The same application happens to hit bugs in the Ubuntu kernel and Rails core that haven't been fixed all the way to 3.2.9? And they have 100k to spend but couldn't be bothered to try different kernels and distros? Really? Have they seen the AMI launch screen on EC2?
Nobody could identify it.
Our sysadmin was a sysadmin for one of the first, biggest, and most reliable Rails app ecosystems. He couldn't trace the Ubuntu issue.
It sounds like you don't have a lot of experience running a business. When you do, you will realize that just throwing money at problems doesn't solve anything.
PS - We have servers at Rackspace. 3 of them. Where do you think a lot of that money went? They're $3k a month. And guess what — they won't manage servers with the configuration we required. Fab.
Lots and lots of people run Rails apps on Ubuntu. Many more run them on non-Ubuntu distros. There is not an epidemic of "welp, Rails magically crashes the kernel, time to pack up and go home", and citing that as the reason to close down after spending over $100k on infrastructure smells really, really funny.
"A poor workman blames his tools."
. . . but even a master carpenter can't build a house out of rotten wood. Of course, to me, rotten wood is OSX and Windows (why anyone would build servers on those instead of one of the BSDs or Linux, I'll never know; to me, development is also incredibly painful on Windows and OSX), but Ubuntu is not without issues. The rallying cry of "usability first!" is all well and good, until things like taking shortcuts by way of binary drivers causes crashes. Still, one has to wonder why they didn't just try another distro. If you think your tools (or your "lumber") are the problem, try different ones.
Agreed; however, if they were having problems with Ubuntu (or indeed Linux in general), why didn't they try something else? That's the quick and easy way to see if it's really the platform or your application that's the problem (quick tip from my own userland developer experience: it's almost always not the platform; the times I've had Linux crash on me I could always trace to one of three things: 1) flaky hardware, 2) binary drivers, or 3) I was mucking about in kernel space ;)
The whole situation just makes my brain stare blankly and wonder what on earth was happening over there.
My mind just exploded, too, due to your willful inability to read the details.
$100,000 and no one said "hey what about running it on <insert other os>"?
It's quite fascinating really, especially in 2012 to give up because of software failure.
We moved to debian after numerous problems and have had no issues since.
I don't have the report ids any more.
I don't even know what we're running Freckle on, because it's an app that's had zero problems. Alas, Charm is a troubled baby.
Just saying.
Maybe if they were a bit more familiar with Linux they would have done better, or at least have saved some money figuring out what went wrong:
Sounds to me they would rather blame Ubuntu than their inability to fix the issue.
Later edit: to elaborate that last point. I'm not saying Ubuntu is not at fault, but rather that problem solving goes with the territory. Given the wide array of alternatives, this was not the sort of thing you could openly complain about.
I, for one, would not like to work for you. I'm not trying to be offensive; in fact, I respect the decision to cut bait when you realized you were over your head. I just don't think I, as a Linux user, would fit in your organization. Many of the good sysadmins also have this same viewpoint, so the above certainly is well founded.
Anyway, kudos for not submitting their users to a service they're not confident they would be able to provide with adequate quality and better luck next time!
Or even just Debian. I ran into issues (years ago) with Ubuntu stability due to closed source drivers on a laptop; sure, I have the skills to debug that sort of thing, but why waste my time? I had been running Debian just fine on desktops and servers for years, and after switching the laptop to it, the lockups stopped.
I'm more than willing to admit there may be real technical issues they ran into, but it sounds to me like they gave up too easy, and we may never know what the problems are. Would've been nice to at least have seen a bug report (honestly don't know if they submitted any, but I'm willing to bet they didn't).
Also, with Arch, you get less by default, so you'll end up with a cleaner system. I didn't realize how much stuff I didn't need on a default install of Fedora until I set up an Arch system.
It's easy to armchair quarterback… especially if you're not hampered by actually reading the very brief bit of content you're speculating about.
Software is deterministic. If you can't explain why it's doing something, then it's not leprechauns fiddling with your bits - it's something that you just don't understand yet, and will have to dig a bit to find it. I'm not speaking from armchair quarterbacking here - I'm speaking from experience.
I am exceptionally dubious that you were encountering problems that were just not solvable, let alone explainable. Doing the handwavey "nobody can explain it" has a particuarly bad odor to it.
If you want to do a full writeup on the problems you encountered and what you tried to do to solve them and why you eventually gave up on them, I will be first in line to read it.
We did have 3-4 of the Rails Core team members investigating our Rails 3.x problems and none of them could figure out what it was. The problems were so terrible it made it impossible to develop with Rails 3.x any more. Obviously Rails Core members were both A) friends, and B) highly motivated for us to keep using Rails 3.x.
The problem with trusting your abilities is that you can't imagine a scenario where they will let you down. And yet, many such scenarios exist.
I get your suggestion of hubris here, but again, I'm speaking from experience. I've done the solo technical founder thing, and that came with a whole host of problems that seemed unsolvable and which I didn't have anyone to appeal to for answers. Those are the problems that taught me most of what I know - most importantly that when you run into something that you don't understand, it's an opportunity to learn about it and solve it, rather than to just give up. In the Rails world, where the entire software stack is open-source down to the kernel it runs on, there is literally nothing to get in the way of understanding what's happening at any point of the application's execution.
I'll reiterate that if you want to do a writeup on why your circumstances were unique, and the problems you faced which were so crippling that they forced a rewrite, I'd love to read it. The suggestion that Rails 3 has landmines so critical in it that it necessitated a full product rewrite in Rails 2 is exceptionally weighty and should not be asserted without a very specific cataloging of what those problems are. That particular assertion - "Rails 3 has problems that cost us $XX,XXX and the Rails Core team doesn't have a clue what they are" - is a big one.
Blaming this on Linux or Rails is ridiculous. And as for 'mystery technical problems'...
I think the reality is more like they got in over their head, decided it was all a bit too difficult, and gave up.
Probably a bad idea to advertise you can't keep a simple webserver up. Probably won't get funded ever.
The fact that they could not achieve break-even/speed-gains on Rails 3 tells me the application was likely a less-than-stellar codebase to begin with.
I'd agree, if you can't get a Rails application running after investing $100,000 [minimum] in it, it's probably wise not to advertise that fact.
>Probably won't get funded ever.
I do so hope you're right.
The investment looks like a way to formalize a long term relationship both sides desired. It doesn't resemble anything like 95% of VC deals that are associated with SV or the types of starts ups associated with HN's readership.
http://37signals.com/svn/archives2/bezos_expeditions_invests...
Haha, won't get funded ever? I have VCs knocking on my inbox doors on a monthly basis. We don't take funding. We have, however, brought in >$1 million in revenue on products since we started doing products, including http://letsfreckle.com.
The sysadmin I mentioned is one of the most respected experts in the community. He wasn't cheap. He had a fantastic legacy of very good, very public work. And yet, here we are.
When you run a real business, you will be faced with tough decisions. Hopefully you will be able to make wise choices as well.
1. Open Source OS
2. Open Source web server
3. Open Source programming language
4. Open Source web framework
5. Some (relatively) simple customer support ticketing app [not likely to be a system pounding Gorilla]
And they claim that they've not been able to root-cause a Kernel panic and a framework slowdown?
And they've spent $100K on that with no results to show for it?
That sounds really strange to me.
[I've debugged Linux kernel panics on custom stacks, with no disk/logs, only console, 32MB of RAM, PPC cross-compiled - and never run into a dead-end like this]
Over the next few months our role expanded to include a complete rewrite of the user interface, a reimplementation of the credit card billing processor and a few more major changes."
This is the first time I read a case like this
100k for "infrastructure alone" seems like a massive investment for a startup. How far along were they?
I didn't get to see Charm before they shut down, so I have no idea what they were doing.
Could anyone speak towards how their experience was unique enough to cause kernel panics?
https://twitter.com/amyhoy/status/271128780058267649
The rest was infrastructure costs ($40k) and porting from Rails 3 to 2.
Googling for "rails charmhq" doesn't reveal much, just that one of the Rails contractors was a company called Queve: http://www.qurve.com/clients/charm/
You can open a bug with distros and try to work around it. It is usually doable
"unfixable kernel panics" don't seem something a person familiar with Linux would say. Also, there are several distros, kernel versions, and Ubuntu isn't my first choice for a server.
And the gist is over the rate already.
It's better to go conservative on your infrastructure, get it stable, then experiment if required.
I'd love to see an Ask HN with _proper_ technical details. Who knows, maybe they'll get better ways to solve this rather than close shop.
Found this so far: http://charmhq.com/Charm_Bootcamp.pdf
I have no idea what this team was trying to do, but I imagine there _had_ to be more than what they wrote in this gist. That, or they simply ran out of money to troubleshoot the issue further.
There are also many different ways you can run Rails apps. They could've tried to run them on varoius Rack compatible application servers (Passenger on Apache or Nginx, Thin or Unicorn, et al.) -- If they migrated backwards to Rails 2, they could've used an [obsolete] version of Passenger with Ruby 1.8.7-Enterprise, which had many speed and stability improvements. (Assuming their app wasn't 1.9.x compatible.)
tl;dr: There has to be more to the story than this, they had so many options. For a $100,000 investment in administration, I'd have trouble believing they _didn't_ explore other options.
The email isn't "Wah we hit a server problem, bye bye."
The point is:
"What this has really shown us is that, if we open Charm to the general public, we won't be able to provide you with the kind of service you deserve. We are a tiny team, and so far, we've had zero luck in our attempts to grow by hiring developers. Problems which are small now will only get bigger."
If you've never run a serious product, or a real business, or tried to hire for technical positions, I can understand why you'd zero in on the "facts" about the technical situation and ignore all the "foofy personal window dressing," and write things like "I call BS! It's a smoke screen!" or "Why didn't they just try BSD?"
And yet I addressed the actual problem in very clear terms in a paragraph you can't possibly miss.
Next time somebody makes a hard business decision and you hear about it on HN and come out, irony guns blazing, may I humbly suggest you read more than the subject line written by the unrelated HN submitter?
As for any poor silent, lurkers who wonder if this is how they will be treated if they -- gasp! -- ever find themselves in over their heads, or in a business they realize they don't actually want to be in… our customer reactions have been uniformly:
"Aw, I'm so sorry... Charm is such a nice piece of software... your email was so touching."
Why? Because we've always shown our customers respect by creating great software, and we're showing them even more respect by ensuring we do not make promises we can't keep.
Our friends and technical acquaintances have been full of nothing but sympathy, understanding, and for those closest to the situation, praise for making the right, hard decision.
Yep, it sucked. Yep, we poured something like $200k into development, redevelopment, and infrastructure all told. Yep, it is a fucking amazing piece of software and the best thing I've ever designed.
But is it worth the constant heartache of the impossible task of finding people equipped to work on it? Of having it stuck in some kind of product half-life because of that? Of feeling responsible for, but incapable of, being "on call" in the middle of the night?
Of feeling guilty because, unless we can somehow suddenly be great at those things, we're taking money for a service which might let our customers down?
Nope. It's not worth it.
And boy do I feel lucky and privileged that because we spent nothing but our own money on it, we are free to decide to do whatever we think is right.
See also my principles 10 and 11: http://unicornfree.com/2011/lessons-learned-from-16-years-of...