Being a Solo Founder of a 24x7 Hosted Web Application
garrettdimon.com
garrettdimon.com
A lot of the fear I detect here is along the lines of "what if (something breaks|customer calls) at 4am on the day my wife is giving birth???" The reality for me in the last five years is that's not something you need to worry about.
My sites generally don't break, and not because I'm awesome or anything. They don't break because they're built on a stack that doesn't need any attention pretty much ever, and for the most part they don't try to do anything particularly remarkable. Reading/writing from databases and displaying the results on a web page is pretty much solved. You get four nines right out of the box unless you go out of your way to choose a flaky platform.
My customers do actually call from time to time, but it's not the sort of thing I'd consider getting out of bed to worry about. The ones that pay money generally get a response within a day, and the ones that don't pay me anything will probably get a response when I get around to it. That, also, has worked perfectly well for the last five years.
So while it did in fact turn out that I had a "crisis" at 4am while my wife was giving birth, it was in no way a priority for me to do anything about it just then. I had much more important things on my mind. Putting off the disaster recovery process possibly led to losses in the two, possibly three figure category. Fortunately as a single founder, that's the sort of decision I get to make all by myself.
This is something that Rob Walling talks about extensively. He personally deals with customer support for the first several months of his app, so he can understand the pain points and what needs to be added to self-help resources. Then he writes up an extensive training manual for outsourced workers and begins to train them.
Eventually, he stops dealing with "OMG I lost my password and don't know where the reset is?!" questions...and only deals with customer support that is actually worth his time.
Do I push out a new fix that stops some bugs, upgrade a feature that customers have been haranguing me about, do some marketing to get new customers or work on support so I can devote more time to <everything else> in the future?
As with most things in startup land, it's all about compromise, which I'm sure you already know =) Loved the post btw, looking forward to more posts coming from you
Also, we have a very different view on customer service. I strive to respond to all customers immediately. We could reply within a day, but going above and beyond for customers is important to me. The support requests in the middle of the night aren't even a problem. It's just part of the gig.
Indeed. Though going above and beyond is important to me as well. We just seem to have different ideas about what that means too.
I think patio11 says it best:
"I appreciate that you'd feel safer going with Brand Name X, and Brand Name X will indeed have someone around to answer your phone call at 2 AM. Unfortunately, they won't be able to do anything for you. Their only job is getting you off the phone before you can speak to someone capable of resolving your problem, because those people cost money. And you know what happens if you call at 2 PM? You get the same guy."
As in, you might not get a response for 16 hours, but when you do it will be from me, the guy who built the thing. And it'll be to tell you that your problem has been solved and no further action is required on your part.
Incidentally, I don't have email on my phone. Nor do I carry my phone around most days. I sometimes go a few days without checking email while on vacation, and I'm on vacation a lot. If I could give you one piece of advice, one solo-founder to another, it would be to figure out what you can change about your product so that you can do the same. It's entirely your gig, so it's up to you to choose whether anything unpleasant is part of it.
For us, the biggest change was adding an awesome system admin to the mix on retainer. He's been a huge help and we now have a much more resilient architecture. It's all evolving though as I find time to streamline the processes as needed.
All definitely some good food for thought. Thanks!
As a single founder with a web app, it's amazing how closely the experience in your blog mirrors mine. However, I haven't yet gotten used to that nagging feeling when you're on holiday and you know you would really be in trouble if the system were to go down right now and you had to debug something on a laptop with a flaky internet connection. In reality, nothing more serious than 30 minute downtime ever happened to me. The site is doing really well so I don't have a money issue since, but that also means the financial impact of any serious problem is magnified as well. For me, that has made stress levels go up rather than down, I hope you can keep your cool if your site takes off. Hopefully, I can get to the point that there's so much coming in that I can hire someone, but if I look at the few service-threathening issues we've had so far I really can't see how anyone could have solved those except the person who built the system. Me.
Get a monitoring system in place that you trust. Then get a second, completely independent one in place that you trust. Then also get on-machine monitoring (monit, et al) that tries to fix things if they go bad.
In retrospect, that should have happened a lot earlier. Once we had redundant monitoring systems and a firm belief that if something was down that I'd be notified about it, the world became much more chilled out. If I didn't have an SMS, the services were up and running.
I wildly agree. I'm more apt to deploy a second monitoring system than replace one. I don't worry until it reaches the PagerDuty threshold.
If you're thinking "That sounds like an ideal gig but the technical support burden / server maintenance / etc will crush me", as me for specific elaboration, but in general this is much easier than you think it is.
I used to think that adding a second full-time person was the solution for alleviating pain, but now I see it as the way for all of the work to simply be more fun because there's somebody else to help accomplish more.
Story time: Customer X signed up for the free trial of Appointment Reminder. 3 minutes later he sends an email asking if I can please hop on a call with him. This is at 2 AM Japan time but I am just putzing around on HN so I apologize to Ruriko, say that I'm going to make a five minute phone call, and call him from my cell phone in Japan. He asks a typical pre-sales question, I answer it, and we're done in five minutes.
Fast forward to three hours later: phone rings. Customer X, who now has my Japanese phone number, is calling with a new issue. He has been using the software and thinks that grey text on a yellow button is difficult to read, and I should fix this for him. He sent me an email but I didn't respond to it. I tell him that I live in Japan, that it is 5 AM, and that I will ordinarily respond to emails within 24 hours. He asks another question and I, somewhat testily, strongly suggest writing it in an email and end the conversation.
Next day, he calls back at 2 AM. He wonders whether the software will also send SMSes. I tell him that the first line of text on the front page is "Appointment Reminder sends out customizable phone reminders, text message reminders (also called SMS reminders), and email reminders to your clients, so that they come in on time and you get paid." and that he could have asked this question over email. I then remind him that I am in Japan, that it is 2 AM in the morning, and that the service level for a $29 a month account is best-effort email.
Three days and six calls later I finally lose my patience and tell him something to the effect of: "There is an actual cost involved in having our head engineer wake up at 5 AM to answer your phone calls, which -- by the way -- you have made more of in the last week than our entire customer base has ever. If you need this level of service, you should purchase an SLA, which will cost you north of $5,000 a month, or I will assist you in finding a service provider more appropriate to your needs."
Customer X takes the hint, then begins badgering me over email. Despite having time zones and the 24 hour thing explained multiple times, he gets very upset when I don't respond to emails in 20 minutes. My happiness level deteriorates over the week.
Finally, I tell Customer X that he is making economically non-viable use of customer support, and that if he needs this level of support, I will assist him in finding a provider appropriate to his needs. I then explicitly, for the first time, mention that we can terminate accounts. That's the only time in my life I have hinted at that threat.
This quiets him down. Several weeks later he self-cancels and I take Ruriko out to dinner to celebrate.
But again, compared to the amount of drama I suffered when full-time employed, this is a funny story affecting 2% of my time, I've made a process improvement to avoid it in the future (I now do customer phone calls through a Twilio proxy so they don't get my Japanese cell number and, if someone is X-like again, I will cut them off earlier), and I'm currently enjoying my honeymoon in Hawaii 98% disconnected with no real worries.
I do respond to emails if it seems like the customer could really benefit from a response, and even sometimes when they aren't looking for a response. Sometimes, if it can wait, I'll go back to sleep and handle it first things in the morning.
Being the one person responsible 24/7 for a web app that can not go down is incredibly stressful. Doing that year after year is no joke. I think it changes your life as dramatically as having kids does. I'll never look at technical decisions the same way again, even though I know things will be less stressful in the future.
There were many low points and I know if I didn't have cofounders and customers depending on me, I likely would have quit and found a "normal" job several times over the years.
But I can't even describe how satisfying it is to be able to make the lives of customers just a little bit better every time I check something off my todo list. I'm not curing cancer, but I also have a much closer connection to the work I do and the impact it has on the lives of real people than I ever have before. And our customers are amazing and very appreciative of the work we do. We've had many tell us that they'd quit their businesses if they had to go back to the days before they had our product. That makes all the stress and many, but not quite all, of the sacrifices worth it.
If you can, definitely find another technical cofounder from day one. But the reality is that many businesses just can't make that work without supplementing revenue with consulting or from some other source.
Congrats and good luck Garrett! Thanks for sharing your story.
I think you might be exaggerating about the kids, (I mean, I don't have any of my own, but I'm the oldest of 6... really quite a bit older than all but one of them, so I think I have some idea as to what is involved.) but I really agree with the rest of that. Everyone that makes technical decisions should have to take a turn on the pager.
I've also been on pager duty before. But being the only person responsible for keeping a critical system running for years on end is an order of magnitude or two worse than standard page duty I've ever done. I should add, our business is very seasonal and very unpredictable.
Now, /that/ is impressive. I mean, yeah, I'm not impressed by the pager. I've been on pager more often than not from 17 onward. But, I don't have kids, and my significant other is also technical, also works a lot and is super understanding. (well, and it's really only been the last five years or so that I had a relationship I cared about more than my servers, anyhow. Maybe less than that. I like my servers a lot. and I think there is a trial period for partners.) - I mean, that's the hard part about pager for most people; "hey honey, I've gotta stop paying attention to you, bring out the laptop and fix something real quick" and that isn't all that much of a problem for me. I've never really been a 'planner' so getting high priority interrupts isn't a huge problem.
but yeah. juggling that /and/ kids would be tough. well, kids in general would be tough.
eh, have you ever been the only technical guy at a company? I think it's a pretty similar experience. Either way; being a sole founder, and being the only tech guy at a company is way easier (assuming you have the skills and power to fix the system so that you don't get paged more than once a week) than being on pager on a giant and seriously broken cluster for other people.
I've worked other places where the pager went off three times an hour. Sure, you only had one week out of four, and only 12 hours a day, but if you fucked up even a little bit, the company would lose hundreds of thousands of dollars. It was a completely insane situation, and completely broken, as it should have been pretty easy to automate most of what we did. (most of what we did was telling the cluster 'okay, don't use that node, it's broken') Now, why didn't we fix it? I dono. there were some sharp folks there. Maybe we were all too burnt out from pager? maybe it was internal resistance to just throwing nodes back to our (really shitty) burn-in routine? (I mean, the idea was you were supposed to log in to the broken node and troubleshoot rather than just pulling it. But realistically, you only did that for the first day or two.) In the end, most of us ended up leaving.
This was the only pager that has phased me. Out of being on pager more often than not more than 20 years (well, uh, I guess it was closer to 15 years at that time) that was the only job where the pager broke me. Really, that was the only job where the pager bothered me much at all.
But yeah, compared to that? being a single founder is super easy. I get to make all the decisions, so I can choose things that break less. I can take all the shortcuts I like, and avoid the shortcuts I don't like. If the system is unreliable, it's completely my fault. If I want to take a pay cut so that I can hire someone else to lighten the load? my decision.
I mean, it's kinda like being the only tech guy at a small company, but you have a whole heck of a lot more power, so you can make decisions like hiring a customer out of IRC to help you out, which the boss almost never approves.
>I've also been on pager duty before. But being the only person responsible for keeping a critical system running for years on end is an order of magnitude or two worse than standard page duty I've ever done. I should add, our business is very seasonal and very unpredictable.
I guess my main point is that normal pager (when you are the only/head sysadmin) has all the responsibilities of being the single founder on pager, with a whole lot less power to fix the root cause of those pages. Being the new guy in a large organization usually gets you a little bit less responsibility, (not all that much less if you've sold yourself as some kind of super-expert sysadmin, or at least, that's how I felt.) but very little power to actually fix root causes.
Personally, I think the power to fix the root cause makes the job incredibly easier, and incredibly more interesting and satisfying. Otherwise it feels like you are just playing whack-a-mole with the problems. Nothing bothers me quite as much as fixing a problem without knowing how I fixed it.
So how does pager duty work with your employers? Most places I've worked have asked for a 15 minute response time (which effectively means being within ~12 minutes of home) and being sober. That means no going for a bicycle ride or a run; not going on dates, to restaurants, the theatre or cinema; not visiting friends or relatives; and not going to the gym or going swimming or going out dancing.
Doesn't pager duty basically mean you have to sit at home on the internet for all your free time?
I have ssh on my cellphone which is on my person at all times, and while doing anything involved is difficult, I can usually verify that there is a problem and it's nature, and if it's easy, I can deal with it then. But that satisfies the 'respond in five minutes' or whatever. I responded, I can tell people involved what's up and how long it's going to take me to get more info.
The next step up is my laptop, which usually stays in the car. I jog to the laptop, boot it up, use the cellular modem dongle thing, and it's about as good as being at home, most places. I have a cheap inverter in the car if my batteries are low.
If I'm willing to take a social hit, I can also call someone else and say "Hey, I'm drunk to the point where I can't handle root, or I'm 20 minutes out and my laptop took a shit. X paged me. can you deal with it?" and repeat until someone says yes. I look like a fool in front of my co-workers, but eh, the problem gets solved, and everyone does this every now and again.
(This is the big problem with being a sole founder. You can't just call up a random consultant; you need to know the system to be able to fix it. It's really nice to be able to call two or three people that know the system well enough to help you out in these times. In this case, I think even someone that doesn't have a lot of experience but is willing to be trained, and that you can afford to pay often enough that your system is fresh in their mind, is better than just calling someone really good that doesn't know the system when the system is hoarked.)
So yeah, I can go to movies. I might need to walk out of the movie, but I can go. Restaurants are easy; I don't even have to leave; I just whip out my laptop and fix what needs to be fixed. dancing? same deal. Yeah, I look like a dork or an asshole if I get paged and I whip out the laptop in the middle of a social engagement, but honestly? I'm not sure that really changes anything.
Really, anywhere I can take my laptop that has cell reception and power, I can go. I mean, I know I might get interrupted, but I probably won't be.
The hard part is really the mental state. You really want to make sure you go to bed before you are exhausted, because problems that take five minutes when you are fresh can take, well, much longer when you are not. Alcohol too; I find that the DMV limits are reasonable, and if I stay under those I can handle myself (on a computer; I'm a little more conservative when it comes to driving, as I'm a pretty shitty driver to begin with.) Sleep, really, is the problem I struggle with, but I have a hard time keeping a schedule anyhow.
(Oh, also, the 20 years is clearly a lie. I'm what, 31? something like that. and I think I got my first pager at 17, so considerably less than 20 years. For some reason, this week my sleep schedule has been pretty messed up, and I haven't been mentally 100%. I've been writing a lot, and writing poorly about stupid bullshit that doesn't matter. Odd.)
Overall I think she enjoys that, adding a child to the mix has been fun & surprisingly hasn't changed much at all -- in the first few months it gave me an even better excuse to be up at 4 in the morning.
I've been at this for about ~6 years as a solo guy (albeit, with hires -- so that changes things entirely) and a few years prior at other companies in slightly lesser capacities as an admin.
Which reminds me, I need to get back to you one of these days.
I asked my co-workers on rotation if they were interested in trying to get a cellular modem that we pass around in addition to the pager, but they would rather try to use iPass to get on the wifi somewhere.
I mean, this is what being a SysAdmin is. Something is very wrong if you get paged more than once a week, but you carry that pager all the time. It's part of why we pull down real programmer money with only a subset of real programmer knowledge.
I've got a cellphone with extra batteries and a Verizon dongle thingie that can get me online from just about anywhere, so it's really just the power I'm worried about. Sometimes I carry extra laptop batteries.
For me, the biggest problems have been when I was paged when exhausted, sleep-wise. A few times I've had to hand off because I was in that state. (I've had a problem where I handed it off because I needed to sleep, then Nick handed it back to me in the morning because he needed to sleep.)
So yeah, having someone else is pretty important, I think, and worth waiting a few more years for the 'real money' payout, but eh, if you are doing it right, you can be front line pager forever without too many issues. (that said, I'm going to a family reunion soon, in Montana. probably the farthest away I've been in a while, and certainly the longest time I've spent on a plane. Nick will be primary on pager.)
Automation and monitoring is crucial when you are solo founder. Think about automating every little thing that takes only few minutes to do but the budgeted time is 5 times of actual time it takes to finish it.
Outsourcing, automation and dashboards are your friends. I have either outsourced majority of my activity or automated it.
But the problem with running startusp is the task list keeps changing every day and is like a bottomless bin. Everyday you automate something and you keep finding new things to do.
Healthy work and life balance is so crucial. Due to stress at one time had to visit the emergency department, glad it was just "Stress increasing the acidity" and wife had all the reason to take control of my routine (just few days). After that episode, I immediately put breaks on all the stressing out. I realized there was simply no need of making things happen all at the same time.
I put back my meditation routine on track ( avg 2 hrs/day) and that has helped so much in terms of health and clarity (plus other things). I recommend everyone who wants to be involved in startups to start meditating. It will be a huge favor for your family and yourself.
Goodluck and cheers to all the solo founders!
This has actually been an interesting process. I came up with three people, had one in mind as an ideal candidate, and approached all three. My idea was that if others really worked out, we could go with more than two people. the top candidate and I are still talking, but the other two never went anywhere.
There are a bunch of reasons for this. First it is easier to apply for funding if there is a team, but more importantly having a second individual provides needed feedback and makes total screwups a bit less likely. It is always important to keep folks around who will give good, honest, and occasionally brutal feedback. Nowadays I am not even sure we will apply for funding. We might bootstrap.... But getting a team in place is for now the critical task.
So I completely agree with the article's conclusion that he would have tried to get a co-founder if he did things differently. This is so important to do.
I'm bootstrapped, and don't ever plan on outsourcing anything - except for maybe writing blog posts. But the feeling of needing to be close to a phone or Internet connection at all times in case of the inevitable really concerned me.
But then I sat back and breathed - downtime happens. It's not the end of the world, my customers will survive. I've recruited a close sysadmin friend of mine to be on call should anything ever happen, who I'll pay per incident. But that's it, life is too short to worry constantly and be glued to your phone while with your family or loved ones.
Thus far, without being able to hire full-time, outsourcing that stuff was the best decision that I've made yet. It enabled us to launch some features that might have otherwise taken me forever to get around to building.