Your First Hire Should be a Sysadmin | Hacker News Reader

Your First Hire Should be a Sysadmin

engineering.crowdtilt.com

36 pointsknighthacker27 comments

Sysadmins are what you bring in once you've grown enough that operational issues detract from development rather than add to it. A sysadmin should absolutely not be your first hire. At most a consult with an engineer in the early days to get you started. This would help if your coders are used to handing off deployment to someone else. Bluntly, if you completely lack ops knowledge, buy an hour block from an ops specialist that can get you over the rough patches.

My concern with a sysadmin first is that you'll want your early development very tight with your production systems. A sysadmin is the last thing you'll need while doing fast iterations on an MVP. In the early days your need for feedback is high and your operational issues are low. After some significant growth your ops needs will be high, and the feedback value of those issues will become lower (more repetitive less new information). This is where you bring in a sysadmin to manage/automate the repetitive and filter some of the ops issues into actionable information for the development team.

Edit: Ops guy for over a decade btw.

> A sysadmin is the last thing you'll need while doing fast iterations on an MVP

I'm not sure I agree with this. Someone needs to be working on your CM scripts and debugging production issues... and if it's not a SA, it's you or your developers. If you really want to iterate as quickly as possible, you need your developers focused on developing, not system administration.

Sounds like you agree with everything I said until you edited my comment down to that sentence.

If there's honestly 40 hours a week of ops issues during the first hire/MVP/market fitting phase. One of two things is happening:

1. Your startup is so wildly successful out of the gate that operations/infrastructure needs to grow exponentially.

2. Your startup has produced something that is complete garbage.

Neither situation is likely, but I'll let you guess which scenario is more common.

A much more realistic scenario is that ops work starts off as 5hours/week and grows to 10, 20, 30, 40 etc.. over a couple years without anyone noticing. By that time you've got a team of 4-6 devs and operations issues have become an annoying interruption to all of them. This is where bad IT decisions happen.

On that point (needing a sys admin for say 10 hours a week), where should a currently devs only team be looking? Remote freelancer? Dev with ops / sys admin experience?

Depends on the nature of the work. Begin with the understanding that operational concerns will never be zero hours. The best freelancer in the world will need guidance from the internal staff.

If you've already got something in place and operating at <10 hours a week, I'd keep it in house with whatever dev has more ops experience. Or whichever one drew the short straw after no one admits to having ops experience. You could hand it off to your newer hires to gain an understanding of what you do. The key is to define who responds to what.

Keep an engineer on retainer that you can call in to handle one-off builds and emergencies. Big changes/deployments where you can't afford to gain experience the hard way.

Author here. Admittedly, the title is a bit strong (though mismatch in titles was by a mod) but I was hired simultaneously with our lead dev and focused on ops/automation, as well as, product development in the early days. More accurately, my point is that it's extremely important to have _someone_ focusing on these issues from day one.

edit: the title mis-match was by a mod

The article makes more sense from that perspective. The idea that ops needs should be planned and responsibilities designated makes sense. Otherwise it's easy to end up with fragile systems that are no one person's responsibility.

I completely disagree. Traditionally, Sysadmins are what you bring in once you've grown enough that operational issues detract from development. Not only this made sysadmins feel terrible over the years having to deal with nasty production environments, but also developers were always slowed down because "Ops" were the bottleneck. And they were the bottleneck because developers didn't think like sysadmins when they started. It keeps going back and forth.

I hope the audience is smart and intuitive enough to understand that if none of the co-founders is a coder, then obviously your first hire can't be a sys-admin. With that assumption, having a sysadmin early on will only improve the productivity of the development team as is the case at Crowdtilt.

I guess the better word than Sysadmin is DevOps. Because what the author did at Crowdtilt wasn't just deploying and maintaining servers, but actually impacting and streamlining the development process.

I could not disagree with this more.

Focus on building your business and your product, not devops infrastructure. Unless your business is devops infrastructure.

Your initial technology stack should allow you to deploy code seemlessly from day 1. This is why Heroku is a great start for early startups, despite the drawbacks.

Having a DevOps first was THE reason we were able to focus on building the product and not invest time figuring out infrastructure.

Having spent nearly a decade of my two decade career cleaning up totally broken developer-architected systems that were so poorly coded for performance that they could barely handle 10 concurrent users, I have to strongly disagree.

It's getting better, but most developers still focus on making it work right now over making it work right. They don't think about caching, or query optimization, or debug logging, or stateless scenarios for portability and clustering.

These are all things that need to be thought about before you start.

That said, a generic sysadmin won't be the kind of architect that is best, but they'll be thinking about more of that stuff than the devs, plus he can implement and manage your infrastructure. Seems like a win to me.

No, a full-time sysadmin is dumb idea for first hire. Your first deployment is not going to be complex and there will be a lot of moving parts. If you dev team can't understand deployment I'm not sure what your expecting your first hire sysadmin to do.

Especially when services like heroku and take alot of the heavy lifting of the start ups hands, initially anyway.

I don't think you should hire a sysadmin until you have a larger set of developers. Still, I think there's a good point here: you should think about deployment early on. Having things automated will allow your early team to spend more time writing features and less time fighting fires and hacking production configurations.

We are a small 5 programmer company, and we implemented a couple of ansible scripts to deploy our applications a 2-3 month project.

For us, at least, a full-time sysadmin is overkill, and I would expect any decent programmer to be able to implement this himself as part of his basic skill set, or to be able to learn it quickly.

I imagine it depends on your focus. Want to pay someone else to do your sysadmin for you and take advantage of the economies of scale - use a PaaS provider and makes plans for when you outgrow it.

Want your developers spending time debugging CM scripts and install quirks? "Full stack" developers it is.

Want developers developing, and your application doesn't fit into a PaaS architecture? You absolutely need to hire a sysadmin.

Your first hire should be the person who fills in the missing gaps in your product.

For example, good product, code is done, bad design => hire designer.

yes!

I'm also going to have to disagree with this one. There are plenty of frameworks (Ruby on Rails) and companies (Engine Yard, Heroku) that take care of these issues for you on a time-share basis (ie. 20 companies use them, they hire 5 sysadmins). I would say if you're not spending 7 figures on server costs a year you absolutely do not need a personal sysadmin.

Disagree with the seven figures number. Six figures, sure. Seven is just silly. You can have a lot of gear or servers for mid/low six figures.

I really disagree with this. For your first hire, a full time sysadmin who cannot work on developing the product itself is just as useless as hiring a developer who cannot ship the code.

I think what this article is trying to get at is that your fire hire should be a full-stack engineer who can also play the DevOps role when needed.

I think saying a sysadmin 'cannot work on developing the product itself', is silly. I think it's a given that anyone joining at such an early stage of a company is going to be developing the product. You will definitely need a sysadmin that can develop the product and move fast. Your architecture is part of the product, and the sysadmin should help with the design and implementation of the architecture, as well as possibly parts of the codebase going onto that architecture.

Why was the title changed from "Should" to "Could"?

While I don't agree with the author's argument, he's at least making a specific point.

"Could", on the other hand, reduces it to being meaningless. Your First Hire Could Be a Weirkeeper. Or a milliner. Or a janitor. Or, you know, anything.

The post was changed by a moderator. Our blog post title is what we stand by

Sorry if I wasn't clear; that's what I assumed happened.

Which seems an awfully silly thing for a moderator to do since it effectively guts the premise of the article. Glad to see it's been changed back.

>Your First Hire Should be a Sysadmin (an imperative and definitive statement)

Written by a Sysadmin....

He goes on to say how he built infrastructure and how it benefited the company but... He was far from the first hire at Crowdtilt - which invalidates the whole premise. why compromise the whole integrity of the article with a false premise and untrue suggestion.

The title is strong, yes (on purpose). I'm not certain how you came to the conclusion that I was far from the first hire at Crowdtilt. Your statement is simply false.

Reply on news.ycombinator.com