Ask HN: Hacking concerns: does it make a difference which OS I run my SAAS on?
So I'm launching a new SAAS startup.
Does it make any difference from a security perspective if I use Linux or FreeBSD or something else?
Does it make any difference from a security perspective if I use Linux or FreeBSD or something else?
Make sure you're able to do firedrills like e.g. upgrading the version of Nginx or OpenSSL which you use in response to a vulnerability in them getting discovered. We've been doing a lot of that these last two years, and it probably won't get any less common.
ps. Make sure you have source code and database backups happening on a regular basis. If the server dies tonight, are you able to restore it to a point in time? What about getting the application stack installed quickly? Think about these things.
The reason WordPress tends to be root city is that there are a great many installs out there, of which many have never had step one done to harden it, and were last given security updates ~ when they were installed.
I would still outsource WordPress, but that's mostly in the vein of "the cost of outsourcing is less than the cost of me having to think about it one hour a month, while 1 hr/mo is a reasonable floor for the time cost function, but the actual value will probably exceed that at least once in a given year."
A WPEngine subscription costs $30 to "I don't care, why are you wasting my time with numbers this small." Developer time costs between $75/hr and "everything is on fire and you can only put out one fire at a time." Ergo it makes economic sense to configure nginx / DNS once, then outsource further complications to paid external support.
If you're looking at all Internet-facing servers in aggregate, yes, but if you know what you're doing then the odds of getting owned by a random botnet are very slim. Assuming you have a good security posture, your risks are dedicated and intelligent attackers and 0-days, not botnets scanning for low hanging fruit.
Is there anywhere you can check how good or bad a particular CMS or framework is regarding security?
I can never fathom why Joomla is so popular. It's horrendous to work with. Also, the attention given to security by the majority of the plugin development community, and even Joomla's core dev team is minuscule.
Web service software is not secure unless you patch it rapidly. There is no silver bullet. Last year there was a big security hole in Drupal, which we patched. Much like the ones that routinely turn up in Wordpress, which they patched. Much like the ones that happened, what, three or four times to Rails in the last few years? Which are like the ones they found in OpenSSL, which are like the ones they found in bash - bash, which is not supposed to be a web service! But which is nonetheless connected to the web in lo these many ways.
You have to watch for patches and apply the patches. Ubuntu gets six of them every week. There are tools that apply them automatically, or with one line of typing (which rapidly ends up in the autocomplete buffer, let me tell you), and server reboots are only needed once a week or so, but you have to do it. You have to spot the patches and you have to apply them. It is like brushing your teeth, with the occasional excitement of an urgent tooth-brushing emergency. It's mostly boring.
You must also have a plan to recover if a patch doesn't get applied in time, because these patches can be hard to write and can take several tries, and because you won't always be perfectly fast at getting your chores done. So you need to keep your family jewels away from the Apache server process, preferably in an alternate universe.
If you are a company, this stuff graduates from boring to "hard", because how do you know that the employee who swore to do the upgrades every week didn't forget for six critical days? But mostly it is just about patience and routine. Routine turns out to be surprisingly hard.
What's more I think you hit on exactly the reasons why: I'm much more likely to screw something up by misconfiguration or lack of attention than a team of security and maintenance people at a company whose sole job is to make sure that I don't get burnt by these things.
OP is trying to get a SAAS off the ground and not (presumably) doing something with handling distro security and server maintenance - for the minimal cost difference in getting started it's hard to see how the time+effort difference in being your own sysadmin is worth it.
For protection from cyber crime, you might be better off paying an insurance company than Heroku prices.
And it's this total cost of running a service (or at the very least the opportunity cost of not working on things essential to your business and the risk of your site getting screwed by somebody malicious) that I'm trying to convey.
For a business, $240/year to get a SSL terminating load balancer maintained by a dedicated security sysadmin team to sit in front of their app looks like a heckuva deal.
There is just so much work that is involved in keeping a system properly up and secure that unless you're valuing your time at near zero dollars an hour, or you have so much traffic and cash coming in that you can get a dedicated person to handle it Heroku makes monetary sense.
For someone competent with sysadmin work, I think it's maybe worth the time. It's not like for any hour not spent on my systems I'd otherwise be earning $200 automatically.
My message would just be for people to not be intimidated by the concept of something being so much work that it's not worth your time. I saw that argument for years with email as apparently conventional wisdom. I've been hosting my own email for years now with about 3 hours of upfront work and (almost) none thereafter.
The symptom of any such problem is that email starts falling into the garbage, perhaps silently. But if you are watching your stats like a hawk (a costly and very boring hobby ) - or, much more likely, if your customers start screaming about the very-real high costs of lost email - you will figure that out within a few days, so it might not take more than another day to figure out how to get yourself off the blacklist (a manual process) and how to apologize to customers for the lost email (a manual process). But, if you value your time at in any way, your first encounter with that problem will be your last, because email services are cheap and easy. I have yet to pay for mine; in small quantities Mailchimp, Mailgun et al are free. Indeed, given that low cost, why wait to get burned before you stop playing with fire, even if the fire looks cute and innocent and warm right now?
[1] Actually, I'm lying: A much more serious problem which occurs at any volume is that email is another piece of the security perimeter. Even if mail servers had almost no configuration and a history of bug-free operation - which they do not, I am joking, and the joke is not very funny - one of the basic rules of security is to put as much distance as possible between your data and services which connect to outside machines. Mailgun is free in small quantities and runs outside your entire datacenter. The math kind of does itself.
I agree with you for transactional email.
For any DIY project there's a limit where you really should just pay someone. But for instance I run wrote my own Stripe-based invoice generation, subscription management, and dunning vs paying for a service. It works for me for now. If one day my little app breaks down if/when I have thousands of customers then I'll outsource it.
I run my own private Git repo vs paying $7 to github. Etc.
Edit: to add, I am selling software I wrote basically from scratch (and there's no way around that). Like a comment way up above, my biggest risk is in that stack. I suppose with the uber-paranoid mindset I should have outsourced my own development work. At the end of the day, someone like me is responsible for something important. It's not life or death with my software so at least I got that.
I disagree with yeukhon. You don't need to hire someone just because you asked that question. Unless you are storing highly sensitive data, then I'll leave that up to you.
Just remember to constantly update your server.
There are two types of Linux OSes in general: versioned release and rolling release, which I will explain below.
When choosing a versioned release distro you need to pick something that you feel comfortable you can update on a regular basis without breaking your application dependencies AND plan for the eventual migration to the next versioned release of the distro. Any release you choose will be supported for several years with security updates and then fazed out. When that end of life date hits you must move your application to that next version. Examples of these types of distros are CentOS, Ubuntu LTS, etc.
An alternative is building your application with containers and choosing a rolling update distro. The advantage for security is two fold:
1) It makes it easier to roll out new versions of your application on a regular basis because it is possible for two versions of your app to live side-by-side. And for a new web based project most of your security problems will likely be from your web frameworks, custom code, etc not from the underlying OS. So, having a system in place to get those updates out will be critical.
2) The Linux OS can be updated on a much tighter cadence because your application isn't relying on much from the OS besides the container runtime. Rolling release server distros include Arch Linux, CoreOS, Gentoo, etc. CoreOS even fully-automates the OS update process to remove the burden of remembering to audit and apply updates.
Using a rolling release distro without containers is an option too but can be bumpier since you will regularly be going through migrations like /usr/bin/python being 2.7 and then migrating to 3.0 without you noticing.
The final thing you might consider, particularly if operations aren't your forte, is using something like Google App Engine, Heroku, etc. These systems are ideal for building a prototype and focusing on the initial minimal viable product.
disclosure: I work at CoreOS
Use something you're familiar with, and that you have a good source of information flow, by which I mean, subscribe to security updates (including for whatever stack you put on your server), and certainly have a test server simply for playing with things, getting used to functionality.
I have a shell app on my phone, which I love, so I can log in at a moment's notice, wherever I am, just to check things out or apply a change.
The Linode and DO tutorials are pretty nice, and cover the vast amount of eventualities. If you're unsure which to choose, go with the one that has more documentation for your use case. DevOps is not cloak-and-dagger mysticism; having control and understanding of your stack is, imho, far more important than choosing between a distro to run it on. Just make sure your server is locked down and that you regularly apply updates (that you've tested on your test server).
At least now the poster has multiple perspectives on the matter.
Yes and no. Running a business is different. Sure it may be too early to hire someone right now, but not the worst thing to think of. If OP can secure funding, get someone who has good knowledge with system and security. It's important. I am jack-of-all-trade to many, but as a devops I can't do everything at once.
> storing some web analytics and personal finance information
Fair. My company's finance data would be under compliance regulation, but even as a SaaS to provide easy-access easy-to-understand web analytics means you are going to have access to customer's data, or the customer is passing you data that the customer doesn't wish to share with others. Take an analytic SaaS to see your AWS usage. That requires a separate READ-ONLY access to your AWS account.
Twitter recently dropped employee access to internal data. People suggest this was done to avoid insider trading. I don't know if that was the real intent, but let's take that as a yes, then web analytic is just sensitive as one's financial information.
Security is not easy, just like writing code is not easy. I am suggesting OP to do some security audit before the launch. If OP ever hire someone to join the startup, try to secure a seat for someone good at security. FYI, I did some whitehat for some startup and had found numerous serious security bugs (no I am not here for hire, no time and definitely NOT as good as others, and I am 100% happy with my current job...). Just my genuine advice: don't just focus on features :)
But as soon as you can afford it and/or as soon as you get big enough, you should have at least one FTE who focuses purely on security. This applies to any company with an Internet presence.
With that in mind, you must, must, must do a few things:
1. Get really, really good at rollback and recovery. Make sure you can completely rebuild your operation from a clean system in as short a time as possible, with no data loss. Certain types of compromise will require this, it will be the cleanest, safest way to recover.
2. #1 requires you to be really good at backups. Unless you have experience here, they're harder than you think.
3. Notice that #1 starts with "rollback": You will occasionally be your own worst enemy, at least until your release processes get really, really good. Every now and again you will apply something that breaks something else. Being able to get back to a known good state quickly and easily will save you countless hours and much stress.
4. Find an experienced penetration tester that you trust implicitly. Have them review your operational architecture; they will find flaws (that's just on paper). Have them review your hardware and software configurations; they will find flaws. Listen to their proposals for risk mitigation. Once you have a system that passes their paper attack, have them actually kick at it, from three different places: one, from the outside, from the Internet everyone comes from; two, from just inside, from within your first tier, between outside interfaces to the world and app logic; three, from deep within, behind the app tier (using test data, of course). The three reports should be very different, with increasing levels of access and vulnerability as they go (because you are letting them past the mitigations). This will give you considerable information about how at risk you are.
(I know quite a good one, does this sort of thing professionally for government, private sector, the military, police forces, etc. He'll break your stuff. And tell you how to make it better.)
5. #4 implies, strongly implies, that you have a restricted access test environment that matches your operational environment as closely as possible, with two exceptions: one, no production data; two, it may be a patch or two or a major update ahead of production, since it's where you will soak changes to make sure they work properly and break nothing.
6. Managing #5, #3, and #1 means you must, must, must understand and get good at change management. And risk assessment and risk management.
IT Security is mostly about governance and management. If you focus on technology first, you've already lost.