The SaaS CTO Security Checklist Redux | Hacker News Reader

The SaaS CTO Security Checklist Redux

goldfiglabs.com

395 pointsvikrum73 comments

It's a good list, but like most (all?) of these lists it doesn't offer too much advice on organizational best practices.

IMO a security program which cannot justify its own worth to others in the organization is incomplete.

Most security professionals I've met (and I've been one of them, too) assume that security is or should be everyone's top priority. They struggle to deal with people for whom security is just one of many competing concerns.

I think recognizing that security is just one of the many engineering challenges in building software is important. It must always be prioritized and budgeted like any technical debt. It can’t be ignored, but it is rarely the supreme concern for most software. We do a lot of turn key security engineering for our customers and it’s always about prioritizing the right concerns, both from an implementation perspective and from an organizational perspective. Another thing a lot of folks struggle with is knowing where your product is and where the security fights should be at a given product and organization scale.

Risk assessment and threat modeling are huge for creating such a prioritization. Whether it’s formal and in depth or informal depends again on where things are.

Beyond these concepts, well engineered software simply has less security defects, so focusing on quality and managing technical debt often means less security debt as well.

It’s down to being an immature practitioner to think security concerns are always the primary ones. I think it’s a hard skill to master though, so I am not saying that with great judgement, someone should be a champion for security at most organizations and advocate for it. I think using empathy and curiosity goes a long way to maturing as a practitioner, but it took me many years of practice.

I agree, it is absolutely a matter of judgment and is heavily dependent on the stage and specific threats a particular organization faces. It is difficult to balance product velocity with the need to protect a growing "something to lose" that the company is accumulating.

I think one of the best things we can do as security professionals is to identify or work to create security measures that have outsized ROI and advocate for those. Using battle-tested software is one, as are, I believe, measures like MFA.

I'd also submit that one of the most important things is recognizing that ROI requires a net positive return. It's not just the time required to implement a control, you also have to factor in the opportunity cost of the increased friction. I've seen way too many times infosec organizations completely ignoring that the loss outweighs the actual risk. Hyperbolic analogy, but like forbidding driving delivery routes to avoid a parking ticket.

Yep, this. OMG THIS!

There are three things that most security types don’t understand:

1. Security is a small component of risk. Impact (in dollars) x Likelihood = Risk

2. Cybersecurity insurance can change the risk equation dramatically

3. It is possible for a security remediation to kill/harm more people than the underlying insecure system that the fix is being applied to

Once you realize that security is a risk problem then you’re able to quantify that problem in dollars. Lives affected can be used as a metric too but dollars are the magical language that everyone can understand.

> Cybersecurity insurance can change the risk equation dramatically

Do they check if you implement best practices in security? I could imagine, that this will have an impact on the premium.

Yes and the audits around this change arbitrarily each year and you will be denied coverage if you are doing something otherwise competent but miss their particular flavour of the year

Fully agree that impact x likelihood = risk, although I'd extend it beyond dollars. Impact can include reputation, health, etc. Certainly these could be measured using cash-value as a proxy, but this risks chasing the wrong metric.

I also recognise that "when a measure becomes a target it ceases to be a good measure", and cash as the bottom line risks increasing this tendency.

Too idealistic, probably :)

Corps don’t care about your health, they care about you being healthy enough to bring in the dough.

And when you accept this fact, you can skip lamenting about all sorts of ethical questions and just know that everything will be a monetary decision in the end.

Using cash-value as a proxy for impact gives you many benefits. Here are two particularly good ones that are relevant to planning security work:

1. It clarifies your thinking around impact. The simple exercise of sitting down and trying to put a dollar amount on the reputation damage your organization would suffer if Bad Thing were to happen will help you better understand the likelihood and potential consequences of Bad Thing.

2. It allows for more easy comparison to the costs that will be incurred while mitigating the risk. If you think the impact of Bad Thing will be "significant reputation damage", that's bad, but it doesn't help you decide how much time, effort, and money you ought to dedicate to preventing Bad Thing.

Blindly chasing metrics can lead to pathological behaviour, for sure, so you need to balance that with strong cultural values, but overall I would say the security industry is terribly heavy on cultural values right now and pays little to no attention to measuring what actually matters to organizations.

I just went through a "risk triage" they called it for a client and while the client is more focused on being able to work AT ALL the risk triage is solely focussed on the monetary damage of losing IP and data.

Two things:

1. Security to some extent doesn't always need to impede engineering. Enterprise IT security is very much focused on removing risky behaviour, without providing any alternatives.

2. The cost to the business losing IP it cannot generate due to excessive security practices should be taken into account when doing these exercises.

I'd argue that the security part of this organization has gone "rogue" and is solely focused on justifying its own existence. It's as if the security part is more focused on reducing liability for itself than it is on actually securing systems.

That said, I still think that most organizations put too little emphasis on security, not too much. And oftentimes, too late.

EDIT: an analogy I like to use is that oftentimes these people try to build a castle surrounded by massive walls without doors or gates, but since people inside still have to get things from the outside they just dig a tunnel circumventing the whole thing. But since it wasn't approved there is no liability problem for the security part of the org.

I've found that using a risk-based approach to security requires a mature organizational framework for handling risk. A mature risk management process keeps good records of how risks are handled, what the estimated impact and likelihood are, will promote accountability, and will learn when risks are realized.

An immature one, of the sort many companies have, may have difficulty with some or all of these steps. Often the immaturity is invisible until it's too late - an impact or likelihood is far higher than estimated (and thus your quantification was wildly off and you took your estimate too seriously), there's no organizational accountability (so nobody who makes serious mistakes has to learn from it), and so on.

I don't know how many SaaS startups have mature risk management processes, but I know I've seen companies well beyond the startup stage that suffer from immature ones.

My reading of this list is that it's not designed to be a starting point for a full security programme, but a starting point for small organizations who don't even have a security function yet.

In an ideal world, every company would have dedicated resource for this and a well targeted/justified programme of work, but I think there's quite a few where responsibility falls on the CTO and there lists like this can be useful.

This list and others like it are for sure useful, but the CTO of a smaller organization still needs to be able to:

- Prioritize items on this list relative to each other.

- Prioritize items on this list relative to other work.

- Communicate the reasons for that prioritization to other members of leadership, at least.

An interesting parallel is how these similar issues are solved in other engineering domains. In civil engineering you have heavy regulation and accountability. If a building falls then the engineer is responsible and she could get a jail sentence. To prevent buildings from falling there are regulations around minimum requirements, audits, material quality, etc.

Regulation in software "does not exist" as the actual threat is minimal. If your program crashes then nobody dies ('m not talking about rockets). Therefore it always looks like security people always have temper tantrums as there is no meaningful quantifiable risk.

In our org we ended up asking pentesters to get into the systems instead of just giving us an automatically generated report that contains links to CVEs. If they cannot use existing risks to get into the system then that risk is trivial.

Good point. There are also some software solutions (picus security, etc.) that tests your validates whether your environment is exposed due to specific CVEs. It's a good way to prioritize which vulnerabilities that you should tackle first.

I think of myself as pretty security-conscious. But it's striking to me just how _long_ this list is. Even as someone who is trying to constantly think about this stuff, if I were faced with a list of this magnitude, I'd be tempted to throw in the towel and admit defeat right off the bat. It's no wonder so many SaaSes end up getting exposed. Even with a lot of best practices built in to so many cloud providers... how did we as an industry end up in a place where it's this dang hard to keep systems secure?

As a CTO of a resource constrained small startup I totally empathize with you.

I thought security checklists were long until I ran into ISO.

Edit: Typo (and coffee).

Another thing that comes to mind is the list of NIST Cybersecurity Framework’s controls. The length of these lists is a reflection of the real complexity that are inherent to computer networks.

Empathize? :)

I can empathize as well. And it gives me a greater appreciation of how most developers must feel about accessibility, which is my own specialty and passion.

I have to work with both constraints and I find accessibility standards to be easier to align with as they can become part of the SDLC pipeline (like code level testing). Securities' scope is far more all encompassing. The hard part for accessibility generally comes down to ensuring that people don't forget they need to think about it when they are designing features and especially changes.

Awareness is half the battle. Maybe some of the items in this list are beyond the scope of what’s reasonable for a given organisation at a particular size or with limited resources, but being able to identify them early and plan them into your roadmap so you can begin to address them as your product and company matures is very useful. It’s all about assessing the risk you can afford to address now and in the future against the cost of implementation and whether that’s the right balance.

> how did we as an industry end up in a place where it's this dang hard to keep systems secure?

Shipping is easy. Shipping makes the product better. Shipping gets the customers and the funding and furthers the mission. You do what it takes to ship, to get the thing working, to do what you're burning to do, and the rest is details.

Taking the time to ensure you've thought through your authentication strategy and done your key management well does not directly do any of these. It feels like a waste of time by comparison. What does it matter? You'll fix it later. You don't need bullshit corporate malware endpoint management, you only hire good professionals.

-----

It's so easy to slip into that mindset. Because you mean well, you want only the best, but you have priorities to balance. I can empathize with everyone involved in a list like this... but speaking as a security professional I ultimately prioritize the safety of the user.

The list is pretty short compared to the audits we do every year or two. I've spent the last few years getting my company up to standard and this is the (simplified version of the) process we undertook.

1. Do an initial internal audit against a (few) known security standard(s) that are relevant to your org (ISO27001, NIST, PCI etc).

2. Create a System Security Plan document that outlines what your intended security posture is for each control in the standard you are following and also recording what your current posture is. Record evidence in this document too (screenshots etc) along with an caveats and mitigations.

3. Once you have the results create a Security Risk Management Plan (SRMP) with a risk register annex for an itemised record of gaps. This document should outline what gaps you have and categorise according to the the risk they represent for your company using the risk matrix https://en.wikipedia.org/wiki/Risk_matrix

4. Once you have the SRMP you can create a roadmap starting with the highest risk items and working down. It might take years for you to get to a good point but at least this process will focus efforts and give you a plan to tackle what seems like an insurmountable volume of work.

If you read this comment and thought "We should do this" I highly recommend to contract in a security consultancy to act as your organisations CISO to help you with this process. They will provide experience, structure and knowledge that is no doubt missing from your org. Rating risks can be non-intuitive for example.

I don't think a NIST heavy documentation centric process is as applicable to private industry, particularly startups, nor does NIST itself track the state of the industry around defenses & countermeasures. Am saying that since you are using NIST language if it not apparent to all readers.

Given that NIST is tracking so many controls (1190 controls & enhancements in the current revision) it becomes hard to see the singular burning trees from the forest view that is NIST.

Each control is a tree. You have to work through it one by one but as you do that you will find many of them (50% would not be unusual) are not applicable to your business. The standard is basically a guide rail. There are things that aren’t in it, there are things that are in it but are not relevant and there are also things that are outdated and/or against prevailing industry best practices.

This list is the "brute force" security algo. Eg 'just make everything more secure'. This would have someone upgrading a door lock with a hole in an adjacent wall.

There are at least the following major factors:

* what might happen (event A,B, ...) ?

* how probable is eventi?

* what is the cost if eventi happens?

* how can one deal with eventi? (action 1, 2 ...) ?

* how much / how long does actioni take?

* what actions should we perform with the skills/resources/time we have?

My algo:

1 enumerate possible security issues

2 assign approx probability and cost to business

3 re-order by prob*cost

4 imagine ways to address issues, assign cost to address

5 decide on courses to address issues

6 for next epoch, address issues

7 goto 1

That’s a good list and will improve the security of your own software. But in a bigger team, how do you make sure everyone is following the list effectively and consistently?

A starting point would be to have a dedicated team perform steps 1-3 as a service to all teams, individual teams become responsible for steps 4-6, and management assume responsibility for step 7 as well as overall goal setting (e.g. deciding by how much they want all teams to reduce/mitigate their risk over the coming epoch).

This definitely presumes a certain company size. The company I work for has 5 employees (2 in an engineering capacity), there's zero room for dedicated teams for anything.

I've found even in a 100 person startup there's probably appetite for 1, MAYBE 2 people who are 100% dedicated to security.

Well sure, the comment that I replied to specifically asked how you might apply the heuristic in a larger organization. With just a handful of people it can be applied by one security-conscious member of the team, and at that scale it's also viable to get everyone on board with applying it consistently.

This is a great comment. There's a bit more nuance in practice, but getting people thinking and talking this way is such a strong start.

I'd suggest that the best practice list approach may be a function of the target audience (SAAS CTOs) who may well not be well versed in security.

Whilst a security professional should take an approach like the one you describe, it a) takes time and b) experience. Things like enumerating possible risks and assigning probability require a level of experience in security which may not be something a non-specialist has.

Best practice approaches are necessarily less targeted, but I think they still have value in the sense of being good starting points.

> I'd suggest that the best practice list approach may be a function of the target audience (SAAS CTOs) who may well not be well versed in security

Perhaps the best place to start is with the lack of risk aversion that most startup founders have. In my experience, your are more likely to sell a startup type on the business opportunity that being secure creates as opposed to risk. You are likely dealing with a person who cannand will go all in on pair of threes.

I don’t see any mention of domain names. Domain names should be set up properly, like implementing dnssec, dmarc, dkim, spf, and using a secure registrar. So many overlook domain security.

Any resources on this?

https://labs.bishopfox.com/tech-blog/2017/05/how-we-can-stop... is a pretty good primer on spoofing (SPF, DKIM, DMARC)

What's a good secure registrar?

I can tell you which ones not to use based on which registrars have had domain names stolen frequently.

That helps as well.

Cloudflare, but you can’t register with them, only transfer in. I buy at Namecheap, and then transfer in.

Edit: Based on Austin’s below reply it looks like you can register directly with Cloudflare now. Thanks Austin!

You actually can register with them and they promise to provide registration at cost forever: https://www.cloudflare.com/products/registrar/

Note: They released the beta fairly recently, which is why a lot of folks don’t know about it yet.

Implementing DNSSEC is definitely not a security best practice; virtually none of the best-regarded security teams have implemented it.

Just because virtually none of the best-regarded security teams haven’t implemented yet doesn’t mean that you shouldn’t implement it.

Honestly, I thinks it’s a case that most people don’t know about it, so they don’t know to implement it. The same goes for DMARC as well.

> Just because virtually none of the best-regarded security teams haven’t implemented yet doesn’t mean that you shouldn’t implement it

Unclear how it makes sense to do something allegedly security related that best-regarded security teams recommend against. Sounds like security theater.

Meanwhile, consider the failures: https://ianix.com/pub/dnssec-outages.html

Misconfiguation can cause the sites to disappear from the internet.

I haven’t seen any security teams recommend against implementing dnssec.

Interesting to see info about the failures, though. Thanks for pointing that out.

I’ve been running a ton of sites on domains with dnssec enabled and haven’t had any issues that I’m aware of.

The IETF has been trying to light a fire under organizations to deploy DNSSEC since 2008. You can take any list of popular domains --- last time I did this, I used the "Moz 500", whatever that is --- and write a simple shell loop around `host -t ds ${domain}" to see how many of those domains are signed. You'll see some! But they're overwhelmingly affiliated either with academia or with the US government, which, until 2017, mandated DNSSEC, but later rescinded the mandate.

DNSSEC has virtually no real-world commercial deployment. There have been years, I believe, when US deployment went down. It's dead. Let it lie.

I agree, we aren’t seeing as much implementation of dnssec, and a lot of other checks that when check the DNP Scores of domains.

I recently ran the DNP Scores of all domains of companies in the Fortune 500 and only about 8 percent had high scores, and we check for dnssec as part of the algorithm.

So less than 8 percent of the Fortune 500 have dnssec implemented.

(For transparency, I wrote the algo behind DNP Score.)

The numbers get even starker if you look at large tech companies, as opposed to companies across the entire F500. And large tech companies have the best security teams in the world. They've rejected DNSSEC. I'd remove it from your score; it's not a reflection of anything in the real world.

If you were to rewrite this list what are the 10 best things to do for a small company to do to protect themselves?

The principle of least privilege should be at play at every level. That can mean different things in in different technical contexts, but it’s universally applicable.

If you'll accept 7 things instead of 10, my answer to this is linked in the post (search for "seven things").

DKIM increases your chances of getting mail into an inbox. There's no "negative signal", that is, a recipient has no way of knowing emails were meant to be signed and weren't.

I know you can provide indications with DMARC, but the usual thing is that if an email is SPF verified then it's considered legitimate even when not signed. Accordingly, DKIM can help land emails but only helps security in the unusual situation where you're trying to argue something in an inbox definitely came from you.

If you're hosting user content I always recommend hosting that on another domain.

There is plenty of prior art about this on hacker news. Example: https://news.ycombinator.com/item?id=25802366

Unless they're rapidly editing this based on the thread you missed the mention. I currently see "Ensure that your domain names are protected" under "Your company." It doesn't really mention spoofing controls and deliverability, but does talk about registration and hijack prevention!

While I'm generally a fan of the Sqreen checklist that this is built on, looking over it with fresh eyes I have quite a few quibbles:

"Require 2FA wherever possible" - Given the target audience, it would be nice if this was explicit about the reason to use hardware keys (including those builtin to TouchID + chromebooks).

"Accustom your team to locking their computers" - This is good advice, but I'd recommend configuring locking on inactivity a higher leverage effort

"Hire your first security engineer" - "do we have a security roadmap? do we manage to deliver on it?" is not a good heuristic for whether you need a security hire. I'd argue that most startups will lack a formal security roadmap when they don't have dedicated security staff. For example, the linked First Round article [1] has a more actionable recommendation, with justification: "Onboard your first, full-time security hire between 30-100 employees."

"Set up a bug bounty program (NEXT)" and "Monitor your user’s suspicious activities (NEXT)" being placed before "Have a security incident response plan (LATER)"

[1] https://review.firstround.com/how-early-stage-startups-can-e...

On the security engineer item, I think this piece falls into a common trap. It regards security work as mostly technical work, to be handed to an IC who can also handle technical questions.

As the company will probably lack a decent security roadmap and be poorly positioned to develop one, the right move is generally to hire someone equipped to do so. Which is to say a Director of Security. They will need that rank for the inevitable political battles with Product and Marketing / Sales. Those early political fights will do more than you think to shape the culture for years to come.

That's why I recommend to start with a external consultancy until you are mature to bring it in house.

Previously — https://news.ycombinator.com/item?id=16615593

With Sqreen's acquisition, the list's previous home unfortunately redirects the their acquisition announcement. We're grateful that they released the list under CCA and we look forward to keeping it updated and relevant to startups on beginning their infosec journeys.

2FA on itself is good, but should not be an excuse for corporate to own my cell phone.

They want to send messages with codes to my phone? I'll live with it. But then they want me to buy a phone with more recent android version. Then they want to enforce biometrics and encryption on my phone. Then they want to remotely erase my phone?

No. If central IT wants that kind of access, they should buy me a phone.

This is the first I'd heard of RASP (runtime application self protection), so thanks for that, I think. Now I need to decide which RASP tool to use.

Just in case you weren't aware - this Checklist is adapted from the Sqreen SaaS CTO Security Checklist. Sqreen is a RASP that was recently acquired by Datadog. Keep in mind that potential bias before diving in too deep on the sales pitch!

We are experimenting with a couple of RASP providers, let me tell you that it does help with certain aspects (of finding issues) but it does not replace the other tooling we are already using.

So the costs / resources are not to be underestimated.

Good thing about many RASP solutions is that they integrate easily into our existing developers processes / tooling, so that's a big endorsement.

Company that want to become SOC2 or iso27001 can automate all of this and more with https://www.vanta.com/. This helps us a lot at https://mergify.io for our SOC2 certification.

Whenever I'm forced to use 2fa the most probable outcome is that I will get locked out when I need to log in the most. Not to mention most implementation in the wild trivially reduces to 1 factor.

I could make a case for MFA if really pressed, but M=2 is such a bad choice too.

I didn't see ransomware mentioned anywhere in there.

At best being ignorant of that risk is going to result in fiscal losses and dubious legalities wrt sanctions, at worst existential risk to the company.

It makes sense (to me) that ransomware isn't mentioned, as this is a checklist of controls - not of threats. Many of these controls do help in preventing ransomware: 2FA, "Backup, test your backups, then backup again", "Isolate assets at the network level", etc.

Oh didn't notice they mentioned backups, my mistake then.

I didn’t see ssh security mentioned maybe I missed it? I recommend, in addition to public key only authentication to also use an 2nd factor as well

SSH security becomes less important when all the machines that are accessible by SSH only expose those ports to a VPC.

It's hard to implement if you are doing machine to machine automation (for example SFTP over SSH).

I work within the Cyber Security domain, and I have to say this was a very solid list. Great work!

Reply on news.ycombinator.com