Three CISOs walk into a startup | Hacker News Reader

Three CISOs walk into a startup

lastweekasavciso.com

75 pointsmooreds61 comments

CISO - and IS overall - is actually much more people-focused than most realize. You'd think you're hacking away and/or find vulnerabilities all the time, but in reality - at least in my experience -, you spend most of your times fixing the easy stuff and selling the measures to both management and your users. Quite a few people I know (including me) tried to get into IS with a good technical skillset, but it turned out that what we really needed was patience and people knowledge.

Of course there are pentesting companies that provide a far more hacker-like job, but even then, it's still a lot of report writing and explaining.

My experience dealing with IS professionals is they're either spending all their time writing documentation, or they're checking boxes in automated scan tools, or they're uploading results into centralized systems.

It always seemed to me that a tiny fraction of a percent of the entire IS workforce is actually involved in defining strategy. Most of them just become experts at bureaucracy compliance.

IS has more in common with being an accountant than being an IT professional, in my unjustifiably harsh opinion.

You're not wrong, but ironically this often depends on the CISO!

But yeah very few sec jobs are actually doing real hacking or dev. A much more common position is more like a PM mixed with a lawyer: Reading through mountains of compliance regulations, talking to engineering to find out if we comply, and documenting the compliance. If we don't comply, you'll often create high-level tickets and track engineering progress.

Then of course there's the tools you run and the boxes you check. Back in the day this often involved writing scripts if not full applications, but nowadays there's a SaaS for everything and everything is a REST API (or GraphQL) and very, very few CISOs will invest in custom tools. The only exception I saw was at a company where we were developing a layer 7 protocol on top of UDP. It was a lot easier to justify since by definition no tools existed for this protocol, but this is super rare.

Infosec people are, by and large, auditors. Someone once told me you can understand auditors better if you assume they hate you for no reason.

This advise has served me well.

That's a pretty large generalization. There's a wide range of careers under the umbrella of what gets called Infosec.

Some that really don't fit the definition of "auditor"

- Vulnerability research - Pentesting - Incident Reponse - Threat Hunting

Even the type of Infosec work which can end up being closer to audit should have a different focus, if done well.

Yes, they "should", but they rarely do.

My comment was largely directed at the type of infosec person any other IT person is most likely to interact with.

I'm not sure I'd agree with "rarely". There is some cross-over with auditing, but the infosec people (in my experience) are generally more risk focused than auditors who are more "rules" focused.

Also I've worked with quite a few security people who have strong technology backgrounds and have worked in IT before moving to security, so definitely understand the frustrations of operational teams.

In a way this is a very nice explanation of something we called "offensive mind" :)

Auditors don’t hate you for no reason. They have the tricky job for being (formally) responsible for signing off on practices they know too little about, get shitty documentation for while not getting enough time to understand. To make matters worse they get paid a lot, but get paid by the people they audit whom fire them for withholding signatures. Looking at firms they they think “why do they always send me the wrong documents, at the wrong time while withholding the documents I need and only bringing them up when I’ve escalated the matter to the board”. It’s a crazy relationship from both sides.

Honestly, as a free market person with 15 years working with auditors I sometimes think that external audits need less market and more intervention. Then I remember most government oversight is even less effective. Every government agency (EU / NL) I know of is both awed by a big-4 rubber stamp and an avid consumer of big-4 hours. My humble conclusion is that oversight is a hard human problem.

> responsible for signing off on practices they know too little about...

Ok, well, there's the problem. Or at least most of it. No one should be even considered to audit a system unless they understand how it works.

Aand there's a problem :) So if you consider the size of most audit teams relative to the size of the IT departments they audit, how do the auditors become subject matter experts in all the different types of technology in use in the organization?

It's a tricky problem. One option is to audit against "known standards" but that's often a horrible hack as the standards themselves need maintained and with more cutting edge technologies (think the rise of cloud native) the standards may not even have been written yet (or are hard to maintain as the underlying tech evolves so quickly)

Another option is to rely on external companies (e.g. the Big-4) but they don't have any more ways of getting large pools of SMEs in all the different technologies either (despite what their sales people might tell you).

I work with red teams and auditors as a blue team dev, and I don't think the documentation is necessarily because of excessive bureaucracy. It's because modern infrastructure is complicated as hell and it's several full time jobs just to translate what the red team finds into a systemic view of current weaknesses, and then to translate that into a plan that the inevitably outstretched blue team can actually act on. Writing things down is a critical part of making the problem tractable.

In my experience, being able to think clearly end to end is the hardest part of the whole operation, and the lack of that capability is the reason it feels like paperwork and mindless checklists.

Securing a system is like flying an airplane - if one thing goes wrong, the whole thing is fucked. At large scale, a good CISO will be more of a process fiend than a whitehat hacker, the same way that Jetpack Guy is probably not the guy you want piloting your vacation flight to Maui.

Can confirm.

I've been interested in security since I was a teenager on the wild west internet in the 90s. I went into Computer Science and subsequently gained work as a software engineer, but stayed interested in security. I did my masters in Cybersecurity and interviewed for a number of security focused jobs (including pen test jobs), but I ended up taking none of them because the majority of them were not writing code. Usually we were either running existing tools (nobody wants to invest in developing their own tools, but that's another story), or doing code reviews of horrendously nasty code. I ended up staying in software engineering and the security work usually finds me.

> You'd think you're hacking away and/or find vulnerabilities all the time, but in reality - at least in my experience -, you spend most of your times fixing the easy stuff and selling the measures to both management and your users.

As I see it, whatever technical measures are put in place people in the org will always need access to data at some level. With that in mind, it's about educating users so they know what's risky, what best practices are etc. Easier said than done of course!

>CISO - and IS overall - is actually much more people-focused than most realize. You'd think you're hacking away and/or find vulnerabilities all the time

Finding vulnerabilities is just a specialized type of software QA that sometimes looks and acts more like an internal auditor.

I think "the security community" is too caught up thinking that hacking is the end-all-be-all. It prevents them from realizing they're just another category of business analyst who's job is to complain to the development (or ops) team that their requirements should be prioritized over every other stakeholder's requirements.

Pentesting involves report writing 'cause you've got to tell people what you did somehow :)

A better model (IMO) is for the pentester to get onboarded into the bug/defect tracking system and put things there. Pretty much every pentester I know would prefer this to writing long reports :P

Those reports tend to be 90% boilerplate.

Depends hugely on the testing team and scope :) I've been a customer and tester over the years and I've seen everything from obvious scanner output, through to actually good bespoke content.

There is going to be some boilerplate though, especially on lower price testing, that's one of the reason why testers would prefer just to file bugs.

Fair enough, there are exceptions.

It always depends on what your client knows. For most clients you could probably cut 30-70% of boilerplate, it's just that you don't know what. I've seen people who could exploit an unpatched Linux box in minutes but would have a hard time exploiting CRSF vulnerabilities and also the other way around.

Of course, some people sell you a lot of paper for a lot of money, but even the good companies include a lot of you-know-if-you-can-skip-it boilerplate so that the client understands the report. At least that's my experience.

I've found that "Do not be Chicken Little" is far and away the most difficult thing to do in startups. This is because the CEO, the Product org, and similar are often deeply invested in the idea that they just need a little help and maybe some IR. Helping them understand an accurate assessment, no matter how well-communicated and at the appropriate level of technical detail for the audience, will often come as a rude awakening. In short, Chicken Little-ness is often a matter of the severity of the gap between expectations and findings.

Have a clearly communicated, well explained, prioritized, and data-driven plan is valuable. It's especially valuable for the next CISO when the third CISO gets fired for costing too much engineer-time as PMs plead for them to bear in mind the impact on team backlogs of patching.

There isn't always a good way to do it right. There's sometimes not any real negotiating space to be more than a figurehead who handles the occasional incident.

> There's sometimes not any real negotiating space to be more than a figurehead who handles the occasional incident.

Should you run from those situations, then? Or is there still valuable experience (or benefit to the company) if you are a figurehead?

If you are someone junior and ambitious and looking to build your career, it's worth asking yourself if the first real incident will lead to leadership learning and adapting or you becoming the Chief Sacrificial/Scapegoat Officer. Do you see clear evidence of them learning painful lessons in other areas, or are they true believers who ignore evidence in favor of convictions?

If you are being paid enough to accomplish very little for years, it might be worth considering. In cash, because there's an elevated chance of death-by-incident and you getting nothing for your equity.

Much of what you will learn in an environment this unhealthy may prove maladaptive in a healthier one. If you genuinely believe you can win out and make real progress, even after correcting for sunk cost fallacies, it may be worth it. Being able to build an effective security program from scratch in an hostile environment is a huge accomplishment... but also a big if.

I watched a CISO come into a startup and take it from a laughing stock (outright fraudulant PCI audits and zero policies by predecessors) to world class compliance.

There was a lot of bull-dogging, confrontation, arguing, political back-stabbing, fights with nearly every developer. It was chaos. A lot of it was funny if you were the one sitting back eating popcorn. Most of the big fights involved him trying to get shit done while navigating executive level politics and rediculous antics.

This guy is a very rash person and can be abrasive if he "thinks you're dumb" but I don't know if it would have gotten done at the level he was able to do it any other way. It was literally war 24/7 for years.

How did this person not get fired? Did they have board/executive backing? Significant ownership? Was it life or death for the startup to achieve "world class compliance"?

This is the million dollar question. I think there are cases where executives take this kind of aggression like it's tough love and so they argue with the warlike technical folks but ultimately concur, like old habits dying hard

Possibly due to this person having exceptional stakeholder management skills.

Speaking as someone who works in management. Knowing the difference between key stakeholders that you cannot afford to lose, and ones that are disposable are essential.

> Was it life or death for the startup to achieve "world class compliance"?

Most likely.

Nobody cares about security until they lose actual money because of it.

After that, your CISO now has the budget and importance defined by how much money the company got taken for.

There were factions and he was in one that had his back. The faction that belonged to a new CEO who brought with him a CTO. Most of the antics and cringy back stabbing flowed from the CFO who was brought in by other board members.

After we got acquired he started openly insulting management and the new company to their face in meetings in front of everyone. He did it on purpose but it still took quite a bit to get fired.

Sure there is a good way to do it right. Yet most IS teams got drawn too much into the polar reaction of either chicken little or boilerplate.

From a client standpoint, which came to me as a no surprise but I've learned the hard way for something so obvious, is that they don't understand nor care the whole explanation, articulation of the security issue. All they wanted is you to take care it, don't waste everybody's time by only coming in to find security problem at the worst timing and making impulsive decision as a IS methodology.

This calls for build security by design, merge the lifecycle of security management into the modern software engineering, some refer that as DevSecOps, which sounds bolts-on so I doubt that's all there is to it. The software engineering never has given security the weight it deserves.

Why now and why this has never been the case since the beginning? Probably because most IS professional are from auditors, operation etc. background. Very few brains are from software engineering background, this is also different from a mere programming background. Script kiddy is not software engineering.

Contrary to the article's claim, what it needs is system thinking, STEM brain. People skill will not fix it, only to make it go away.

You're absolutely right. The key idea that's badly needed here is to build secure systems by design, building in security-critical features and design aspects from the start. As you correctly say, this is the best way to do this and addresses what the clients want - just get it done.

Having personally experienced several different aspects of this process, I've found that it's almost always much easier said than done. It's been my experience that clients/executives don't just want things done. They want security, but without impacting product roadmaps, development timelines, or design choices. Engineering rarely appreciates someone else trying to tell them how to make design tradeoffs. Product wants security, but this commitment often wavers in the face of increased development time or an ongoing vulnerability management effort. There's a target date to hit, and maybe security isn't viewed as part of the MVP... can't things just be done when they're shipped to prod?

How do I, or another security specialist, get leadership of other parts of the organization to enable me to take care of it without wasting everybody's time? Or anybody's time? Generally a lot of people skills, negotiation, and prioritization effort. Security work, no matter how integrated into the overall effort or DevSecOps the implementation, is never completely without cost.

Again, you're completely correct. The best way to do what the client wants and just take care of things is to use systems thinking and merge security work into every part of the product lifecycle. Do you think it is perhaps possible that there might be a place for human-wrangling skills in the complex human-computer system that controls how software is developed?

>The first CISO comes home, complains about what they have seen at the company and is worried nothing will get done, calls the CEO and complains about everything they saw and how much of a mess it was. They get fired on the spot.

>The third CISO comes home, calls her boss, the CEO, and tells her there is a lot of work to be done and to be prepared, but that it’s an excellent opportunity to get it right. The CEO backs them up.

Don't forget to be honest. I know there's this cult of positivity crap floating around, but the best CISO in this scenario calls the CEO, gives them the honest grim assessment of the current situation, says what can be done to fix it, and is eager to get to work.

> Don't forget to be honest. I know there's this cult of positivity crap floating around, but the best CISO in this scenario calls the CEO, gives them the honest grim assessment of the current situation, says what can be done to fix it, and is eager to get to work.

The job of a CISO is not to convey problems to the CEO but to solve problems. The first CISO didn't understand their role.

No not really. The CEO has to balance between the "fix nothing" CTO and "fix everything" CISO. I'm exaggerating a bit of course, in reality it's a blend, but in general the CTO and CISO often have opposite/incompatible goals. The CTO wants to ship fast and ship often, which the CISO doesn't want to ship until everything is done. The CEO ultimately needs to decide where to put the line, and a good CISO will execute under that and escalate when needed.

I don't see it as "cult of positivity crap".

>honest grim assessment of the current situation, says what can be done to fix it, and is eager to get to work.

Complaining is 1 without 2 and 3. It isn't complaining as long as 2 and 3 are there.

You can have 1 & 3. The fix isn't always immediately known, but doesn't mean that its concern doesn't need to be raised.

The key difference between 1 & 3 is how the concern was framed. For former sees it as a problem and issue with no immediate solutions, whereas the latter sees it as a unique opportunity for improvement. Most CEOs embrace the "growth mindset" and would invest in individuals that possess that quality.

You might not have a full plan but you would have something. Even if it is just "my top people are on it, and full plan expected by ...".

The thrust of the book I've been reading, "Why CISOs Fail," is that being grim is probably the main mistake CISOs make. It's their job to find opportunities that fit within the company's existing priorities to improve the security posture, not to be delivering bad news in huge doses all the time. Besides, the CISO is there not to do the work but to be a partner at the table.

Having been a CISO, I would also add that this really depends on the company.

Some companies want to be technically secure (Google, Facebook, etc.), others want to be compliant to regulations (do the basics, check the box and move on) and a few just want to have a figure head for IT security to fire when things go wrong. So finding the right company that fits you and understanding their culture is important.

If you are hardcore CS/ECE and very technical, you'll want to work for the first type of company. If you did business/management/audit then perhaps the second type of company. I would not encourage anyone to work for the third type.

Just my personal opinion. Hope it helps someone.

I've found that very few companies in the third category understand that they are. As a result, they are unable to admit it and a security person only finds out when a need to bid for resources arises. Then they learn that the company sincerely believed that hiring the security person is all the resources required.

Six months later, the startup gets wrecked when attackers inject ransomware into their product and it infects 28% of their customer base.

The first two CISOs are currently employed. The third goes and grabs some coffee with the CEO as they discuss career prospects.

You miss 100% of the shots you don't take (or something like that).

I'm definitely on the weird side of weird, but I love a challenge like that.

The only exception is if the CEO just needs a CISO to check a compliance box but doesn't really value or support them. Someone else can title mine that opportunity.

Which places ransomware about 37th on the list of existential threats the startup is deciding whether or not to hedge against.

If there is anyone out there who would have to google it as well: CISO means ”Chief information security officer”

I'm afraid that in quite a few organizations, CISO stands for "Chief Incident Scapegoat Officer".

The 3rd CISO sounds just like the 1st with extra buzzwords.

They are. We get exactly one unit of information about #1, that they "complain[ed]".

If we don't treat the assessments of #1 and #3 conversational prowess as tautological, there is no information to be had. Why isn't #1 the straight shooter and #3 is an suspected silver-tongue bullshitter?

No reason at all, because the whole article is just-so or non-falsifiable.

These aphorisms could apply to anyone whose role it is to assess the state of things and recommend action.

I worked as VP of Engineering (we had no CTO) with a CISO at a hypergrowth unicorn startup. When I started we were 30 employees (series A), when I left we were 350+ (series D). He did many things well but the best thing he did was be pragmatic about what we can tackle today to move the needle.

He did not come out with a 70 page document saying this needs to get done period. He understood that product still needed to be built, and he understood what to attack first to satisfy auditors and B2B enterprise clients.

A shitty CISO would have collapsed under all that pressure. It was 14 months of hard work and living in god forsaken google docs of rules after rules after rules.

Exactly. The "fix nothing" and "fix everything" are easy positions that you don't need anyone for; the whole point of a CISO is to identify the things that can be fixed when looked at from a cost/benefit analysis (and often the "benefit" is on the side of audits and client requirements, not on "percentage chance we get hacked").

I would also add to this do not be a box ticker. Yes we need policies and certifications but a piece will not make you secure. You need a culture of security, that means both paper AND technical.

Also, remember that you can achieve the best outcome when you adapt best practices to the specific context. Just because every other place you've been at does it a given way doesn't mean that is the right solution for your new company. Yes, solutions have to work and meet the requirements, but no they don't need to use the same products or techniques you're used to.

The more you can adapt security to the way people work the more effective it will be.

I appreciate CISO's that keep up with the technical side of things. They're hard to get b.s. past them.

The measured risk part is important. A place can look like a shitshow and still have all the right security controls. There are places that fire people for clicking on phishing links or say stuff like "we don't collect logs because it is a liability" (as in, get pwned is cheaper lol). Or an imbalanced/outdated/perimeter-centric mindset. I mention these because it comes from the top, doesn't matter if you hire the best hackers in the world, you're just burning money that way.

The premise of this post seems odd to me. Are there really a lot of high level employees who think they're being hired just to point out problems? Similar to how nobody cares about your half though out startup idea, pointing out problems and then hanging up the phone isn't going to get you very far in most situations.

Understanding problems is a good start, but devising a feasible way to address them, and executing on that approach is how you succeed at, well, all kinds of things; CISO problems included.

Typical case: the fourth CISO explains what to do, like the third, and the CEO tells them that the resources aren't available and that in any case troublemakers don't fare well.

The second best CISO is the one that has never had the organization breached under their watch.

The best CISO is the one who got breached and managed the incident in good form.

A CISO can either be the sheriff who's there reign in the wild west, or a coach that can help the team achieve the goals they need to achieve in a safe way.

Almost all CISOs I've met fancy themselves as the sheriff.

Very few that I've met have the background or temperament to understand what the team is trying to achieve, and then help them make the necessary adjustments that help them achieve those goals safely.

It's hard for me to imagine any manager being so lacking in awareness as to take a bunch of complaints to the CEO. You take solutions to the CEO, or maybe a few options if you really have to. My experience as a CISO is that most processes and technology will have security deficiencies and changes will be slow and incremental at best.

> The CEO backs them up. lulz. More likely, the CEO says "as long is it doesn't get in the way of our quarterly sales goals, then i really don't give a sh#* and you can do whatever you want."

But sure, we can all dream.

Great article format, to be reproduced in different fields

Reply on news.ycombinator.com