Hacker News Reader

It's IT via checklist. I can't imagine a more depressing way to go through my life. Talking to the cybersecurity people I know they all frame it like they're elite warriors who are locked in a titanic struggle with cunning adversaries. My take is...you followup on tickets generated by third party tools by filling out web forms. Yes you're getting 'probed' by Russia and China all the time but thats from botnets looking for ancient vulnerabilities. You're not single handedly keeping the barbarians back from the gates. You're paper pushers.

It might be something similar as it is in software development.

At the end of the day it is a job.

But when you part of the 1% of the good people you have your team and more leaway and potentially get called for the more critical and more interesting things.

A roommate studying for aerospace engineering described that field as "Everyone gets in because they want to work at Skunkworks and design the SR-71. In reality, 95% of graduates will spend the next 40 years optimizing the efficiency of a winglet on a 747."

Security feels similar. The edge of the spear is fascinating, exciting, challenging work.

Unfortunately, no one needs that work. What companies actually need is mind-bogglingly slow, comprehensive, steady progress and improvement of their postures.

Its so true, was very depressing when working at my first aero job

Optimizing the wingtip on a 747 actually sounds interesting, and it's the sort of thing that could meaningfully affect the world. It might even prevent more wars than the SR-71 program in terms of lessening ecological and environmental pressures.

A much worse career would be convincing regulators that new aircraft like the 737 MAX don't need any additional training. Maintaining lists of open exploits, and keeping them secret from vulnerable parties is that kind of job, where you're making the world less safe in a perversion of your ostensible goals.

Would that task keep you interested for many years? A lot of the people I know in Aerospace end up working on a single component of a larger system for so long that they lose interest and burn out. It's a very high paying job for mostly very boring work. There are always exceptions though.

> Optimizing the wingtip on a 747 actually sounds interesting

That was my first thought as a software dev - I'd love to be able to spend time _optimizing_ something rather than breaking my "stories" down to one-to-two hour "tasks" and justifying my "estimates" every morning.

Ask anyone in Aviation, checklists matter. Cybersecurity is often drudgery, but avoiding excitement is the entire point.

Currently, IT checklists are security theater. Reducing liability vs improving security.

How many orgs are transitioning to zero knowledge networks, encrypting all data at rest?

> encrypting all data at rest

That’s a common checklist item. Implementing it is of course more work than just checking the box, but ensuring it’s actually done means it’s added to a lot of different checklists.

checkmate

At the field and record level? Sufficient to guarantee privacy?

Never store PII as cleartext, akin to proper password storage.

Translucent Databases https://www.amazon.com/gp/product/1441421343

Encrypting databases, file systems, and backups remain necessary, but insufficient.

Encrypting data at rest at least is getting better if only because more systems do it by default as time goes on.

http://atulgawande.com/book/the-checklist-manifesto/

Yes, Checklist in aviation and aerospace are crucial, but at the same time, you have to avoid checklist for checklist sake.

But they also do a lot of failure response testing, simulations where you walk through a failed checklist or incidient and how you would response. Cypersecurity does pen testing and phishing attempts, but how about dry runs where you act as if you are compromised and everyone runs a "fire drill" scenario?

If you're looking for a career with literally no bureaucratic overhead--no checklists, no tickets, no paper pushing, then IT is not the field for you. Every IT field has bureaucratic overhead. Yes, beginners rely on checklists more than experienced people do, but that's the same for any field.

Don't be so hard on checklists :) the bigger problem is applying ill conceived checklists no?

I suspect the underlying psychological tension is between "learn-design-create" versus "obey-follow playbooks-be reliable" People like me took decades to stop fighting the latter. Security is "for those that pay" in most cases, which also can set up some social tensions for those who consider larger social issues. make sense?

On one hand, the security checklists are laughably insufficient, you can tick all the boxes and still have systems with as many holes as Swiss cheese.

On the other hand, so many real breaches have happened because very basic things weren't done, and a basic checklist would have shown that they aren't done. But of course, that checklist would not cause the organization to provide the resources and motivation to actually fix the issues.

The big problem with checklists is that organizations inherently don't really want to invest to fix these problems, they have other priorities, and if someone else forces a checklist on them, then they often will explicitly prioritize ticking off the boxes (with any caveats they can negotiate or hide) at the expense of actual security.

I've worked in cybersecurity for ~7 years, and what you said is accurate. I'm actually switching to a government job and taking a big pay cut because the field is pretty miserable to work in for the most part. I think I've had one cybersecurity job that was actually enjoyable, and that was a Fortune 500 customer that saw the value in keeping their company safe, so their only limitations on our activity was no social engineering, and don't break anything. They knew that real hackers won't follow checklists or a strict RoE, so they didn't want us to either.

The field is also full of people who switched careers to make more money, so they blow minor security issues out of proportion to try and justify their paycheck. It's really frustrating, because then management thinks you aren't doing anything if you don't do the same.

If you want to work in cybersecurity, I think your job as a software developer would actually be very valuable. The biggest problem we have, besides management thinking we're a waste of money, is having too much data, and no good way to go through it. Dedicated cybersecurity tools are pretty awful, and very expensive. If I had any professional experience in software development, that's what I'd be focused on. There's only a handful of automation tools used by most of the industry, and they are only used because they're the only option, not because they're particularly good.

>Dedicated cybersecurity tools are pretty awful, and very expensive

Can you give a few examples of these?

Is there somewhere already extant I can read about the shortcomings of the available tools, or would I need to do some networking with people to find that information?

I've considered making tools of my own for some things, but knowing what would actually be useful to people doing real work would be a good motivator.

I did it for 3.5 years and then I jumped ship back to just regular development. When I started, we were actually really great at discovering and solving actual issues. Our VP got replaced with someone who wanted us to only work on compliance checklists, and the joy I found in that job went from 100 to 0 in an instant.

Yeah this is largely accurate but there are more options than that.

There's e.g. product security, which can be a bit less nihilistic. You take your security knowledge and use that to influence application design and implementation process so the result has fewer issues. At the tippity top end this can mean adding e2e, differential privacy, suppressing harmful features, OS security improvements - work that can meaningfully improve security or privacy for millions of people.

There's all sorts of forensics and IR, malware analysis and RE. Exploit dev, scanner monkeying, bug triage, just all sorts.

A lot of pentesters manage to not hate their lives by doing #3 and focusing on the interesting problems while not caring that the customer just wants a compliance input.

Suck levels vary a lot by firm. Just like dev jobs can be meaningful and intellectually fulfilling OR meaningless ticket punching.

There are a few bespoke consulting firms that actually try to improve their clients security beyond checklist whack-a-mole and automated scanning. Annoyingly bunch of charlatan firms pretend to do this, but just toss an intern with a scanner at the customer and/or double book their staff so they don’t have time to think beyond the basics. This lets them always underbid the firms that do honest work.

The problem is that its hard to measure if someone had deep thoughts about the security of a system vs checked a box as it can take years before you notice they did nothing (eg: hired another firm the next time that found a pile of issues).

Disclaimer: I work for one of the rare non-checklist/scanner firms.

The pentesting industry has a big problem with being a good "market for lemons".

It's very hard for customers to differentiate the good and bad companies, without having their own internal expertise, and even then you need to go down the line of getting named testers and speaking to each one.

Another problem is with how many/most pentest companies report, which is by exception. There's no requirement to state all the tests they did, just the results, so it's hard to tell the difference between "we've got a good system and they didn't find much" and "they didn't do good work and missed things"

Our reports always have a section on things we looked at or for. Doing so takes a majority of the time it takes to write the report. Since it adds to the cost, firms that don’t do this can underbid the ones that do.

For slimmed down reports that are published/given to third parties (given to the customer’s customer) it’s extremely important that the scope of the test is detailed in that letter. The charlatan firms will be happy to omit the fact that they only tested the “about us” page while blind-folded.

So if you are reading a “letter of assessment” for something you are thinking of adopting, just look for the scope of the test, and if it sounds reasonable they potentially had a good test. If it’s missing, the test wasn’t worth the electrons in that letter.

From the other side of the fence: I've been hiring companies to do external pen tests for fifteen years now. Some of them have been giant corporations with security divisions, some of them have been just past the startup stage, and some of them are recognizable big names in the industry. I've signed one year, two year and three year contracts.

A few of them have distinguished themselves, slightly, in the first year of a multi-year contract -- and then regressed to the mean in the rest.

Quoted prices vary by a factor of 4, approximately. Work done does not. Quality of work appears to be basically independent of price.

Arrogance scales with price, though.

Even if you are a non-checklist firm, I can't justify hiring you at a higher price because I can't differentiate you from the bloviators, and basically nothing you say other than "I am ptacek" can change that.

This is a good perspective. I pushed for a higher priced pen test firm in our last engagement due their name/reputation (i was sick of nessus scan outputs) but ultimately their price was just SO MUCH cheaper than the prestigious named one. I couldn't provide a justification to leadership outside of "they have these awesome writeups in their research division". Asking to see a sample report to prospective firm turns out to be a crapshoot as well (but seems to be one of the few things to go off of). Can you provide any insight on how you evaluate?

Security unfortunately is a creative process, which is hard to get consistent. I would love to use something like statistical product control, but I have yet to see a good way to apply it (or similar techniques) without forcing pen-tests to be “follow the checklist”.

I should add that burnout is a massive problem when it comes to quality. Even if you found a good consultant (or group of), for round #2, they could have just finished testing all of your competitors products and are now worn out.

Scheduling in variety to try and prevent partial burnout is difficult if everyone in a vertical only wants to use the same person.

This is what I am working hard towards, moving to a firm like yours.

4. Most places you'll work will treat you as a cost center. Nobody wants to pay for security.

5. A lot of the "Cybersecurity" folks are closer to the sysadmin-cert-associate degree than the Engineering CS side of things.

I'll add one more, the problem with compliance jobs is that it only really becomes important (to the bigwigs) when it goes wrong.

Most of the time you are just a cost-center, a necessary nuisance. It's a lot harder to extract money / get promoted when your good work isn't immediately noticeable.

The cost center mentality is a huge part of the problem.

As a consultant I've been trying for years, with limited success, to persuade people to think in terms of business process integrity, since that more clearly ties the necessary work to the revenue output. It's an uphill slog though as prevailing culture is checklist compliance.

Doesn't help that major consulting and advising firms make more money on checklist mentalities.

4. Software development within cybersecurity. Ie. developing the tools of the trade. See [0] for an example of an opensource tool. I would say this is pretty interesting work.

[0] https://www.zaproxy.org/

It's specialized type of SW. a little more interesting than webdev, i guess, but not by much.

I echo this. I was also interested in cybersecurity through CTFs. I also do a bit of bug bounties in my spare time. But i've sort of come to the conclusion that cybersecurity as a career isn't that great in general, most roles are just administrative or compliance types from what I've seen. But then I again I have seen opportunities with sort of smaller boutique security firms that mainly focus on exploit research, but for those it feels like they're looking for more specialist knowledge.

The industry gets a bad rap, because it's full of ego maniacs, but the reality is that there are tons of interesting opportunities that fall outside conventional compliance/pen-testing roles. Research is actually a huge sector, because you can apply security research to so many emerging and existing industries. You can specialize is specific things as well. Take for example blockchain. !0 years ago there were no cryptocurrencies, today it's a multi-billion (trillion?) dollar industry. People with skills to audit smart contracts are in huge demand. What about all of the other emerging technologies? Criminals are always quick to exploit and abuse emerging tech and emerging markets, so there are always opportunities presenting themselves for technical & security minded folks. The world of cybersecurity goes beyond just pen testing or sitting in a SOC as an analyst.

Security research fascinated me after reading a number of papers on really cool exploits, but it was quickly evident it wasn't something I'd ever be cut out for. Anyways, do you typically have to be highly credentialed to do it? How many people are getting hired as security researchers without prior [academic] research?

Honestly, it's something that you can just "do". I know not everyone has much appetite for speculative, unpaid work, but that's the lowest rung of the ladder you can start. Bug bounties for example... if you go deep into enough into a single category of bug class that you can find a new way to approach it, that's demonstrated security research and can start you off on a track record that makes you employable. Not all security research is low-level stack smashing. I also think there are highly overlooked opportunities in emerging sectors. Off the top of my head, VR is taking off, but it's still so niche. It's an opportunity to go deep into something and make some discoveries that might have meaningful impact. So I guess as a long winded answer to your question, I don't think there's a typical path, because not all security research is academic by default in my opinion.

Your impressions mirror mine, and is also why I chose to stick to software engineering. You can still apply your security knowledge. OSCP doesn't apply to most of that depending on what language you use (caveat: I haven't seen the new course) - OSWE does however. Generally companies won't pay extra for your security knowledge unless they're specifically looking for it, even if you've really found and fixed things in code proactively.

I also found some interviews in a similar format as developer interviews (hands on, timed), but with salaries that make it not really worth the trouble and stress (that's what OSCP was for!).

IMO, one of the problems with things like the OSCP is that they don't really mirror how real pentesting works.

I've never heard of a pentest that's done as a 24 hour thing where someone is watching you through a webcam and you can't collaborate with people, and then the report is written up the next day. Also (last time I looked at it anyway) it quite a few topics that whilst interesting again don't mirror day-to-day pentesting, and restrictions on using things like Metasploit are just weird.

I’ve been in the industry since the 90’s. It’s been taking a pretty hard pivot into ‘devsecops’ and related merging of a variety of software engineering and development practices and traditional security domain practices.

I still love it. There’s a lot of interesting challenges and equally interesting people to work with. I do agree there is a perception problem, however, most of which is self-inflicted by a small minority of the those that work in the domain.

>2. Working in government,

obviously there are two sides to government, one side is the same as #1, checklist and say our site/solution is now secure, #2 is the side that has to do with generally attacking other governments but also handling dissident/criminal parts of their own society.

The stereotype I heard was that security is under appreciated, so you get less respect at work while simultaneously getting paid less. I don’t know first hand, is this true?

I'm convinced that if you are doing cyber sec at any large company you spend most of your time in Excel and not a shell - whack-a-mole is the best way to look at it.

The majority of work in government is on the defensive side, not the offensive side. But that would still be what you call whack-a-mole most of the time.

Yes at the end of the day your main job is it to either educate people about always the same security issues,

Fixing the same security issues,

Or singing of on procedures.

is hunting for bounties not viable?

It’s a good side hustle and a fantastic way to develop your skills, but depending on it for income is extremely stressful and leads to an array of pathological behaviors that spoil it for you and those downstream of your efforts.

As a primary source of income? Not really.

You could spend weeks looking for bugs and find nothing and not make a dime.

This is me.

There's nothing more boring than those positions. (Or to me, unethical than working for the US government)

I'd much rather create something that benefits the world.

Maybe the partial solution is bug bounties? As much as I'd hate those positions, I actively poke around for holes in websites for fun. I love when I have to do business with a small poorly run website, seems you can always find something they don't want you to have access too. Most recently, no right click to save images, but chrome developer tools got me the high res version anyway!

> (Or to me, unethical than working for the US government) I'd much rather create something that benefits the world.

I’ll be a little more nuanced. I’m very appreciate of the hard work nist does posting best practices and guidelines on privacy, zero-trust, crypto, and other things. The SP series of publications are well respected and do lift the entire industry up to some minimum standard. In some cases they actually move things forward by a lot (like the AES competition)