Hacker News Reader

Comment by dmhmr | Hacker News Reader

The past few years have made me feel sour on how many organizations run cybersecurity in general. The industry is full of individuals who do not understand the tech they are protecting, and often they barely understand the security tech they use daily. A lot of places are simply doing compliance check-marking and barely have a shred of technical aptitude. They struggle with basic fundamentals like inventory and patch management. It is an industry that is hard to stay upbeat about if you are looking at anything larger than how it benefits your personal paycheck. If you want to get insight into the reality of how the government operates, just look at GAO reports, they are alarming: https://www.gao.gov/highrisk/ensuring-cybersecurity-nation

View on HN

Add to that the general lack of education around cyber security, hardly any mainstream CS course teaches cyber security as a mandatory course. We have CS Phds engineers who are experts in their domains but struggle to understand basic security concepts. We need to educate engineers to care about security of their code and systems just like they care about performance, reliability, maintainability etc.

The problem is further exacerbated by a class of people who received their MBAs and think they know it all. Just yesterday, a lead product manager was arguing with security folks about why his service needs to be patched for log4j vuln if its not internet facing. He had trouble fathoming that even though his service is not internet facing, it processes and logs user controlled data.

Look at recent Azure vulns, I am pretty sure their internal security team knew about these and after some back and forth some exec might have signed off an exception. They would rather be shipping features than fixing the mess they created. Most infosec peeps have trouble getting teams to prioritize of security stuff and some of the blame falls of infosec teams too for making everything sounds like a end of world scenario. But did Azure lose a single customer or did the stock price go down or loss of revenue? Nope, so whats the point of investing so much in security if it truly the only harm was some loss of reputation.

Even most security execs I have had a chance to interact with dont understand security topics properly, surely they can use some jargon to throw around in all-hands meetings and such. Unless from a security background these execs often confuse security with compliance and instead of investing in defense in depth techniques they look for check-boxes against security controls.

>Add to that the general lack of education around cyber security, hardly any mainstream CS course teaches cyber security as a mandatory course

Paradoxically, when someone has a pure (or at least focused) cybersec program (a few 3-4 year programs are taught by reputable institutions near me), and a Sec+ or equivalent, all of the old guard shout about needing years of experience (decades preferably) before you should be allowed to even think about security.

It only takes a few days in r/cybersecurity or r/securitycareeradvice to see these people in action, yelling at kids coming out of a 4-year university course focused on cybersec to "put in their dues" and work a call-center/help-desk for a few years resetting people's passwords before being allowed the honor of applying to an "entry-level" security position.

If a 4 year program cannot prepare you for an entry-level position, either the program is broken or the hiring expectations are broken.

Just in this thread someone was saying they would require 10 years of system administration AND 5 years of security experience before considering to hire them. In the same amount of time you can become a doctor or lawyer, and be operating on people or have established your own law firm.

I'm tempted to rather rudely suggest that the people who managed to get a job on a helpdesk without any qualifications and then worked their way up to an "old-school" bureaucratic security manager position might feel threatened by graduates with new fangled ideas about DevSecOps.

Exactly what counts as an entry level security position? Manually analyzing alerts or something?

Really, any cybersec role but with "Jr." in front; lightened duties and lightened responsibility, under the management of someone with more experience, doing whichever duties their manager thinks they can handle.

- Compliance auditing (PCI, ISO, WebTrust, etc.).

- Software auditing.

- Delivering basic consumer-level security awareness training.

- Tier 1/2 SOC and NOC duties.

- Member of an incidence response team.

- Member of a penetration testing team.

- Policy development, deployment and management.

- Jr. Researcher for XYZ (PKI, cryptography, authentication systems, malware, etc.)

> Add to that the general lack of education around cyber security

Part of the problem are the for-profit schools and bootcamps cranking out 'cyber security' graduates. They know the least out of all the people I interview. How can you pretend to know anything about cybersecurity when you don't actual know anything about programming or networking?

The classes cover buzzwords like vishing/phishing/smishing, you run Kali Linux and 'hack' something, and then you get your certificate.

>Just yesterday, a lead product manager was arguing with security folks about why his service needs to be patched for log4j vuln if its not internet facing. He had trouble fathoming that even though his service is not internet facing, it processes and logs user controlled data.

I got a lot of good mileage out of explaining the Equifax Struts vulnerability, which allowed attackers to move freely through Equifax internally once outer security was breached because internal security controls, especially around patching, were so weak. Might be worth trying if you encounter the same situation again.

> Most infosec peeps have trouble getting teams to prioritize of security stuff and some of the blame falls of infosec teams too for making everything sounds like a end of world scenario.

So much this. I had a security review failed because an API would respond with http 422 on invalid input. When I asked why that is a security issue I got shut down with “defense in depth”. After a longer discussion the problem was that 422 was not part of “the original http spec” but rather some ldap extension.

> Add to that the general lack of education around cyber security, hardly any mainstream CS course teaches cyber security as a mandatory course. We have CS Phds engineers who are experts in their domains but struggle to understand basic security concepts. We need to educate engineers to care about security of their code and systems just like they care about performance, reliability, maintainability etc.

Here's the issue: cyber security is seen as a cost center. As long as it's viewed a cost center good CS programs won't care for it. Which means right now it's relegated to certificates and extension schools... we all know what that means.

If companies/governments start caring for cybersecurity, ie, create a prestigious and visible organization that directly reports to the White House for instance, then you'll see the good CS degrees adding more of it to their curriculum.

> The problem is further exacerbated by a class of people who received their MBAs and think they know it all. Just yesterday, a lead product manager was arguing with security folks about why his service needs to be patched for log4j vuln if its not internet facing. He had trouble fathoming that even though his service is not internet facing, it processes and logs user controlled data.

I remember being in a room like that. At one point several people were arguing and the lead engineer just tapped his brass rat on the table to get everyone's attention. I remember the PM was furious but what was he going to do? They don't sell those at the gift shop...

Truth is, PM orgs need to exist in a parallel way to engineering orgs. PMs managing engineers is a red flag, and a true tech company should ideally have engineers all the way to the CEO position. So if there's security work and engineering deems it necessary, it's done no matter what some non-technical employee thinks.

Engineers could honestly take a page from MDs here. Opinions of non-MDs are basically regarded as irrelevant...

Have you ever worked with someone in information security, only to find out they're checking off features but don't know what they're doing? Has it been scanned by this piece of software (which produces 832 fans positives) and provided a remediation plan? Has everyone taken the on-line cyber security training? Do you have a documented architecture? Are you using the approved software versions (only they didn't get the memo we've moved on from Java 1.8)?

I once had to argue back and forth with someone (circa 2008) that JavaScript did not mean "mobile code" in the sense of their checklist. I had to explain what JavaScript was, how it worked, but they were more than willing to tell me I had to remove it from the app I was working on. Which would have rendered my app and all the other apps for that client much less functional.

> JavaScript did not mean "mobile code" in the sense of their checklist

What the hell does that even mean?

What they were referring to were ActiveX controls, Java applets, dynamically downloaded JAR files like on Sun's Java browser, etc. Like rollerblading, it was the unfortunate trend of a bygone era. And yes, it was a ridiculous term. And just more evidence that the "cyber security" is just an organizational fig-leaf.

I think the general mindset is, if something can be used for exploitation, then for the sake of safety we should block it entirely.

Of course the mainstream stuffs are tolerated but anything outside of that would need a long list of approvals.

This. The only person I would trust is a person that was a sysadmin for at least 10 years and decided to specialize in security for another 5 years. So you are looking at minimum of 15 years experience to be decent. Without deep sysadmin skills - I am at a loss of what they would contribute. You are going to update our firewall without understanding what CIDR notation is? You are going to create a VPC for the dev environment not knowing what a subnet mask is? You are going to monitor security with thousands of VMs with no cloud background? Security is a specialized specialized field. Not only that you need to be a bit of a bully. You are always fighting PMs for more time to vet things and patch things - all while being a cost center.

Why do we have so many security disasters? Because those people are rare unicorns, ridiculously expensive, with no way to show added value.

If it takes 15 years to teach someone only to a decent level, the industry needs to start laying out plans for effective training and credentialing. The solution here is not higher years required, but more effective teaching. Learning on the job is practical, but not efficient, one example being you spend time dealing with org issues rather than learning something technical.

I don't agree that it takes 15 years though. I think you're setting the standards way too high for no good reason, especially for "decent".

> effective training and credentialing

By the time you're done creating the perfect "Information Security Certification" test everything will have changed. Even the most nebulous of security certifications (CISSP; which has a super generic test that doesn't cover much in the way of "practical security") still requires 5 years of experience before you can even take it.

It's just as bad as the JavaScript ecosystem. Maybe even worse, actually.

Information security is an ultra fast moving target. The only way for companies to effectively manage it is to hire people who constantly fuck around (with technology) and are always learning (the limits of) new things. It's incredibly hard to hire (and retain) people like that.

The "safest bet" for someone who really wants to be a great InfoSec professional is to get really good at Linux systems administration then start learning how to break into things. Because once you've broken into a system you need to know how to create/execute payloads. Otherwise you're going to get stuck on the first step every single time: Finding the vulnerability. You need to be able to exploit one host and then use that one to break into another system (pivot).

Learning Windows systems administration isn't as useful IMHO because there's fewer systems and they're all the same for the most part (monoculture). You can pick up everything you need to know about exploiting Windows in a short time and then exploit it limitlessly (haha) later. Whereas Linux sysadmin skills are applicable to a very wide array of systems from embedded stuff all the way to supercomputers.

Also, if you're going to get into hardware hacking or making physical devices that help you test the security of things Windows skills are basically useless. Nobody actually loads Windows 10/11 on to something like a Raspberry Pi in order to place a physical back door somewhere (or interface with SCADA systems, air conditioners, etc).

>By the time you're done creating the perfect "Information Security Certification" test everything will have changed.

>The only way for companies to effectively manage it is to hire people who constantly fuck around (with technology) and are always learning (the limits of) new things

Then make the curriculum about fucking around with technology, taught by people who fuck around with technology for a living. Then you get a nice certificate that says you fucked around with technology for a bit and showing that you're capable of fucking around with more technology.

You're right about all the previous certs, but the solution is simple: make the curriculum match how people actually learn in the industry.

>The "safest bet" for someone who really wants to be a great InfoSec professional is to get really good at Linux systems administration

Don't get really good: get pretty good then go learn programming. You talk about jumping between embedded and supercomputers later, but programming/AppSec is more important and way more useful (esp. if you don't already know how to program)

>Learning Windows systems administration isn't as useful IMHO because there's fewer systems

Oh no mate, AD is everywhere and those skills are immensely useful to a large amount of companies. Offensive Security even changed their OSCP exam to have an AD target set.

You could say the same about security folks without software engineering experience, too.

When I was working in infosec consulting, by far the best colleagues were those who had software engineering experience and could empathise with developers at the client in order to understand how systems would be built, where corners might be cut, which areas might be more ropey than others etc (and then use that understanding to help inform their thinking from an attacker's perspective).

You could tell at interview too - the folks with a Computer Science background and a side interest in security were much, much better than those who took the dedicated-cyber-security degree/masters route.

You absolutely need a real generalist for security. With that said, I don't think it's unreasonable to expect a developer to know about CIDR notation, networking and cloud systems though we're perhaps straying into more DevOps-y style roles.

It's amazing that we can train someone to be a doctor and allow them to operate on your heart, brain, etc. within less time than you'd allow someone to touch your precious environment.

This points to two issues: education needs to be addressed with more input from industry, and expectations for hiring need to be realistic. 10 years before you're able to work on something security related is not realistic, nor is it sustainable.

I would not trust a surgeon to operate on me unless they have over 65 years of experience. I want them to have operated on patients since 'Nam. You went to a state medical school? Pfft. Go kill some other patient than me.

In all seriousness, your point brings up the idea of where does the responsibility for this immensely difficult task (securing networks) fall? If we could spread out the "required" 15 years of experience into each of the developers, would that have the same effect? Building software with security baked in would reduce the need for so much work after the fact.

General security awareness training in CS programs (not the 'don't get phished' type of security awareness) would certainly go a long way, in my opinion. Security being taught as a fundamental necessity of programming would, down the road, lessen the load everywhere else.

But there is also a fundamental disconnect between what schools are teaching and what industry is hiring for. The answer right now is "Go to school for cybersec, get your certs, then work for X years as a low-level help desk agent or call-center phone jockey".

Industry needs to tell educational institutions what candidates get from being a password-resetter that isn't taught in school, and work with those institutions to get those skills into the curriculum.

I have a lot more to say on the topic of cybersecurity and hiring, but I'm getting into rant territory.

Edit to add: You mentioned 'spreading out the 15 years of required experience'. I firmly do not believe it takes anywhere near 15 years of experience to become competent at cybersec.

you are utterly missing a business dynamic here in America and elsewhere.. Companies that originate in, with strong-ties to, established finance, literally push skill down the pay stack, not up. What does that mean? If a certain engineering skill is rare, it will cost more money to pay someone, and harder to find. Therefore, commoditize and automate where you can, via cloud accounts and "best practices", outsource to another company where you can, and promote internally for ruthless cost-cutting, firing and aggressive contract manipulations. This is not extreme, this is normal and daily for decades.

The imaginary skilled professional you are describing clearly originates in the mind of an engineering worker.. a person gains skill through experience and is promoted. This is opposite of what management builds over time.. Management specifically and exactly destroys this career path because it costs them more money. As long as you can commoditize and outsource, you drive costs down, not up.

Meanwhile, it is "eternal September" in the job world, with streams of 20-somethings lining up to get into the markets. Add lower cost engineers, for example in Eastern Europe, South East Asia and South Asia. Rinse and repeat.

Thank you for this. This is the first post I've read on this thread that acknowledges the reality of what it's like for career-minded infosec folks.

I'm a 15-year infosec vet. I'm not nearly as technical as some of the HN crowd would like for infosec guys to be, in large part because high technical is not something employers generally want and are willing to pay for. If you want to maximize pay, the best path is to learn just enough to be regarded as competent, then move into management, sales, or PM work. There's barely room for the highly-skilled, highly-technical cyber guy in most large companies, let alone SMBs. Most companies chop this ideal infosec role into multiple parts too minimize cost and risk, just as you describe.

Agreed. And it's actually not an American phenomenon, but a global one (as long as it needs to be listed on US stock market). The model basically is to remove IT functionalities from branches and congregates all IT power (think DBA/DevOps/etc.) to HQ so that you only need to maintain one single big IT department. In the middle of this HQ will also try to replace custom solutions by one single solution that works for all branches/departments. The branches still need to maintain a small IT team but essentially they are just configuration pushers.

Then it outsources to Eastern Europe.

Amusingly I would never hire a sysadmin to do security. Sysadmins were taught how to administer and apply this knowledge. With security the field is constantly shifting, you don’t want someone who knows how to apply you want someone with a deep understanding of the underlying principles able to design and explain new solutions. It’s really an engineer job.

Amusingly, this is a recent development.

Once upon a time there were programmers.

Then there were systems programmers and application programmers. Systems programmers wrote operating systems and utilities for them. App programmers wrote apps. There was a lot of crossover.

Then there were operators, systems programmers and application programmers. Operator was a junior position who did physical things (mount tapes, plug in cables) and ran commands to do things on the systems. They usually moved up to being…

Systems administrators, who did some programming in service to the systems, but not too much. The more senior a sysadmin was, the more time they spent programming and the less time they spent doing physical things… unless they wanted to do that.

Sysadmins started to specialize. People who configured switches and routers and talked to telephone companies became “network engineers”. People who spent time working on firewalls and security policies and thinking about that became “security engineers”. Junior people who read scripts to end users became the helpdesk. And so forth.

Then we noticed that a bunch of people were doing things manually when they should be automated. This was especially bad in places where there were no senior sysadmins or systems programmers. But we did have the internet, and senior sysadmins got together and started writing tools to make their lives easier: infrastructure automation.

Remind me, which kind of engineer?

That's very cute but that doesn't mirror my experience with reality at all. Sysadmins are people who are hired with specific knowledge regarding maintenance and management of the infrastructure. They are expected to have operational knowledge, are not asked to design novel solutions to unforseen problems and are paid accordingly. Most sysadmins I have worked with become unpleasant when they reach the limits of their expertise not curious. Good sysadmins tend to leave the field for better paid position in engineering or get hired as SRE by large companies which is aking to moving to an engineering position because SRE work like engineer.

The idea that senior sysadmins are behind the push towards automation is amusing. The biggest shift in the field in the past two decades came from Google when they decided to solve the tension between developers and sysadmins by more or less firing their sysadmins and hiring engineers to do the job instead.

> The idea that senior sysadmins are behind the push towards automation is amusing.

Many true things are amusing, including this one. I think you are operating with a remarkably narrow and historically ill-informed definition of "sysadmin", and your prophecies are self-fulfilling. If you have a tension between developers and sysadmins, it's a cultural problem.

This has been exactly my path. Can write data security guidelines and also read PCAP files fluently. However, you will not find very many of me.

Sysadmin/ops has too many offramps that drain talent before year 10. If you can integrate software/systems well, manage projects or do advanced troubleshooting; you will likely be pulled out of ops. Conversely there are an ocean of security certifications being issued to people who have very little operational/technical experience.

Data security in practice is being reduced to a policy and procedure checklist. It is frustrating for an engineering group to receive non-specific or contradictory policy guidelines written by non-technical people, but I have yet to see that change hiring or decision making. Businesses want someone who will agree to check the box. If that someone doesn't know all the details, that makes checking the box easier.

The future of cybersecurity is not skilled coordinator/PM but instead yet another non-technical management arm handing down mandates that are blind to technical reality. There isn't another option. There aren't enough people to fulfill demand, and the compensation for cybersecurity positions are often less than a senior infrastructure role. How many sysadmins really understand networking, programming, databases, etc; While also having the people skills to not alienate both management and highly technical development and operations teams? We will never have enough people at the intersection of that many skills.

I think the problem is sysadmin is NOT something you can learn using a personal account. Same for most DevOps things too, you simply don't have 1) the $$, and 2) the many services that you can play with. You HAVE to join a large corporation to learn the real stuff.

Plus nowadays more and more companies are going on cloud, so there are fewer sysadmins jobs anyway.

I'm someone who really wants to be in some admin jobs, be it DBA or sysadmin, problem is I first joined as a business analyst, and now I'm a DWH developer/Data engineer hybrid, every step towards a admin-ish job takes way too long and difficult for me :/

Dang second thought, actually I kinda agree with you on the number of years though. Humans are strange animals that HAVE to learn from mistakes, and their OWN mistakes. Mistakes of other people rarely ring a bell loud enough.

But I do believe that for an entry level you don't need 15 years. Maybe 5 years of sysadmin or devops should be good enough.

Most jobs in "cybersecurity" are essentially just around for CYA purposes and not actually for improving security in any meaningful sense. Indeed, deployment of "security measures" for managerial CYA purposes result in things actively detrimental to security, like widely deployed, invasive snakeoil and many other things.

It's not just the jobs - it's the WHOLE fucking "software security" industry from the top down (I worked in it for 5 years - I refuse to touch it with a 20ft pole now).

The entire industry plays a game of:

- Create a checklist (or use an existing checklist - ex: FIPS)

- Check off all the boxes on the checklist (any way they can - however they can, with complete and utter disregard for the spirit of the checklist)

- Confirm with legal that checklist is complete

- Advertise that they are "secure" to customers who happen to care (not many do, honestly) and present them with the required completed checklists

- Get hacked LEFT AND RIGHT because the whole fucking game has nothing to do with security, and everything to do with liability.

- When they're hacked, whip out the checklist again and go "couldn't have been our fault! we followed the checklist."

Repeat.

----

Now - Software security is hard. Unfathomably hard to most people (as in - they literally don't understand). People STILL fail to realize that software security is not like building a bride - I see it still even here on HN, where folks spout off bullshit comparisons to things like restaurant health/safety inspections, or architectural reviews.

The difference is that the bridge is not constantly being assaulted by an intelligent, evolving, malicious, human force. The software usually is.

And the security team can't just win one battle - they have to win every battle. Whether that's old systems, or a tired employee clicking an email link.

So I think you're basically between a rock and a hard place as an honest security worker. The job is literally impossible - so the folks who make money are the ones who compromise fastest and check off the most checklists (again - spirit of the checklist be damned).

I think the ballooning insurance payments (and the obvious eventual halt to offering cybersecurity insurance) will eventually bring the whole house of cards down, but we're still a few years out from that.

Those comparisons are a pet peave of mine. What kind of bridge can you automatically tear down and rebuild from a blueprint? Or be reliably built on top of layers of technology you barely understand. That does create moral hazard. But it can also be a solution.

In light of this, how would you propose a developer or a would-be developer to really understand software security? I have the feeling that unless one is well versed in sys-prog concepts it's kinda just a checklist factory. I know I need to do A, B and C but I don't know in depth why.

Physical world analogies are appealing and easy to visualize but they work so poorly that I wish we stopped using them altogether for IT security. A bridge under attack is a bad analogy too. One cannot possibly build a bridge which is impossible to destroy having budget 100x of bridge building. But in IT at least in theory it is possible to build such system. If you make no mistakes. But the more complex system the less likely that no stupid mistakes where made.

That's because having good security doesn't make you more money.

"A penny saved is a penny earned."

It may not make more revenue but poor security certainly affects profits.

As long as poor security is cheaper than effective security, nothing changes. Equifax, Solarwinds, and Colonial Pipeline are all still in business.

Change doesn't only occur with death.

This. We really need more competitions to help.

Competition won't help - it is impossible for an outsider to accurately measure a company's security practices and pick a company based on that.

What we need is regulation regarding putting personal data at risk to provide a financial incentive for companies to take security seriously.

That is the last thing business wants. The credit card brands developed PCI to avoid regulation. But in most circumstances, there is no 800 pound gorilla to enforce security standards.

If you do an in-depth read of the PCI security standards, you’ll see that the standards are about protecting the card brands, not you.

PCI is a very bad example because when it comes to card fraud the liability is on the merchant, bank or card networks. So in that sense it's actually normal that PCI focuses on protecting card brands and not you because you are already protected by them and they're just trying to recoup the costs.

You can't compete against free.

Risk is free.

A risk-aware competitor faces a higher cost function and a market which won't support it.

What we need is regulation, and direct liability of corporations, stockholders, creditors, and executives.

The funny thing about this adage is any profitable endeavor will have higher revenue gains so as stated it fails to convince.

"A stitch in time saves nine" is probably more relevant to security.

Neither does GDPR compliance but here in EU I know some companies who're real nervous of being fined, while at the same time doing their best to comply.

Fines would therefore be the obvious solution to the lack of cybersecurity. Network breach / data leak due to not patching software x days after vuln disclosure? Here's your fine!

Unfortunately, most of the alphabet soup compliance programs have perverse incentives - they encourage ticking check-boxes, while do nothing to improve the security as such.

I believe the real problem is effective security is hard, and most merely want to pretend than actually invest in doing it.

Accountants have made loss-risk-based evaluations for a few centuries now.

Now that insurance companies are aware of all those infamous exploitation stories that make headlines...

Well, maybe more checklists and consultants.

I had our sec team try and blanket ban base64 strings on our WAF in response to log4shell. I'm talking body, url everything.

The reasoning was we probably don't use base64. I was amazed.

I love that if you brought up that you can’t tell the difference between base64 encoding in URLs and other random alphanumeric strings, your cyber team would then get even more wide-eyed about “ex filtration attacks”.

Please ask them to write a regex that filters out those naughty base64 strings! Yes, "12345678" is a valid base64 string and so is "clueless".

> A lot of places are simply doing compliance check-marking and barely have a shred of technical aptitude.

Why would they? Does capitalism incentivise "caring" on a technical and ethical level about doing the right thing, or does it incentivise spending the minimum amount of resources to be covered by insurance and not criminally liable for anything? If they did the "right thing", someone in management is wasting resources.

Of course, if your company is private and the shareholders are decent enough people to make sure the board are doing things properly, this can work. With public companies I don't see how it is remotely feasible?

We have to legislate to compel companies to do this and expand the definition of negligence, which itself is quite complex. Make the people at the very highest levels criminally liable for breaches that happen due to lax, box checking behaviour on their watch. It is the only way.

Reply on news.ycombinator.com