Hacker News Reader

I have worked in cyber for 20 years, here are some of my greatest hits:

- being able to translate vendor BS to "if you don't do $X, $Y will happen with $Z likelihood"

- getting things to play nicely with each other, specifically in a SOAR or SIEM context (integrations suck, they are buggy, have varying levels of completeness, etc)

- knowing whether a tool is deployed, working and healthy

- having so. many. agents.

My biggest recommendation for selling/pitching/positioning anything in cyber is to solve problems I already am dealing with. Do not go to a customer with a flashlight that points out new, unknown issues and requires additional resources to fix... it's fine to point out problems, so long as you have an easy button I can smash to fix it, without having to pull in someone to deal with it.

> - being able to translate vendor BS to "if you don't do $X, $Y will happen with $Z likelihood"

This kind of makes sense to me, having worked the last year a lot together with our industrial CERT teams and others. Most cyber security teams, due to lack of human resources and human time/skills/even more time, seem to have adopted this perspective to stay sane in the ocean of noise alerts.

I want to iterate on that thought a little further: Is this an effect that was caused by shitty tooling?

In an alternative product that would tell you "there's 0% risk of affecting the running apache2 service with these package upgrades which would fix 1234 different issues altogether, so we automatically updated it for you" ... wouldn't that fix the risk assessment management problems, too?

With the peer-to-peer approach of our Agent, we can mitigate most of the critical issues automatically, and the deployment and rollout/update strategies are also measured in whether or not they affected the health status if a running web service (e.g. did the web traffic drop? were there a lot of segfaults after it? etc)... so this also leads to a better risk assessment because the data of whether it was successful as a mitigation strategy is shared among the peers in the same network(s).

> I want to iterate on that thought a little further: Is this an effect that was caused by shitty tooling?

Not shitty tooling per se, but ineffective communication, which can limit the actionability or requires context switches on the operators side. MITREs ATT&CK framework is helping solve some of these symptoms by tying things back to tactics/techniques and a common taxonomy, but there is still a long way to go.

> With the peer-to-peer approach of our Agent, we can mitigate most of the critical issues automatically, and the deployment and rollout/update strategies are also measured in whether or not they affected the health status

Most companies are trying to have less agents on their boxes, not add more. Tools that are agentless tend to have much greater chances of success, even if that means having a more complex architecture.

eBPF + SSH configuration of for example a local log system or even pushing a collector/broker can be your best bet.

You want to avoid an agent when selling to security people for 2 big reasons:

1) most of the time, security teams will not own the systems they want to put agents on, they have to get permission from the app/service owners and many times there is significant pushback. Many, if not most orgs have been burned by "just deploy this agent, there isn't a performance impact".

2) when you don't own the systems, you can increase the sales cycle and complexity of the sale, since you now are effectively requiring multiple groups to agree on something and agree that this is a priority. If there is disagreement, you can get vetoed by a group that wouldn't even be responsible for it.

I am 100% not telling you that agents are impossible, but in most enterprises, it will be more difficult than you think to get it deployed. It might take what is normally a 3 month proof of concept to something that is 6 or 12 months, or could kill it before even starting.

In regards to the suggested collector approach:

Currently, the peer nodes can deploy themselves to other surrounding nodes, and there's also a "discovery" strategy that tries to find out as much as possible about surrounding systems in the network. This usually leads to running the binary on the other system, collecting the information, and removing the binary and the report from /tmp afterwards.

Initially I wanted to push this as a pentesting approach, and simulating actual exploits inside the network to see "how far" someone could go, where the critical hops are etc.

Maybe I am seeing this a little wrong, but would such an approach work for organizations where multiple teams share an ownership of a system? Maybe on a scheduled basis so it doesn't conflict with high network load times?

This way there would be basically a collector that re-routes the S/N/SaaSBOM reports back to the dashboard where the cyber security team would operate in.

Inside the program, it's already implemented an basically just a different action that's running. In that case it would be the "tholian-guard <collect>" action vs the EDR-mode "tholian-guard protect" action.

Thanks much, this is very valueable feedback I need to think more about, and also the way I'm communicating/designing the dashboard workflows for this scenario.

> My biggest recommendation for selling/pitching/positioning anything in cyber is to solve problems I already am dealing with. Do not go to a customer with a flashlight that points out new, unknown issues and requires additional resources to fix... it's fine to point out problems, so long as you have an easy button I can smash to fix it, without having to pull in someone to deal with it.

I hear this one. I can't count the amount of times some salesperson has managed to get the ear of a decision-maker and try to insist that we need to solve some new urgent problem category that the vendor's sales department has invented. The solution always promises to solve all our "newly discovered" problems but the solution requires us to dedicate someone 20-40 hours a week to just working their platform. Another fun one is they advertise that it's "automated" but the setup is so long and arduous with required organizational process changes that break workflows that it sits there doing 5% of the advertised capabilities like a gym membership you don't use often, but feel like you've admitted defeat if you cancel it.

> while leveraging a mixture of golang for userspace and eBPF for kernelspace

This severely limits your market share. Most on-Prem customers are still using pre 5.0 Linux kernels, and most cloud first customers are already using Lacework, Aqua, Orca, or Wiz for cloud based asset inventory, monitoring, and even enforcement. Plus SentinelOne and other existing security companies are entering the market (not necessarily a bad thing, but would be difficult to compete without a compelling vision and GTM).

In my experience, the asset inventory problem is mostly seen in large Hybrid environments where asset ownership is split across multiple teams (a CMDB team, cloud platform team, application team, security team, networking team). This along with an SBOM story is very hot right now, but you will need to find a way to differentiate yourself from the list of competitors above, along with the hesitancy to manage yet another agent.

The bigger issue is that while this does sound like a cool feature, it would be a massive lift to make it defensible against companies with an existing Rolodex and also to manage the sales motion.

Also, customers hate installing and managing agents.

Currently we have a map of discovered nodes inside the network, and an auto-install and auto-penetrate feature, which is meant for batch installing a lot of nodes with the agent.

The agent itself is also able to spread itself to different architectures and operating systems once a single system is installed, depending on the selected aggressiveness after the discovery phase for the surrounding network.

There's also a prototype that's trying to use a bunch of RCEs to "conquer back" lost nodes, and to reinstall the tholian guard on those systems even when the hacker got (temporary) administrative rights and locked the "old agent" out or managed to delete it.

From a legal standpoint this has to be separate product, where companies and their assets/systems/networks have to be manually vetted before they're able to use that feature, but I think you know where I'm getting at with the idea.

Don't get me wrong. Your architecture is solid, but managing agents becomes a very political issue internally due to conflicting responsibilities between teams

As a internal security person (now a consultant for third parties): your approach is interesting, but still fails where all security tools do, who is going to install the agent on every box. The reason why every asset tracking solution is incomplete is because they are trying to correlate agent data and platform data. This is exacerbated by cloud computing since resources are much more transient and new servers lack gating by a governance org.

Complex AV tools are cool but they are so far down the chain of actually exploited vulns, they are not super useful most of the time. Usually the vulns used are old and just unpatched (check the latest DBIR data to see average age of exploited vuln). So a lot of time and effort goes into cajoling teams to update packages and having them say "we don't even call the vulnerable function."

My biggest issues: -Asset inventory (SANS Top 20 #1)

-Software inventory (SANS Top 20 #2)

-Tagging of ownership (what does this box do and who do I call if it goes bump in the night)

To answer your other questions: -Yes, but that is because we created it

-Yes, but only because we have tooling to do it across cloud envs

-No, we only look at deployed bins

-Signature based

-Getting useful code deployed to a box is hard enough, unless there a RCE this is so far down the list of threats on our threat model

-No idea, I assume pretty good since we use MITRE references, but making sure those are accurate to what we find is tough

-Yes, all of them (Fedora, Ubuntu, Arch, Alpine)

> the MVP is trying to start with a better inventory of everything

That's a logical place to start and it definitely seems like a common weak point. It's also hard to justify spending in this area since it's not emphasized in compliance and it's hard to explain why it's important in a blurb.

> Do you have a reliable ... inventory

As you allude to, it seems pretty common to have multiple existing inventory sources and no way to integrate them. To me, the most intractable part of the problem was, ironically, workstations and servers. They are managed by different people using different tools and neither of these groups want additional security agents on their systems. Also, these types of agents tend to be prohibitively expensive when you have thousands of endpoints.

> systems themselves decide to mitigate issues, propagate patches

This seems problematic. I really don't want the security system breaking things without staff intervention. Security already gets blamed for any unexplained problems that desktop, servers, networking, and development have.

Most security tools are noise generators. Figure out how to separate the alerts that need attention from the rest.

This is such an undervalued comment.

Completely reflects my experience. Most dashboards have 99%+ noise that is useless from an operational perspective, and the inefficiency of Blueteam vs Redteam usually represents "how long until I can find the actual data points that matter" in the dashboards.

(That was the reason I went with the peer-to-peer and SBOM idea, trying to reduce as much noise as possible)

This seems incentivized by marketing, as every security demo I've ever had has included something like "See? Our tool found 46,291 possibly critical issues in your environment" and then you drill into them later and they're relatively insignificant issues you'd fix in a perfect world with unlimited labor but drown out the actual things you actually want to know about right now. I think there's also incentive to upgrade the threat urgency on alerts to look like they're capturing more urgent threats.

Been in the space about a decade. Hope this helps.

1. Pain points: most corps already have tons of agents running everywhere, usually with one or more inventory management systems already. You would have to add a ton of new value to warrant another agent install. More agents = more ways in, which can weaken the system’s security. Discovering “unknown” network assets is also a constant struggle (qualys, rumble). Plenty of vendor apps and quite a few elastic search dashboards. Agent-based asset management is a very crowded space already.

- SBOMs: rarely see it, never seen it in third parties

- antivirus: rarely see it on linux servers outside of perhaps clamav scanning user uploads. You call monitoring systems for servers “IDS” intrusion detection systems. These are different from virus scanning and usually allow for runtime detection and alerting of anomalous behaviors.

- golang malware: all malware (shellcode included) must make syscalls to use system resources from userland. Your IDS should monitors syscalls. There’s no issue here.

- every medium to huge company I’ve been at has run Linux servers for everything, with AD boxes being the rare exception when the company uses windows clients.

Dealing with people running automated scanners that do nothing but spam you about nonsense.

The spam is bad, but often I don't think they know what the results mean.

I love the spreadsheet I get every month. Just a list of vulnerabilities instead of anything curated. We noticed 5 vulns in an installed version of Firefox. Better list it 5 times just in case.

A few times now there were connections from some malicious IPs on our mail server on the mail port, and I was asked if we could block inbound connections. Open ports are a vulnerability, you know?

Never mind that the firewall already blocked that connection in particular.

Cybersecurity seems to be rife with people who did some sort of bootcamp or cert or associate degree and who don't really seem to understand what's going on. It seems to attract people who consider writing code too complicated for some reasons. So you'll get all these "best practices" and "vulnerability analysis" but no understanding beyond "I ran this tool and here are the vulnerabilities it found".

From my experience, hiring for these roles is like trying to find a needle in a haystack.

Background: 25 years of information security in very big text companies, from IT to CISO.

People deploy whatever they think is useful, do not maintain it and then a vulnerability arises. Nobody knows where the implementations are.

Business pressures for availability, but did not want to sign off that they accept their systems to be insecure. You have a weak CISO and disaster strikes.

Reluctance to use cloud based systems because of a weird belief that local teams will do better.

Politics and who-has-the-biggest-one contests.

Lack of knowledge about what data actually matters in the company, and once it is known lack of will to invest in protecting it.

When crisis strikes legal is everywhere and teams waste time with useless feedback.

ISO certified companies where it is dangerous to sneeze because the whole system will fall apart. The whole ISO obsession is maddening. I have never seen a hacker hacking the certification to get in.

Generally speaking, a burnout because of the blocks cybersecurity gets, until there is a hack and then it is all their fault.

Finally, a small hack here and there helps to raise awareness.

Ah, and dreadful asset management.

Background: 25 years of information security in very big text companies, from IT to CISO.

People deploy whatever they think is useful, do not maintain it and then a vulnerability arises. Nobody knows where the implementations are.

Business pressures for availability, but did not want to sign off that they accept their systems to be insecure. You have a weak CISO and disaster strikes.

Reluctance to use cloud based systems because of a weird belief that local teams will do better.

Politics and who-has-the-biggest-one contests.

Lack of knowledge about what data actually matters in the company, and once it is known lack of will to invest in protecting it.

When crisis strikes legal is everywhere and teams waste time with useless feedback.

ISO certified companies where it is dangerous to sneeze because the whole system will fall apart. The whole ISO obsession is maddening. I have never seen a hacker hacking the certification to get in.

Generally speaking, a burnout because of the blocks cybersecurity gets, until there is a hack and then it is all their fault.

Finally, a small hack here and there helps to raise awareness.

Not sure if biggest pain point, but SBOM tools seem to be a wild mess. A tool that can generate a reliably complete SBOM in a heterogeneous landscape, considering the source as well as the end products (like containers) would be helpful.

The only tool that seems to get this even partly right is syft but it is still far from being an acceptable solution.

We use BlackDuck for SBOM at my workplace and let me tell you, the false positives are driving me up a wall. Also, the binary tool seems very primitve (basically just takes a SHA1 hash of all your files and compares them to a database).

This is the problem my company solves. I won’t plug it here, but my email is in my HN profile if you want to reach out

As a fan of Syft, which parts are roughest for you? Agree that there is work to be done.

I think that real potential of SBOMs can only be realized if they are reasonably complete. I think the only way to achieve that is to scan the source code as well as the container. The source code lacks information about things that get added later but the container lacks information about the components that comprise natively compiled binaries. Also package dependency info is mostly lost in the container but would be valuable to have in the SBOM. So both scans are necessary.

syft does get this right because it can scan both but it falls short in combining the results. I think one product should have one single SBOM, with all components and without duplicates.

Even if syft can scan the source it does not seem to write the package dependencies into the SBOM.

Also licenses are missing. I'm not sure it that falls in the scope of syft but I wish it would and it would also detect and report licenses.

It's nice that syft supports many output formats, because conversion was so far a big disappointment, regardless of the tool, including syft.

Lastly I find the command line interface is a mess. The useful but undocumented and deprecated power-user command doesn't seem to have a successor and the other commands seem inconsequential and buggy, for example

    syft [command] --help

always prints the same generic help for all commands.

I think syft is promising but still a far way from being a good solution.

> Also package dependency info is mostly lost in the container but would be valuable to have in the SBOM.

Agreed on connecting code->container and I'd go one step further with code->container->running server. I've helped folks work around the dependency metadata issue by adding the package-lock.json into the container for the explicit purpose of having syft pick it up. Not ideal but it works.

> I think one product should have one single SBOM, with all components and without duplicates.

Yes, and it's trickier than it sounds. My current line of thinking is to model components of an app (frontend, backend, queue worker, etc) as a stream of SBOMs per commit and use label queries to aggregate the SBOMs together dynamically as they change.

My biggest pain point is translating the internal compliance/security requirements into technical measures and monitoring them in real time. Not only infrastructure but also application security. In addition templating solutions and make them easily integratable.

Are you hinting in the "what packages do I update to mitigate CVE #123?" direction?

Or do you hint to something like "How can I mitigate this SQL injection issue in the codebase?"

The former is what my system already does automatically, the latter is something I don't think is solvable in a failsafe manner because it would need a very good Web Application Firewall approach that also knows correlations about the running processes, their opened files and memory dumps (which probably are too performance-cost intensive to be of use in production systems).

A common trend with vendors is they can provide you all the data in the world that you want, but few condense it into actions to be done. The better ones do and even they only do surface level actions like one action for each detected issue. A smart person worth their salary takes in the data and turns it into a plan, these 500 issues get fixed with this action. If you want to lead in the automation state, do that. Go beyond simple detections and turn it into actions. Use data to infer states about systems you haven't scanned. Feed that into a tracking tool to showcase progress to make your customers look like they are moving goalposts.

The biggest pain points I have cannot be solved by Software. They can only be solved by a culture change, which I'm convinced is never going to happen. I want decision makers in my company to listen to me when I tell them to implement something _now_ be it update windows at once, restart the server, get rid of windows XP, patch your vulnerable dependency now, implement this security control. Rarely is any of this sane advice followed because it is considered to be disruptive, expensive or blocks the delivery of "real features" (meaning security is not a feature).

Hard to fix

25 years in IT, 8 of those in security, mostly as a Red Teamer and Red Team manager, mostly internal to very large orgs.

It's identifying owners, both for infrastructure and apps. When you're rampaging through a network and you find the Very Scary Thing that should be handled immediately and all you have is the VIP address of the load balancer in front of it, and the configuration manager has a system owner for the load balancer who left the company two years ago, things start to feel a little dicey.

How about MDM that doesn't slow every employee's laptop to a crawl? That's the real white whale of cyber sec.

It's not normally MDM that slows down the laptops, its usually the security agents/proxies that are installed via MDM that slow down the computer. Antimalware agents for macs in particular tend to be very problematic with Jamf Protect being one of the few exceptions. And if you're using an enterprise proxy like Zscaler, you might as well switch to thin clients instead.

Navy Research had funding for adaptive networks like you’re describing a couple years back. Should probably dig up the RFPs and see how that’s going

I don't work on security, but I have worked with security.

The [lack of] operational maturity of security software and skills required to administrate it, particularly at scale, and particularly administration of the data stores are almost certainly one of security's larger pain points.

Security often keeps itself separate-ish from production itself, so "who administrates the data stores for security systems, the data store expert overloaded with dealing with making the website function, or the security expert who knows a lot about security and not very much about data stores or operations?"

Another major pain point is controlling the signal to noise ratio and therefore determining the risk to toil (sisyphusian manual labor) ratio.

> the systems themselves decide to mitigate issues, propagate patches (or even vaccines for zero-days) and also share incidents that look suspicious, so the surrounding nodes can start to quarantine themselves off

Are we talking production machines or employee machines? If you are referring to the former, I will yell at you. I have gotten people fired for doing that type of thing cowboy style. It's one thing to flag a machine as requiring administration by the people who understand it, it's another to administrate someone else's machines, particularly when they are on call for it.

What does mitigate, patch, or propagate mean in terms of actual example of administrative action? For that matter what does peer to peer mean?

> Anyways, my solution currently is a peer-to-peer approach where the systems themselves decide to mitigate issues, propagate patches (or even vaccines for zero-days) and also share incidents that look suspicious, so the surrounding nodes can start to quarantine themselves off, for example, when something really bad happened.

How do you intend to run analytics on a single node? A modern EDR needs a lot of processing power to analyze logs. You are going to have to push down potentially sensitive analytics to endpoints, and have the potentially compromised endpoint decide what to share. How is this scenario accounted for?

> How do you intend to run analytics on a single node? A modern EDR needs a lot of processing power to analyze logs

eBPF solves the performance problem, as it's blazingly fast and can run as an IDS/packet filter _before_ the data packets even reach the Kernelspace. A lot of drivers have support for it already, and some Smart NICS in data centers allow even offloading it to the NIC directly.

Most of the analytics and incident generators that are already integrated are in the milliseconds-range, as this approach does not need to push Petabytes of data to a centralized ELK cluster. The noise is already filtered out based on the processes that are running, and a combination of detection mechanisms ranging from execve kernelprobes and XDP programs filtering the network traffic up to analyzing what's going on in /proc/ and /sys/.

> You are going to have to push down potentially sensitive analytics to endpoints, and have the potentially compromised endpoint decide what to share. How is this scenario accounted for?

This scenario is actually accounted for already. Agents only share the processed strategies (incidents/mitigations/aggrevations) with each other, having the idea that these represent the state of "what type of malicious behaviour" and "what type of target/source" was communicated.

This can be something as simple as the www-data user suddenly executing "whoami" up to a malicious recorded network payload that resulted in a zeroday and changed the library.so files on the filesystem or led to e.g. an "nc" backdoor. The beauty of golang and eBPF is that they play very well together in this regard, and don't sacrifice performance like e.g. python or other scripting languages would.

Is market research allowed on HN

As a community HN is still a fantastic group of passionate early-adopters who care about their work, so yes - I believe this is exactly the best place to ask questions like this.

I hope it is. I'm planning to create a lot of open source projects from it.

Maybe @dang can make an exception?

While not titled as a cyber-security professional, I am a System Admin/Engineer who has worn that hat a few times when there was no one else available.

In my experience, it seems like vendor disclosures, and measures that might impact correcting vulnerable components are insufficient, and slow, if they happen at all.

The larger brands are getting pretty good about disclosing in a timely manner but you always have cost cutting happening unbenowst to you organizationally where inventory as you said isn't accounted for, and then those devices may or may not have disclosed vulnerabilities or had a fix within a reasonable time after public disclosure. Not vetted hardware but consumer, or semi-professonal brands that are much smaller.

I've run into areas where mitigations also were extremely costly because some simple circumstance was overlooked.

As an example, almost 8 years ago I was brought in to help a client who had suffered a ransomware attack where their previous IT staff were non-responsive and they were desperate.

They had Mcafee's DLP deployed organization wide, and the local DLP server that generated the unlock codes was among several other assets that had been encrypted but not documented or inventoried (so that didn't have a backup). The software product true to their word prevented even authorized system administrators from being able to make any mitigative changes without that unlock code which was not possible, their support for resolution was ineffective and useless. Even in safe mode, everything was locked down, and after exhausting all other options even accessing the system physically from a Linux live disk failed, the HDD would hang after trying to access certain parts of the disk (at an extremely low level). We were never able to figure out exactly what caused the hangs, but it happened regularly for long-running processes and we verified SMART passed. We had a guess that they were doing something funky with NTFS where separate files under a certain size get conglomerated into the same sector block which wasn't supported by Linux at the time, but we were never able to confirm definitively.

What could have been a few hour turnaround ended up turning into a week of downtime (where everyone was shut down, 10 days). Fortunately it was during the holidays which was their slowest time when they normally shut down for a week but it could have been much worse.

After exhausting options, we ended up having migrate data, wipe and reimage with almost a completely new infrastructure. We corrected a number of additional issues (misconfigurations) we found during the process but there was a good portion of time where we were just spinning our wheels with the vendor who was useless.

We went to contingency and stood up new infrastructure after we hit a certain threshold on man hours trying to work with them without any progress.

> In my experience, it seems like vendor disclosures, and measures that might impact correcting vulnerable components are insufficient, and slow, if they happen at all.

SAP has a CVE dispute rate of exactly 100%. Just as a fun fact. It gets worse when you look deeper into hardware vendors, where they mark things as fixed even though they aren't. And Debian as a feature frozen distribution has sooo many tags that are similar to "code diverged too much from upstream", yet they mark the CVEs as fixed; even though they are still affected and the old code from 6 years ago on exploitdb still works.

That is what led me to building a vulnerability scraper that scrapes and correlates all linux security trackers, and rates them with a confidence value (with Debian obviously having the lowest) in order to be able to discover those issues that have been correctly tagged, deployed and fixed in other distros (e.g. Arch Linux).

> They had Mcafee's DLP deployed organization wide(...)

Holy shit, that's like every sysadmin's nightmare. I feel for you, man. With a pentester's voice I always like to say that the biggest castle walls are useless when you have an ADFS server running inside it. A lot of DMZ approaches are useless because they underestimate the attack surface of their core IAM infrastructure.

> that's like every sysadmin's nightmare.

Not worst case scenario, but it definitely was up there.

I'll always remember it, both because nothing you normally had access for would work because no access, and my contingencies really paid off.

I had just come off a project fully automating configuration and deployment for our more common infrastructure components; auditing and meeting various security baselines.

Its nice seeing hard work you had just finished and tested in action working better than you thought.

Most of our time spent was ensuring all relevant data was migrated from the sprawl of endpoints to a central fileshare with integrity intact with no malware.

Entrypoint turned out to be one of the employees using a dropbox file-share. The previous admin had been directed to allow it by a person in upper management on a one-time basis for WFH. Their home PC had been infected and spread laterally. It was a brief blank stare/mental facepalm moment finding that out. Ultimately everything was back up and running, We made recommendations; and their IT wrapped everything else up.

Ask HN: Cyber Security folks – what are your biggest pain points?