Hacker News Reader

Comment by korethr | Hacker News Reader

I wonder if they have any sales to large enterprises or similar institutions.

In my experience, the larger organizations will have a "security" questionnaire required of their vendors, and the person administering it is a droid, incapable of evaluating whether the questions, originally written in the mid-00s and only updated for buzzword compliance since, are applicable to modern security practice today, or to the particular product/service/vendor in question. And no firewalls or routers would be massive, disqualifying red flags on such a questionnaire.

Never mind that a KISS setup tends to bring security because of its minimized attack surface. In the minds that write and administer those questionnaires, security only comes from sufficient amounts of the right kinds of complexity.

I'm sure it can be done. IIRC, Cloudflare doesn't use any firewalls, and they do some big business. It just isn't easy to get past the droids programmed to ensure that all pegs shall be properly square, IME.

View on HN

"I wonder if they have any sales to large enterprises or similar institutions."

Yes, certainly.

We frequently fill out very detailed checklists and questionnaires related to our quality policy, standards, internal policies, etc.

We're also very honest about how we approach these issues:

https://www.rsync.net/resources/regulatory/pci.html

... and they generally appreciate the honesty.

It’s hilarious that the first "vulnerability" in the example report[0] linked in this page is basically "SSH is accessible". Well… Duh !

[0] https://www.rsync.net/resources/regulatory/PCI_usw-s005_repo...

EDIT: It’s marked as "PASS" though, so it’s all fine, just funny.

I once had a someone report responding to ping as a vulnerability. For the public facing firewall.

We sent them back a link of prominent servers that respond to ping.

Including the web server of the expensive agency that had produced the report. And whose web server had an expired SSL certificate.

Talking about PCI compliance I always remember this: https://serverfault.com/questions/293217/our-security-audito...

Oh wow this is next level though.

Well, PCI compliance is different from regular server administration (a lot of it being smoke and mirrors, yes).

I do not believe ICMP (ping) is an automatic-fail condition for PCI (at least for certain SAQ levels that I'm familiar with) - however they do show up as warnings, particularly if you can get a timestamp response (to be used in timing-based attacks).

PCI prefers systems that handle CHD be "invisible" to the outside world, in an attempt to hide the systems an attacker might take interest in. Not always feasible (eCommerce, for example), but you gotta jump through the PCI hoops if you don't want to be stuck holding the bag if there's some breach.

PCI compliance is to reduce the chances of legal liability. Better security is sometimes a side-effect of that compliance.

To reduce the liability of the credit card company maybe, by making the process so complex and onerous that it is virtually impossible to complete a survey without some error or omission that would almost assuredly be used as a reason to invalidate any liability for the credit card company should some bad event take place. Source: have had to complete PCI surveys from multiple vendors.

I used to get so tired of having to write up explanations of why my FreeBSD server couldn't possibly have failed a security check for a Linux vulnerability, or that the web server they were complaining about didn't actually exist, or a million other retarded false positives every quarter. Thank goodness I don't deal with PCI any longer.

For the last security report I had to deal with for a client, the main vulnerabilities were reported against a Google site that was merely linked to from the clients site. Not PCI compliance, so more flexibility in dealing with their incompetence, thankfully.

They reported a number of purported (non-existing) "vulnerabilities" against said Google site that included that it stopped responding to their probing soon after they started hammering it with sketchy requests... They did, to be fair, point out that this could be a defence mechanism, but dinged it for preventing them from checking for other vulnerabilities.

At least I didn't have to explain why that one was nonsense - it was rather obvious to my client that the agency they'd hired were being idiots. It's not like it was difficult to see either - the domain name of the site they'd hit had "google" in it.

Sounds like a scan mis-configuration on your client's part. All PCI vuln scanners I've used require you to specify IP addresses and Domain Names you want scanned, and do not follow on-page external links.

Not the client. Third party agency hired to assess the security, and who clearly did not apply any critical thinking before sending it off. And as I pointed out, not PCI.

This was not for PCI compliance.

And the system by definition could not be invisible - the ip in question was in DNS and was what you'd connect to the web servers on.

tenable keeps telling me that something "interfered" with its scans for PCI compliance - isn't the point of a firewall IPS system to do that? but i don't even have it on for the IP in question, so I basically have to just edit my network range to remove it. it really is bullshit.

My parent org is starting to take their vuln scan results and report them to c levels.

When they told me I informed them I stopped using their vulnerability scanner years ago because they would not allow me to chnage anything in it, including exclusions to icmp time stamps or other vulns Ive mitigated while proper fixes were in the works.

So I rolled my own and use that to audit my systems. They don’t care because “policy”. My c levels will just ask and then promptly disregard all future reports, adding to the noise

I did a job where I was given access to a server in the form of a set of credentials for an HPE iLO, which was accessible over the Internet. From there, we could use the remote console to logon as root.

HPE iLO doesn't support MFA or any form of public key authentication, and its security history is much worse than SSH. It requires several ports open and the old version they had required Java plugins on desktops and all sorts of nonsense. Using it outside of emergency repairs is a terrible experience due to console refresh lag and the fact you can't copy + paste.

The reason I had to do this insecure and annoying process is that a PCI assessor had told them it would be a hard fail to have port 22 open on the Internet, but this would apparently be fine.

I don't know... I mean, maybe the security posture is to annoy the hackers into giving up?

("This thing requires a Java applet and is slow as hell. Screw it, let's just pwn the bank across the street")

I'll call it Security by Inconvenience.

That reminds me of a post[0] on alt.sysadmin.recovery.

The hackers were annoyed by the compromised machine so they installed security updates and did other system administration tasks.

[0] https://groups.google.com/g/alt.sysadmin.recovery/c/ITd7OlMr...

Amusing but today there’s two kinds of hackers: people who manually run a campaign like in your story and the endless hordes of bots that automatically exploit systems to turn them into botnet slaves or cryptolocker hostages.

You can’t inconvenience a bot.

I vaguely recall some kind of malware that upon infecting a system scanned the system for other malware and removed/disabled it. The motives were far from pure, obviously. (Although there also was a case, I think, of a piece of malware specifically created to ensure "infected" system had up to date AV software and were up to date update-wise. We sure live in strange times.)

That's why your security solution should include a mix of every known technology, hackers need to know everything from COBOL to rust

Such strategies are remarkably effective, and maybe arguably describes all security in a nutshell.

Every time I notice an obscure feature in a Google product or service and go "hm, I wonder if that could be exploited", I then always go "...meh, it'll take too long and require too much concentration to figure it out."

No, not at scale. You may be discouraged, but there's someone who will have lots of fun breaking that specific feature.

See for example @jonasLyk who spent the last half a year (?) trying to abuse almost only the alternative streams and junction folders in windows.

Wow, this guy is mad

https://twitter.com/jonaslyk

I can't help but picture him as a sysadmin walking away from a bunch of servers that are mysteriously 40% faster than ever before, but then he gets stopped at the door of the datacenter by some unimpressed looking lawyers who glare at him until he puts everything back

Thanks for the reference, and fair point, yeah that's not how it works at scale.

The new Ilo has a "HTML5" and Java console. But 3 wrong passwords and your blocked for 10 minutes (by default).

PCI QSAs are notorious for being complete jackasses. You have to be very careful about vetting them. That isn’t the dumbest thing I’ve heard like that!

Too annoying. I would’ve shut off SSH for the “assessment” then moved it to a different port after.

SSH doesn't have to be on port 22. you can leave an empty honeypot on port 22 and run SSH on whichever port you like

It irks me a lot that GCP always put "SSH port opens" in Security Command Center as HIGH Vuln. while I'm running private subnets, FFS.

I was just lamenting the state of these forms today. I feel like many of the questions are variations on the theme of “what antivirus software do you use to protect your basement mainframe from your corporate network traffic?” “N/A” isn’t a long enough answer, so it feels like you need to keep explaining “the Internet”, LLC of a centralized corporate network, SaaS, etc.

Is there a modern, no-nonsense guide to filling these out honestly without telling the person doing the checkbox checking that their form is dumb?

I realize there’s a lot of rent seeking and money to be made by consultants in this space - I’m looking for the GitHub published guide or wiki to help smaller no-nonsense shops navigate the phrasing and map these vendor security questionnaires to “modern” technology.

This is going to look like I'm a plant with the timing of my comment and this "Launch HN" post that just made it to the front page, but with that out of the way: https://news.ycombinator.com/item?id=26513040

Looks like Stacksi is aiming to fix this pain point.

> Our platform only answers on port 22 with OpenSSH.

I do security and I title this "Most secured platform in the world."

Oh yeah? Well I run one where no ports are open. In fact, I haven't even connected it to the network.

Wasn't that the joke about how the original windows NT server got it's C2/ITSEC rating?

I haven't heard that joke!

I'll raise you-- I have one that I keep powered off...

I can top that: I don't have one.

That's nothing. I don't have thousands.

What are you two even talking about?

The three golden rules of computer security: do not own a computer, do not power it on, and do not use it.

https://en.wikipedia.org/wiki/Robert_Morris_(cryptographer)

A compo what?!

Been a happy customer for a while now, really love what you’re doing. I wish more companies were as direct and competent.

This is extremely honest and transparent. In addition to being good marketing, it probably attracts customers who won’t make BS support requests.

FYI, your "pricing" link at the top of that pci.html page 404's. The pricing link works from other pages however.

I see that that has now been fixed - thanks for pointing it out.

If we’re discussing website bugs, the “SSAE16 / SAS70” page[0] mentions 2015 as “upcoming”

[0]: https://www.rsync.net/resources/regulatory/sas70.html

The “burger menu” does not work on iPhone. Works from the front page.

Man, everything about your service is simple and direct! Amazing.

Isn't the nice thing about having an access fortress, that you can monitor the access more simply? Or is it just as simple for you to monitor access to all the identically configured machines? I suppose it might be.

I believe the usual name is a bastion host, not an access fortress.

Anything is possible if you are willing to get a little dirty and negotiate with other humans.

We initially had some troubles navigating these waters in the financial sector, but once we were able to convince 1 big customer to try our system on a trial basis, everyone else started to play along really nicely. No one wants to be the first one to try a new thing and get burned by it.

In 2021, you can sometimes leverage things like technological FOMO to make a business owner believe that they are going to lose out on future business value relative to competition, who you might frame as be willing to take on a bigger technological risk. And indeed, smaller clients in our industry are willing to overlook certain audit points (at least temporarily) in order to compete with bigger players.

Some might not like it, but being able to engage in the sales process and bend some rules occasionally is absolutely required to play in the big leagues. Once you are in, it's a lot easier to move around. No one has a perfect solution and everyone knows it. It's just a matter of who is the better sales person at a certain point.

I used to (late 2000s) work for a tiny, tiny company that was courting a customer in the mobile banking space. They wanted us to tick boxes. So we bought a box (some sort of Fortinet) that said it was a firewall and IDS. Plugged it in, used it as our new router. "Cost of doing business."

Could we have argued with them during the sales process? Only if we wanted to lose the sale. The Fortinet was cheap compared to the value of the contract.

Cost of doing business, or ... introducing new Fortinet vulnerabilities into your infrastructure?

I know you mentioned 2000s, but it's funny that these contractually obligated boxes might introduce more worry: https://www.bleepingcomputer.com/news/security/fortinet-fixe...

I guess you can still tick all the compliance checkboxes if you install the contractually obligated Fortinet device outside your DMZ and internally treat it as no more trusted that the rest of the internet...

Exactly. My first thought was: did they install Solarwinds as well?

Which is exactly what Kozubik was talking about!

lol. anti-viruses are the virus. the ultimate virus. don't execute any binary and you'll be fine.

Iptables didn't count?

And you can update it at its own rhythm, potentially different from your upgrade path. And you can make them tls-end for you. Your customer might even have 3000 of those and already know how to keep them happy running. Not so bad.

> And you can make them tls-end for you.

Nothing says end-to-end security like terminating TLS at a network choke point so intruders can easily snoop all traffic.

Case and point "SSL added and removed here! :)"

https://blog.encrypt.me/2013/11/05/ssl-added-and-removed-her...

FYI the common idiom is "case in point," rather than "case and point."

The phrase represents this idea: I have a case to make (or an argument), and there is a single element that is conclusive enough to make the whole case in and of itself.

You might say, "I can address this entire case in a single point." This shortens to "case in point." The implication is that the single point you make is enough of an argument to prove your whole case.

What is the threat model there? What if the system can't be upgraded for reasons? What if your service/gateway is just behind the 'network choke' (who said you had to have only one?). Are you paying to upgrade everyone and their perfectly working mainframes or java 8 apps to TLS 1.3? How do your intruders come in? They have to break the appliance? How's the chance you have better tuned/setup your TLS terminator or FW than network security 'experts'?

The threat model is that any foothold an attacker gets behind you TLS terminator potentially allows then to snoop plaintext traffic - which likely includes login creds and auth tokens for all those mainframes and java 8 apps.

(Note, I do exactly this a bit myself - terminate TLS at Elastic Load Balancers - and I feel a little dirty about it ever4y time I'm reminded... I sometimes wonder if I spend more time ensuring VPCs are appropriately isolated and keeping instances running untrusted or less trusted code out of vpcs with production customer data flying around unencrypted, than I would setting up to use encrypted data-on-the-fly everywhere. The big inertia holding that back is that we have so much legacy stuff running on stuff like Grails3 and Java8 that) he benefits of starting "doing it right" are not going to be fully realised for many years while those old platforms still need to run, and the added complexity of running two differently architected platforms is a big issue... I know what we should be doing, but the path to get there and the expense of travelling down it are high. We'll get there in "drip feed" mode where new projects and major updates to existing projects will do it right, but I'll be astounded if we don't still have some old untouched Java8 or Grails3 running in production in 5 years time...)

You don't have to forward traffic onto HTTP - ELBs & ALBs will happily forward traffic to HTTPS endpoints. That gives you an encrypted backend, but still allows you to manage the certificates & TLS policies in one spot. The backend servers can happily run on self-signed certs and the load balancer won't care.

Take a look at AWS nitro instances they give you "free" ipsec type network encryption with some caveats I forget.

The main caveat is that it only applies between Nitro instances: not AWS services like load balancer or databases, which would be one of the areas where it’d be most useful for anyone with legacy apps.

We ditched the the tls termination in nginx strategy, shifted to a nlb with tls termination in app. We used let’s encrypt because we needed something dynamic so we didn’t have to manage certs. Well we received one of those CISO forms from a big enterprise and one of the questions. Do you use LetsEncrypt? We said yes and the responded that it wasn’t a permitted tls solution. For what it’s worth I was happy with it.

Could you share what firms you're working with now so I can skip doing business with them?

Throw a rock and you will hit one

Might be more useful to get a list of companies that don't have requirements like this.

> In my experience, the larger organizations will have a "security" questionnaire required of their vendors, and the person administering it is a droid, incapable of evaluating whether the questions, originally written in the mid-00s and only updated for buzzword compliance since, are applicable to modern security practice today, or to the particular product/service/vendor in question.

And in many cases on the vendor side its some dude from sales filling it out... so pretty noisey on both ends.

Which seems somewhat questionable. It seems like you’d want someone to fill it out that had a less direct interest in a closed sale. Getting hacked is one thing but getting caught lying on a security evaluation could really harm the company’s ability to secure future enterprise customers.

> Getting hacked is one thing but getting caught lying on a security evaluation could really harm the company’s ability to secure future enterprise customers.

“Could” being the key term here: unless it’s epically bad and unequivocal, there are pretty good odds they could skate a long time without anyone asking and even if they did it might end up simply being that they blame long-departed sales guy and patch things up with the CIO over martinis.

> I'm sure it can be done. IIRC, Cloudflare doesn't use any firewalls

This is a little disingenuous because their product is a modern firewall. It drops packets and conditionally allows sessions to your backend.

While I agree with your assessment that Cloudflare serves as a modern firewall, I disagree with your assessment of disingenuousness. My lament was mainly at the more robotic and/or malicious "auditors" and "compliance officers" who will mark one or more line items as a critical failure if you don't have one or more Cisco Firepower/Juniper SRX/Fortinet Fortigate/Meraki MX/Barracuda NG/etc in your network such that its position in the network diagram means that it protects you from Evil Hackers. And AFAIK, while cloudflare's service is a sort of firewall, they don't use any traditional layer 4, layer 7, or 'next-generation' firewall appliances to get that result.

> any traditional layer 4, layer 7, or 'next-generation' firewall appliances to get that result.

Cloudflare’s product is just a web application firewall as a service. It’s not magic. The major difference between Cloudflare and a pizza box WAF is the global distributed nature to absorb ddos in the same product. The things it does to the packets are the same though.

Cloudflare not using any firewalls seems like a strange concept, considering they literally sell firewall-as-a-service.

https://www.cloudflare.com/waf/

A WAF is not the same thing as a general-purpose firewall. Think of it as a web proxy with filtering capabilities.

Yeah, I find myself kinda annoyed at the term WAF, as it overloads the term "firewall". But your description is quite accurate. Whether you're doing your filtering with an expensive F5 Big-IP with its nifty glowy logo on the front bezel, an haproxy instance from some WAF-as-a-service vendor, or Nginx and some plugins running on a VM, any of those, done right, can serve in a WAF role.

I say that now, but I wish I understood that a few years ago, when faced with the WAF line item on an a WAF, promptly went "WTF is a WAF? googles No, we have a WAF." Could have saved myself some security audit pain.

On the flip side, I did manage to get some upgrade budget out of failing that battery of line items.

It seems to me the basic definition of a firewall is:

"A firewall is a network security device that monitors incoming and outgoing network traffic and decides whether to allow or block specific traffic based on a defined set of security rules."

https://www.cisco.com/c/en/us/products/security/firewalls/wh...

"usually firewall \ ˈfī(- ə)r- ˌwȯl \ : computer hardware or software that prevents unauthorized access to private data (as on a company's local area network or intranet) by outside computer users (as of the Internet)"

https://www.merriam-webster.com/dictionary/firewall

This and other "web application firewalls" seem to meet these kinds of definitions. Add to the fact more and more "traditional" firewall appliances are adding behavioral filtering and have had application tracking for decades and we're far from firewall being limited to only a layer 3 device.

What would they need a firewall for? They have full control over the entire environment. They can (and should) just filter host-side.

Host level filtering doesn’t make it “not a firewall”. If they drop packets in the NIC before hitting userspace (they do this), that’s a firewall. Iptables is a firewall.

I am confident 99.9% of the people insisting on having you tick the checkboxes have no idea what "they drop packets in the NIC before hitting userspace" means...

Pretty sure getting your PM to just drop the firewall icon into their Visio diagram is a better way to meet stupid compliance requirements than explaining the difference between user space and kernel space to a just-graduated big4 consulting company intern "auditor"... /s

Exactly.

They may be even aware, they are just bound by their companys ruleset...

> ...incapable of evaluating whether the questions, originally written in the mid-00s and only updated for buzzword compliance since, are applicable to modern security practice today...

You just described my workplace. We have some rules that nobody understands and nobody remembers where they come from, but we have to follow them blindly. For example, they require that any access to the web services should go through a VPN, which would be fine if:

- The VPN actually worked, but it doesn't.

- The servers already uses TLSv1.3, all the services require user authentication, and there are 3 layers of firewalls and an integrated virus scanner in front of the services.

- We are an international project with people from 10 different organizations in 6 countries on 2 continents, and it's really difficult to impose these kind of rules.

So for example, I'm managing a GitLab instance that I can't use myself. I can only SSH login from a very specific computer to manage it, but I can't upload my own code from my office computer.

And I don't want to go into their blind devotion to the firewall and their concept of one way connections...

So I'm just letting time go by, until everybody is so angry they are finally forced to change. Doesn't help that this is Japan, the epitome of rigidness and "even it is broken, don't fix it".

> Never mind that a KISS setup tends to bring security because of its minimized attack surface

Security is also about depth. You should assume breaches can happen and have another level of defense.

That only increases the attack surface but it's a much better approach for imperfect beings.

In my experience that approach actually ends up leading to weaker security. When you have 5 or 6 security layers it's not clear which ones are important; people get confused about which parts can be safely bypassed and how, and you end up with a swiss cheese where sooner or later all of the holes line up. Having a really clear distinction between public and private services works better.

In a way yes. I'm sure everyone here has heard "This service can only be accessed on VPN anyway so we don't need authenticated access or care about security in the service itself".

On one hand, a firewall that accepts incoming port 22 connections isn't that different from only having port 22 listening.

On the other hand, a firewall is an explicit declaration of the ports you want open and who you want them open to, which seems like, at the very least, a useful thing to do. If nothing else it seems like defense in depth. I'm not sure I buy that a system designed around "default deny" is an increase in secrity complexity, certainly it's complexity that would hurt availability, but complexity that would hurt security?

Either way, the real security comes from monitoring the reality of what ports are actually open/listening and verifying a person's assumptions about their systems.

> [...] but complexity that would hurt security?

Higher complexity = larger attack surface.

For example, if they used a firewall with one of Cisco's infamous backdoors.

https://www.zdnet.com/article/cisco-removed-its-seventh-back...

Isnt a firewall largely a defense in depth thing? If everything is working perfectly and the firewall gets compromised, then yes, you'd just be listening to port 22 with SSH anyways, so it wouldn't really matter. But if something went wrong on the system behind the firewall (configuration mistake, software defect, malware, hacking) but the firewall itself was still secure, it would limit the damage that could be done.

A FreeBSD machine running as a firewall might make sense without significantly increasing attack surface. But inviting a new hardware product into your infra is a significant increase in attack surface. SolarWinds is a very fresh example of a catastrophic supply chain attack and a clear example why it might not be a defense in depth thing.

If your network is secure, and a well configured machine only listening on port 22 is pretty secure, you have to ask how the production machine will interact with the outside world. Well every update is an inverse remote code execution. You are getting remote code from an external location and then running it directly in production. So while you might trust FreeBSDs package manager, do you trust Cisco? Do you trust SolarWinds? Even if you do, it's hard to argue that your attack surface hasn't been increased.

I generally agree with the sentiment, but it's not strictly right.

For example you have your main machines set up securely, and requests go through the firewall. Of course a compromised firewall doesn't make it easier to compromise the main machines. But because your users are going through the firewall, _they_ might now become vulnerable to some classes of redirection attacks.

Even in a scenario where you're using the firewall as a passthrough, you're still looking at a scenario where (For example) your DNS entries are now pointing to a machine you have less control after. It might not mean that now HTTPS doesn't work anymore, but that (combined with some other mistake) might be enough.

One potential class of vulnerability might be related to recent git client issues: the software your client is using might have an issue that would be a security issue when connecting to an untrusted source. You wouldn't try to get a keylogger on your clients' machines, and the software is always pointing at your own domain etc etc. But the firewall vulnerabilities have opened up that angle of attack!

It's definitely a balancing act, and dependent on how much you trust each layer of your stack.

they just don't have dedicated firewall. That doesn't mean they can't run ipfilter or whatever, on the hosts themselves with default deny.

In fact he mentioned port knocking. You pretty much need some kind of host firewall for that.

Disney, 3M and ARM are on the website.

Reply on news.ycombinator.com