Barracuda urges replacing, not patching, its email security gateways | Hacker News Reader

Barracuda urges replacing, not patching, its email security gateways

krebsonsecurity.com

369 pointsBender117 comments

Barracuda is providing replacement hardware at no cost. This critical piece of information is buried halfway down the article:

> In a statement, Barracuda said it will be providing the replacement product to impacted customers at no cost

Obviously the time and effort to replace a device isn't free, but at least they're doing the right thing by acknowledging the issue and doing what they can to fix it definitively.

Way cheaper than a lawsuit!

But correct - its the right thing to do regardless of the true, ultimate motivations...

But what I'd like to know is the impact of this? Like - how much corporate opportunity loss may have been created through information breaches which were unknown... That will never be known, unless we can assuredly say 'none' which is doubtful...

>Barracuda said the vulnerability existed in the Barracuda software component responsible for screening attachments for malware

Heh -- uhm... Isn't that like _THE_CORE_ component of the devices job?

Aside from ensuring filters on attachement egress blah blah...

>No other Barracuda product, including our SaaS email solutions, were impacted by this vulnerability

I'd like to hear them directly say that this specific ESG device line was NOT used in their Email SaaS offerings?

And to know exactly what ESG kit they are using?

Seems like a reasonable request if you're a large (or any) customer of theirs...

You’d be surprised how many “virus scanners” are implemented with poorly coded C parsers running as privileged user with no sandboxing… just saying, kind of asking for trouble.

I remember awhile back the lead of the chrome dev team getting into a twitter fight with some engineer from a popular antivirus tool because, as I recall, AV tools are basically just more attack surface. I tend to agree.

Windows/client AV tools have a who-watches-the-watchers problem. If you're trying to detect threats that exploit OS, you need to be even lower level. But if you're lower, then you're an even more inviting target.

Unfortunately, add on that no customer makes a profit off their IT security org. Which disincentivizes excellence and incentivizes feature check-boxing at the lowest price point.

Which ultimately produces a highly privileged piece of software developed on budget salaries.

One reason MS security was a game changer (once the company got off its butt and admitted security was an existential threat to their OS sales) -- they could afford to burn great magnitudes of money to deliver.

(No offense intended to any of the amazingly brilliant non-MS Windows AV folks out there)

Indeed it does happen https://www.cvedetails.com/vulnerability-list/vendor_id-26/p...

I’m still not entirely convinced that virus scanners do anything useful at all besides hog resources.

Malware Bytes helps identify, stop and remove malware Crypto Miners!

In my social circle, the kids Gaming PCs getting infected is very common.

Recently one of my boxes got hacked via a QBitTorrent exploit, and I didnt have Malware detection running, other than the built in W11 system.

I installed Malwarebytes and it detected and correctly removed the malware crypto miner.

FYI they exploited QBT via the web interface, which had default settings, but wasn't exposed via port forwarding to the web. It might have been via UPNP, which was enabled. No idea - but it's a common exploit used to DL a torrent then run a post DL script .bat file to DL and run a crypto miner.

I'd literally had QBT running on Windows for a couple of weeks, having switched from a dockerized setup on a Mac Mini. How Windows allows the running of a .bat file to DL an .exe that can run a crypto miner is just a bonkers lack of security!

Naturally I had to nuke the box from space anyway :-)

You dont have to answer, curious - how old are you?

(you speak with a fluidity I did in my gaming heyday - but I am over the global heap at this point.)

>How Windows allows the running of a .bat file to DL an .exe that can run a crypto miner is just a bonkers lack of security!

Is it any different from Linux allowing the running of a .sh file to DL an ELF executable that can run a crypto miner?

Yeah, all these cool kids showing off their shell skills with their new js frameworks distributed like ‘curl https://djdhdhdb/dish.sh | sh’

I think a big difference here is Linux distros don't include an active anti-virus while Windows does.

The "active" Windows anti-virus is quiet inactive when it comes to viruses.

> No idea - but it's a common exploit used to DL a torrent then run a post DL script .bat file to DL and run a crypto miner.

Who or what triggers the execution of that script? If it is QBT, it almost sounds like malicious intent from the side of its developers.

Perhaps this is what they were talking about: https://www.cvedetails.com/cve/CVE-2019-13640/

> In qBittorrent before 4.1.7, the function Application::runExternalProgram() located in app/application.cpp allows command injection via shell metacharacters in the torrent name parameter or current tracker parameter, as demonstrated by remote command execution via a crafted name within an RSS feed.

> How Windows allows...

It's either that or iOS-type walled garden, which one you prefer?

By the way, Windows even has the lock-down switch, it's just off by (a sane) default.

Also, blaming OS for not protecting from random app exploits isn't fair.

A virus scanner needs to implement parsers for so many different formats on the budget of a medium software company, with a pressure to deliver fast. the chances of not making any mistakes are small, and antivirus needs to run with high privileges.

The parsers do not need "system priviledges" to run.

What could go wrong with a process running with the highest priveliges parsing every piece of data passing through the system. Windows defender has had some nasty ones, uncovered by project zero

>Isn't that like THE CORE component of the devices job?

Chances are it's so core that some buggy LZMA or something implementation was offloaded to ASIC.

Hopefully (and I imagine this is likely) barracuda ships the new device before customers are required to send the original.

HW might be free, but the logistics are going to be really expensive.

The important part is that barracuda isn't saying "we fucked up, now pay us more money if you want it properly fixed"

This actually seems great. They’re taking the hard path here admitting the devices are compromised, that the state can’t be trusted, and that the machine needs to be tossed. Then they’re offering to replace the vulnerable devices.

I’m used to companies taking the “deny, deny, deny” route and these articles are then written by their community saying the response is insufficient.

Agreed. Better response than normal. Also gives them an avenue to see how many have been actively used? Maybe that's just me.

Who would want a replacement "email security gateway" after the previous version was found to do the equivalent of system(<untrusted email input>)?

When is our industry going to agree on a standard where we go "no, that is too dumb, too far"? This "oh there was an issue but it's fixed now" doesn't begin to address the severity of the fuckup this is.

It doesn’t take much for a machine to become irrevocably tainted, so the fact that they are doing a recall doesn’t, in and of itself, say much about the magnitude.

Do you have some other data to substantiate your ire or are you just piling on?

From the security advisory:

> The vulnerability stemmed from incomplete input validation of user supplied .tar files as it pertains to the names of the files contained within the archive. Consequently, a remote attacker could format file names in a particular manner that would result in remotely executing a system command through Perl's qx operator with the privileges of the Email Security Gateway product.

I would not be comfortable to continue using a product that contained such an egregious flaw.

Improper input validation chained to command injection? Those are quite common unfortunately.

So you will never buy a car because every manufacturer at some point already fucked something up with accelerate/break which means none of these can be trusted anymore?

One of the first things I virtualized was a Barracuda spam appliance, the fans were extremely loud and the AMD chip was slow, so the interface to your spam inbox was slow, etc.

Pulled the hard drive and loaded it up into VMmware ESX 2(?). It flew and there was one less server/appliance in my rack.

Barracuda was one of this early companies that made a very useful product out of free software. I idolized that and wanted to work there.

Then they were taken over by some outside company, started making all kinds of products that weren't in their core, support faltered...I'm kind of surprised they are still around.

> Barracuda was one of this early companies that made a very useful product out of free software. I idolized that and wanted to work there.

What software? Is it just ClamAV in an appliance?

ClamAV and SpamAssassin

If I'm reading between the lines correctly...

After a device is compromised, patching it won't help. Their software team (quite reasonably) says that after anyone else has got root on the system, their software patch system probably can't reliably be sure to get rid of the malware.

So, they are offering to replace any device in this position. The replacement is probably a refurbished one from another customer, but has gone back to the factory to be flashed by some debug port - and using that you can wipe all the flash memory and be pretty sure there is nothing evil left.

Given that this looks to be just a particular build of Supermicro server, I wonder what mechanism the malware uses to achieve persistence such that a reformat or FS restore wouldn’t take care of it. Does anyone know if these devices have supermicro IPMIs on them? Those are notoriously insecure (like most lights out managers) and a great place to hide malware persistently.

Edit: Typo.

Is this the same super micro that hadd physical backdoors inplemeted by china?

https://www.bloomberg.com/news/features/2018-10-04/the-big-h...

Yes except that article has remained wholly unsubstantiated by facts for five years. It seems to have been bunk.

That article captured the imagination of the public, but has been derided by any and every cyber security professional worth their salt almost from day dot.

Despite this, Bloomberg have refused to retract it or supply any credible sources, presumably because and continues to draw traffic.

And yet "do we have any backdoored supermicro hardware" is a common question from non technical management, even today. It's infuriating that it hasn't been retracted as I'm still talking about it. It's also at the forefront of insurance questions

Just goes to show how much of this word is theater rather than pragmatic improvement.

Given that outsize influence, you would think they would sue Bloomberg… I guess they don’t really care…

Can this be interpreted as "they dont sue because they can lose - the article is true"?

Easily solved by said security professionals auditing the security of the devices, refuting the claims in the Bloomberg article, and throwing it all up on a blog with catchy “No, china didn’t backdoor servers headed to US gov datacenter racks” title.

It would get awkward when they find the US backoor built in to every server headed everywhere ... I guess technically outside the scope of the headline so you can just leave it out.

The point is, if the original article is bogus and full of lies it should be trivial for the security community to debunk any back door claims and set the record straight, yet that isn’t being done while still calling for the original article lacking any proof be taken down.

Someone on one of these sides needs to provide evidence to support their claims. Clearly the authors of the original article have no intention or it would have been done already.

They haven’t taken it down because it continues to draw in traffic.

It’s trivial to debunk because the story was a hack job with no sources to begin with: the burden on proof is on the accuser in this case.

You’re describing every follow-up podcast on the matter.

Write ups reviewing the malware never suggested such persistence. This seems to be something done out of caution rather than a specific finding.

The writeups I’m reading suggest that the malware was specifically designed to maintain persistence via firmware (i.e. rootkits), typically attributable to state actors.

They're not all Supermicro servers. Some are other mobo vendors, I've seen some ESG 400s with MSI boards in them.

If your IPMI is in any way exposed to anyone other than your administrators, then you have other problems. These interfaces should be segmented away from all other networks, irrespective of any vulnerabilities they could have.

Some supermicro will default to put IPMI on the shared primary nic if the dedicated IPMI nic has no link at poweron.

Some? Every one that I’ve ever worked with, which is several hundred.

and why would IPMI port ranges and IP addresses be accessible outside of the DMZ, if you're an even half competent sysadmin?

If you are halfway competent, you do not use inband IPMI at all. IPMI belongs on a separate ethernet jack or at least a separate VLAN, connected only to a separate secured admin network. Which sysadmins can connect to, but which doesn't have connectivity to other parts of your network or the internet.

But I agree that you should filter all the relevant ports in all networks, just in case somebody screws up.

That's great and all, but if there was a known barracuda OS vuln that allowed attackers to gain root access, then they could gather information about the hardware, which includes the ability to interact with the IPMI (which runs it's own separate OS) to update the firmware or whichever way they're able gain persistent access to the IPMI device's OS.

This points out a major issue with IPMI and "management engine" components on motherboards. If a vulnerability is found at the lower levels, you may have to replace the hardware. Vendors may be more reluctant to put that stuff in if it leads to legal liability.

IPMI firmware can be patched, as can the BIOS which contains the microcode that is loaded. Once the OS boots, updating the running microcode included is trivial (in Linux and I believe also FreeBSD?) and can be done before the system's network interfaces come up.

You can't patch the management engine without the cooperation of the management engine itself, right?

Shouldn’t[0] be able to. It’s all still fallible software written by humans.

[0] https://www.theregister.com/AMP/2022/06/02/conti_rasomware_i...

Careful what you ask for Barracuda: we were already looking at replacing ours with non-Barracuda hardware and now that you are saying old gear cannot be updated all I can say is thank you for making my case to my managers.

They note that "In a statement, Barracuda said it will be providing the replacement product to impacted customers at no cost"

Would you still switch to a non barracuda device ?

If you have to go through the cost of a replacement anyways - as in, the time and effort to actually do it - the cost of the hardware matters a lot less.

I suspect there is a higher level question at play stemming from the "deny" comments. Do you think Barracuda is uniquely vulnerable to these types of threats and, if not, would their competitors make similar efforts to recover?

If this is the mindset of some people it's no wonder that companies would rather deny the problem and stick their heads in the sand.

The mindset that changing a system out has risks and that engineers/ops time is very expensive?

I mean, I hear you, it sucks that being honest about the issue is leading to punishment, but rationally this is just how people are going to end up responding. If they really want to avoid this, don't give it to them for free, pay them for it.

No it’s not. Considering changing vendors because a vendor did the right thing and sent out free replacements for potentially compromised hardware is not something any sane client would do.

The GP didn't say that though. They said they were already considering replacing one vendor with another, prior to this incident.

As I understood it, the calculus was 'do nothing and keep the current barracuda stuff for a while longer or find another vendor and replace the barracuda stuff?' Now the former option is gone, and the calculus is 'replace current barracuda stuff with barracuda or another vendor?' So if you were already considering switching vendors, now is an opportune time to do so. You're going to be paying the eng/ops cost now anyhow, and as many in this thread have pointed out, one or two fairly off-the-shelf hw appliances are not exactly a big spend.

>You're going to be paying the eng/ops cost now anyhow,

No you’re not. The people in this thread saying that have never actually managed technology because a vendor change that also requires a hardware swap is significantly more expensive than the drop in hardware swap.

The only exception to this is if you were already in the middle of the transition and were planning to phase it out shortly anyway.

This notion is detached from reality. A drop-in replacement of the same thing is an order of magnitude less work than switching vendors.

Even adequately researching alternatives, let alone doing some trial testing, will take far more time than a swap.

They've already been researching alternatives.

the physical cost of the equipment is often the cheapest part of a deployment.

Given that the replacement will be free, doesn't that make your case harder to make?

He's already made up his mind, so he'll probably just leave out this important tidbit when discussing the issue with "management"

It’s certainly not free to arrange getting the new device, swap, reconfigure, and hopefully you have HA to avoid downtime.

> certainly not free to arrange getting the new device, swap, reconfigure, and hopefully you have HA to avoid downtime

Compared to migrating to a new vendor?

It is an email server. You can set up some kind of system that “catches” incoming email while you take your hardware offline and replace it. Your internal users won’t be able to send or receive emails but anyone outside the organization would be none the wiser that your system was down. If you notify ahead of time and can keep downtime to under an hour, will it matter?

Which you would need anyway if you switch to hardware from a different vendor.

Can I ask - which competing devices are you planning to replace these Barracuda email security gateway servers with? Thank you.

A large portion of organic at this point run their mail with Google or Microsoft, and are already paying for capabilities in those systems on par with third party offerings.

There is often a counter "but I still get some spam with exchange online/google workspace", to which I would counter that you will still get spam with a barracuda.

You’ll get false positives and negatives with any email gateway - even ML or “AI” backed ones. The offerings from both Google and Microsoft, while adequate, are someways off the efficacy of other solutions.

Is it likely they will analyze the firmware on the ones that come back, then wipe them and resell them either thru a refurbisher or even put the new software on the known-good server and use elsewhere, such as in the datacenter where they have their SaaS offering?

I wonder what bcantrill’s take on this will be.

He’s been on his personal holy war on firmware for years now, I’m not joking, I’m curious to read his opinions on this issue.

Maybe barracuda could use some kind of standalone 2u oxide server instead of supermicro servers? ;)

With all due respect (and I really mean that — bcantrill is absolutely deserving of tremendous respect), why would using an oxide server for the hardware be any different (better or worse) than a SuperMicro server?

And further, is it even possible to get Oxide equipment yet? Is there even a timeline? Or is it still vaporware?

> why would using an oxide server for the hardware be any different (better or worse) than a SuperMicro server?

Secure hardware can attest that its firmware is what it should be and doesn't have any persistent implants in it.

Then something has to safeguard that firmware.

I don't know if you're trying to prove secure hardware is impossible by induction or something, but it does work.

A lot of questions in there! Taking these in order:

1. We aren't making standalone servers: the Oxide compute sled comes in the Oxide rack. So are not (and do not intend to be) a drop in replacement for extant rack mounted servers.

2. We have taken a fundamentally different approach to firmware, with a true root of trust that can attest to the service processor -- which can turn attest to the system software. This prompts a lot of questions (e.g., who attests to the root of trust?), and there is a LOT to say about this; look for us to talk a lot more about this

3. In stark contrast (sadly) to nearly everyone else in the server space, the firmware we are developing is entirely open source. More details on that can be found in Cliff Biffle's 2021 OSFC talk and the Hubris and Humility repos.[0][1][2]

4. Definitely not vaporware! We are in the process of shipping to our first customers; you can follow our progress in our Oxide and Friends podcast.[3]

[0] https://www.osfc.io/2021/talks/on-hubris-and-humility-develo...

[1] https://github.com/oxidecomputer/hubris

[2] https://github.com/oxidecomputer/humility

[3] https://oxide-and-friends.transistor.fm/

Thanks for your reply!

The vulnerability found was an RCE, and apparently at least 5% of their devices are infected.

  > The pivot from patch to total replacement of affected devices is fairly
  > stunning and implies the malware the threat actors deployed somehow achieves
  > persistence at a low enough level that even wiping the device wouldn’t
  > eradicate attacker access.

Very interesting. I'd like to think it's probably some technical limitations in the recovery options rather than some weird magickery to persist beyond drive reset. Because, if it's not, everyone is screwed.

This is a problem of bad engineering: not using an immutable OS firmware model with a spare.

There ought to be 3 physically-separate flash devices: bootloader (never changed), OS (managed by bootloader - updates deleted unless signed correctly), and data (wipeable by a physical reset button).

If you don't do this, then you're inviting implants a priori.

And what do you do when a bootloader vulnerability is discovered then?

Why doesn't the NSA just AUDITS actual source code of the software that it deems it deserves to increase its security?

I mean it's obviously an interest to national security.

They could do that while maintaining the backdoors they want to keep so they can have an edge on cyber warfare.

Why would a vendor give their source code to the NSA to attack and also risk leak?

The NSA does not protect the public's computers, they attack them.

Technically, they do both. Very confusing.

What makes you think this wasn't the NSA?

NSA can’t even stop their own tools from leaking, never mind take on this.

> NSA can’t even stop their own tools from leaking, never mind take on this.

The NSA has kept plenty of vulnerabilities under wraps, the tool leaks were a pretty rare occurrence of the KGB/GRU trying to burn NSA ops without directly involving themselves.

Nice, first time in my life I consider buying something from Barracuda. But looks they sell to 1500+ desktops && 5+ officess. Oh well... ;)

But:

"somehow achieves persistence at a low enough level that even wiping the device wouldn’t eradicate attacker access"

It started. Barrakuda at least is nice to replaces hw but soon vendors will be bored by replacing shit they themselve making and customers will be left with voulnerable systems. Simple x86/PC ecosystem is shit, intentionally crafted or not. And, very, very sadly on whole globe only EU is capable of enforcing something to be better. But chances are minuscule...

Seems like they can only patch the application remotely - and not the OS - at least not at the same time (patch OS - then after reboot - let the app look for updates. The time between could have been enough to compromise it again)

Alternatively, they can patch both but the malware achieves persistence in firmware. Which may not even be a sign of particular sophistication, depending on what protections Barracuda had in place.

Or it could be as simple as that the software update application on the device has been patched to re-add the malware after flashing the software update.

What does this appliance do? Is it just a SMTP proxy with a virus scanner and spam filter? How is it better than software filters on the mail server?

> SMTP proxy

I don't think it's a proxy; it's an ordinary SMTP server, isn't it? All SMTP servers can be configured as relays.

I imagine people buy these Barracuda appliances for CYA reasons, and because there's a support contract.

What's funny is that obviously, the Barracuda "appliance" is a standard Supermicro 2U server :)

This is often the case. VM appliances are all little Linux VMs where the user just doesn't have access to the internals. Heck, that's basically Docker containers too.

Also, Barracuda cheaps out on these massively especially at the low end. I've seen, in the modern Core i years, Barracuda 1U appliances powered by Pentium III processors. I suppose they are powerful enough for the job Barracuda is asking of them, but it's worth a chuckle to see how many years old the chipset they're shipping is.

The hardware is absolutely the cheapest part of the stack for them.

>“One of the goals of malware is to be hard to remove, and this suggests the malware compromised the firmware itself to make it really hard to remove and really stealthy,” Weaver said. “That’s not a ransomware actor, that’s a state actor."

China?

Is the message archiver also affected?

> “No other Barracuda product, including our SaaS email solutions, were impacted by this vulnerability,” the company said.

This could have been a software update if they had used an FPGA.

We know already...

https://news.ycombinator.com/item?id=36061772

https://news.ycombinator.com/item?id=36136705

https://news.ycombinator.com/item?id=36143926

https://news.ycombinator.com/item?id=36156908

https://news.ycombinator.com/item?id=36233472

https://news.ycombinator.com/item?id=36238822

https://news.ycombinator.com/item?id=36248328

https://news.ycombinator.com/item?id=36255901

https://news.ycombinator.com/item?id=36261519

https://news.ycombinator.com/item?id=36263581

https://news.ycombinator.com/item?id=36267639

None of those submissions have more than 13 points.

"We" doing some heavy lifting here

Ah, the Royal "We".

The editorial

I only saw it on twitter, not here.

The convention on HN is to link to previous threads only when there are interesting comments there. If readers click on a link like this and don't see that, they'll be disappointed and sometimes come back and downvote the link. That's probably what happened here.

You're of course right that there were lots of submissions but on HN reposts are fine until a story gets significant attention, and the current thread is the first one to do that.

Reply on news.ycombinator.com