Hacker News Reader

Comment by jandrese | Hacker News Reader

> At the time optimized jar change broke antivirus scanners, which further sped up Firefox startup :)

I'm pretty sure that counts as cheating.

AV overhead is a real issue though. I was recently copying a directory full of small files from one fast SSD to another (larger) one. I started the transfer and Windows estimated about 8 hours to finish the transfer, and the overall progress seemed to be roughly on track. On a whim I tried temporarily turning off Microsoft's built-in antivirus scanning. Total transfer time dropped to about 40 minutes.

View on HN

Once, I was working on an intranet web tool generating .odt documents for the user to print.

It worked perfectly, except on one machine, where the document could be downloaded but LibreOffice raised an error when opening it.

At first I though I had a bug. Checked my code, nope.

Then I though maybe there is a problem with LO, but no, same version, nothing special. It worked with a document I brough from an USB disk.

After an afternoon of frustration I noted Kaspersky running in the task bar, and did one last desperate attempt at finding the solution by disabling it.

It worked.

Inspecting the file revealed some parts were missing, Kaspersky assumed it was a malware, and not only intercepted the download, but removed silently part of the data on the fly.

No alarm, no pop up, just a broken file.

Disabling the antivirus was not acceptable for the client, so I created a self signed certificate, and turned https on.

The virtual vigilante couldn't spy on the wire anymore, and the user could download her files, unharmed.

I had a development machine once that had the weirdest problem building one particular C# project. Always complaining that one particular Microsoft DLL was missing. Double check references and Nuget, everything looks right, do a full package restore, no joy. Antivirus was eating this particular DLL every time it was downloaded.

I copied a bit-for-bit identical version of the file from a different machine into the correct directory, and bizarrely that worked fine.

I had a very similar issue with BitDefender recently. When downloading the WebRTC C++ library, one of the submodules is harfbuzz, which contains some maliciously crafted font files used by the test suite. BitDefender automatically deletes those test files. That means git sees the repository as dirty, since files are deleted. That means WebRTC's fetch script can't run the 'git checkout' commands it expects to be able to run, and the whole download fails.

While my summary here is brief, debugging this whole mess took a while.

Let me present you McAfee copying a folder full of zip files, at least this is what it did five years ago. First of all, it is horribly slow scanning any kind of zipped content. Second, it scans a whole folder if you touch one file. Third, it forgets you scanned a file if it gets modified, even before closing the file.

So you copy a buffer full (64k?) of bytes, and mcafee scans the full folder. Then you copy another buffer full, and it scans the full folder again. And so on.

I saw the windows estimated copy time go down from multiple days to a few minutes by adding an exclusion on a folder.

Another fun McAfee tip is to keep your recycle bin as empty as possible, especially for zip files, because windows will look at the content regularly, so mcafee keeps on scanning it again and again.

McAfee is still utter garbage (like, well... most "security" software) that will make an NVMe SSD look like a late 90s HDD.

> Another fun McAfee tip

Here's my favourite one, which was even supported by the man himself: uninstall it as soon as possible!

It's not only a pretty bad antivirus, but seems to be straight up dangerous, having had some of the most destructive bugs I've seen in any software (like the time it bootlooped millions of PCs around the world- including hospitals and police stations!). There are so many better options available, I'm amazed anyone actually uses it (unless 100% of their install base is people who got it preinstalled).

You're my new best friend! I'd give you the contact for corporate IT, but even they ran away to MS Defender.

McAfee's users are not the people who suffer it on their PC, but the manager who love a big name and hate a migration, and the windows domain admins who love the centralized console. The damage caused by user frustration, lost time waiting, troubleshooting, and actual hardware obsolete before its time is not their concern.

I can tell from experience the windows domain admins disable it on their own laptop, as they 'know what they are doing', unlike the rest of us.

Wild that windows feels it important to scan every file going from one local drive to another. my laptop has gotten a lot quieter after turning off virus protection, I wonder how much electricity is burned just idley checking for viruses.

Edit: thanks for the replies, on second thought it is pretty tricky to know a file you’re writing already exists without trusting the process doing the copying

Interesting question! When mankind will really need to calculate costs and try to get rid of such an entrenched "habits" as av-scanning (same as to crypto-mining nowadays), it could be too late to do something.

Or take banks. I was amazed to see how much manual work is done in financial institutions. Skyscrapers with bank name on top of it filled with tens of thousands people. Everyone has computer, among other things burning electricity. Separately, datacenters. How does it compares to crypto? I doubt it's less, if sum up all banks.

But anyway, av-scanning is doing much good, but can be socially out-engineered in a single click, making it useless in most critical cases.

Re: banks/crypto

Crypto's current energy consumption is doing a miniscule fraction of the transaction handling that banks are. If you scaled up current crypto to match banks, I don't think that would even be possible.

Crypto's power consumption is not truly related to the transaction volume.

Right, the transaction volume it can support will never change, but the energy consumption rises as a function of the price, with no upper bound.

... every piece of that sentence is straightforwardly incorrect.

Do you want to explain how?

AFAIK The number of transactions per block and the rate at which blocks are confirmed has nothing to do with how many people are burning through hashes.

And the price of a token is a kind of bounty for mining a block: the higher the price, the more money can be spent trying to win the bounty while still being profitable. You wouldn't see data centers full of ASICs if the reward was only a few hundred bucks.

So how straight forward is it?

Their:

>the transaction volume it can support will never change

is plainly ridiculous (as proof: transaction volume has changed, and multiple changes have been successfully made that increase the peak volume), and is also covered by your:

>AFAIK The number of transactions per block and the rate at which blocks are confirmed has nothing to do with how many people are burning through hashes.

as for:

>the energy consumption rises as a function of the price, with no upper bound.

though there's obviously some relation to price and electricity use (i.e. hashing power, ignoring tech improvements), that direct relation really only makes sense with the per-block rewards... which keep halving, and that does not immediately lead to 1/2 hashing power. the market has repeatedly demonstrated that price and value-per-block do not directly relate to hashing/electricity. on top of that, transaction fees are taking a larger and larger proportion of it, and that's related pretty much solely to transaction volume (which does not noticeably affect electricity, as you pointed out), not value per coin or electricity.

as far as "no upper bound"... only if you ignore market effects, like if electricity became free globally. since it's market-driven though, as beyond a certain point it's not profitable, you can't ignore that. and again, hashing power has dropped many times in the past, clearly demonstrating effective upper bounds at any given time.

Can you explain in which ways is it incorrect? I am puzzled. I guess that what was is incorrect is using the word "crypto" instead of "POW blockchain-based"?

after a halving (every 210.000 bitcoin blocks, roughly every 4 years), all other things being equal (bitcoin price, energy price, bitcoin fees, … and enough chip production), the energy consumption would get cut in half (as the reward gets cut by two).

This fact should stay true for another 2-3 halvings (including same energy consumption if price doubles which is the same), until the factor 0.5 (energy consumption after halving) starts to increase to 1 when/if higher fees compensate for the diminished issuance of new coins (while issuance of new coins goes to 0).

Because fees for now are very low, in the big picture and almost not relevant in the power consumption calculstions.

But until these about 2-4 halvings happen (=Y), price might increased by factor X and energy consumption only X/2^Y (which could even be a decrease if price does not go up much!). And if price would stay the same, then energy consumption would actually drop by 50% every halving! (if all other things being equal and supply of miners and mining chips were not a problem).

So to sum it up differently: Bitcoin consumes a lot of energy to have a fair new issuance of coins. Instead of being created out of nothing and being distributed unfairly (worldwide privacy issuance of a new coin is REALISTICALLY very very very hard, and has been tried by many projects and all failed), bitcoin gives you new coins for the same amount that you burn energy, so you can‘t fake it or corrupt it in the hands of politicians or any other left/right/religious/… group

I’ve always found amusement in the ironic carbon navel gazing found on message boards.

Sure, but banks those banks are actually providing value for large numbers of real people right now; cryptocurrencies have yet to show that they can do much what has been claimed about them.

I recall paying for a coffe with bitcoin once. The transaction took 30 minutes despite an above average fee which was much more than e.g. PayPal would have taken. This was when I knew that bitcoin is unsuitable to replace other payment providers.

Most manual financial work in banks is not on mere transactions. It's on higher level structures such as doing due diligence on lending. A small fraction of the employees are actively keeping the basic transaction mechanisms going. Rest assured that fraction that is working on those is, per transaction, a mere fart in the proverbial hurricane that is crypto energy use per transaction.

> Interesting question! When mankind will really need to calculate costs and try to get rid of such an entrenched "habits" as av-scanning (same as to crypto-mining nowadays), it could be too late to do something.

> Or take banks. I was amazed to see how much manual work is done in financial institutions. Skyscrapers with bank name on top of it filled with tens of thousands people. Everyone has computer, among other things burning electricity. Separately, datacenters. How does it compares to crypto? I doubt it's less, if sum up all banks.

You'd be likely wrong. Bitcoin now uses the same energy as the whole country of Argentina with 43 million people. For comparison Bank of America has 200 000 employees. Are there more than 200 banks of that size, maybe but I'm doubtful. Moreover Argentinas energy consumption includes vastly more energy intensive endeavours than banking (manufacturing, mining etc). from that estimate I would be very doubtful that the whole banking sector uses more energy than bitcoin mining. Moreover as others pointed out the banking sector serves many more people with many services that bitcoin can't provide.

I think it's probably because the system doesn't see it as "copying", it sees it as "writing arbitrary data that just-so-happens-to-be identical to existing data in another open file". It'd be tricky to figure out (and verify in a secure manner) that data is in fact being copied.

I mean, if you're literally using the Windows utility for copying files (either by clicking copy/paste in Explorer or using whatever the DOS equivalent of "cp" is), it seem like it shouldn't be that tricky for them to make it work at least for the built-in AV, which is what the parent comment was talking about.

I think that depends on whether you want to trust already-executing programs or not. If you trust them, then sure, it's pretty straightforward: you just tell the AV what to scan vs. not. If you don't, then it's not: you have to either offload the copying to kernel-mode, or delegate it to a secured user-mode process, because otherwise a malicious program could just modify the buffer you're copying from. The former makes sense from a performance standpoint but the latter also makes sense from a security standpoint.

An issue is the original files are not guaranteed to be trusted. If for instance your were copying files from a flash drive or a network permission, it would seem natural to rescan every file that is copied to a local disk.

From there the more the "outside"/"inside" line gets blurry (e.g. what if it was synced from Dropbox ? or data got injected by an arbitrary program etc.), the more it becomes justified to just scan every file moving anywhere. If the goal is indeed to scan files in the first place.

If I remember correctly the CreateFileEx win32 api takes a flag and template file handle to copy when creating a new file. Maybe the copy operations can use this api and expect AV software to ignore the operation. (I have no idea about the performance of such approach) Or maybe a syscall (preferably at filesystem level) that can be introduced for bulk copying purposes and AVs can ignore such calls.

The template parameter is related to attributes, not contents. AV would still have to scan the data that is written to the file.

[in, optional] hTemplateFile

A valid handle to a template file with the GENERIC_READ access right. The template file supplies file attributes and extended attributes for the file that is being created.

> It'd be tricky to figure out [...] that data is in fact being copied.

For other operating systems, sure, but the Win32 API has a function literally called CopyFile (and its variants CopyFileEx and CopyFileTransacted).

I don't know under what circumstances Windows scans files but I do no that copying a file to the same drive does not rescan the file. I found this out because grunt-copy-contrib copies incorrectly by opening the src, reading bytes, writing to dest instead of just asking the OS to copy the file. Switch it to ask the OS to copy the file removed the scanning (windows already knows it scanned the file so no need to scan a copy).

One hasn't felt pain until one has dealt with large npm-based projects on a corporate machine that has an hdd and two competing antivirus scanners battling to scan all 18,000 files in node_modules every time you run webpack.

the built in AV scans on close of file handle. moving this to a background thread will really speed things up.

https://groups.google.com/g/mozilla.dev.platform/c/yupx2ToQ5...

Found the article that goes into depth on this issue

https://gregoryszorc.com/blog/2021/04/06/surprisingly-slow/

and the bug link:

https://www.mercurial-scm.org/repo/hg/rev/2fdbf22a1b63f7b4c9...

This kind of thing really makes me wonder how safe these AV systems really are. Code that runs on every single file dropped into the user's hard drive seems like a great vehicle for zero click exploits.

At a previous job we discovered that the software was strangely slow when running locally. It turns out that the antivirus (ESET, I believe?) was delaying every WCF request by 0.5 seconds. After adding encryption to the connections the delay magically went away.

Even worse than that, I've had my antivirus freak out and BSOD the system during copying too many files around inside WSL1, where "too many" was "checked out a nontrivial git repo." It was repeatable. It meant WSL1 was just not usable at all.

You can add the top directory you use for dev projects to Windows Defender exclusion list. Also, your build tools or IDE processes can be added. For example excluding devenv.exe has a very important impact on Visual Studio performances.

Anything that reduces the startup time users experience is not cheating.

Took me longer than it should of to realise Windows Defender scanning everything in wwroot everytime we published a new build of the site was significantly increasing downtime!

A part of me wants to suggest that you look into using something other than Windows for hosting sites, since in my experience GNU/Linux servers are far less painful to deal with, especially when it comes to services and file permissions. However, it's likely that you have your own reasons for using it (e.g. many projects out there are stuck with legacy versions of .NET and Mono wasn't really stable for that use case either).

On the other hand, i can't help but to wonder about why we don't see more anti virus/malware scanning solutions for GNU/Linux, especially with how coupled our development workflows are to the Internet - i've had Ruby gems with malware in them be stopped by Kaspersky on Windows, and honestly the thought of pip, npm and many other package managers being able to compile god knows what and execute random scripts in a non-sandboxed environment is horrifying.

Market share aside, i bet we still see malware targeting *nix be developed, especially in the age of pre-compiled binary blobs and such, especially when the complexity of everything increases to the point where you can't audit everything that's present with your current resources, which will probably only become a larger problem with time.

So, why don't we have an equivalent to Windows Defender that's either preinstalled or recommended during installation, like you can choose between different desktops while installing distros? Why don't we talk more about something like https://www.kaspersky.com/small-to-medium-business-security/... ?

Windows Defender is the worst AV in term of performance impact on the system. It's so significant in IO application that I don't understand why it is not number one top priority for the Defender team. It has been like this for several years. I have a paid AV just for this issue (avoid the free one).

https://www.av-comparatives.org/tests/performance-test-octob...

That's primarily a Windows problem though, right?

McAfee exists for Linux, it's just as bad as you'd think it is, and some of us have the misfortune of being forced to use it on every workstation and server.

> I'm pretty sure that counts as cheating.

It also shows how useless most AV is.

Reply on news.ycombinator.com