Hacker News Reader

Comment by cebert | Hacker News Reader

We allow uploads with pre-singed URLs at my work too. We added a few more constraints. First, we only allow clients to upload files with extensions that are reasonable for our apps. Files uploaded to S3 are quarantined with tags until we validate the binary contents appears to match the signature for a given extension/mime-type with a Lambda. Second, we use ClamAV to scan the uploaded files. Once these checks have completed, we generate a thumbnail with a Lambda and then make the file available to users.

I’m honestly surprise this isn’t a value-added capability offered by AWS S3 because it’s such a common need and undifferentiated work.

View on HN

Virus scanning user uploaded files always seemed like a very sensible thing to me.

I've used ClamAV before in web apps, and it was fairly easy to get working - but it has a few big flaws: it's very slow, it uses a huge amount of memory, and detection rates are... honestly, pretty terrible.

I've used Verisys Antivirus API for 2 recently projects, and it works great, a huge improvement over self-hosted ClamAV. There are a couple of other virus scan APIs available, including Scanii[1] which I think has been mentioned on HN before, and I Trend Micro has something too - but their pricing is secret, so I have to assume it's expensive.

[0] https://www.ionxsolutions.com/products/verisys-virus-api [1] https://scanii.com

How are those better than ClamAV? I am not saying they aren't but you just said the detection rates of one is terrible compared to the rest.

Things got a lot better for ClamAV after the Cisco purchase, but detection rates remain attrocious. There are some commercial signatures available, which would certainly help, but it still doesn't come close to the detection rates of commercial scanning engines.

ClamAV needs lots of memory to work - we needed at least 8GB in one project I worked on (where we also used commercial signatures), 4GB was just about enough for another. The more signatures you have, the more memory it uses.

ClamAV chews through a lot of CPU, and it's so very slow. For example, scanning a 25MB MSI file from my test corpus takes 65 seconds - and that's on a fairly beefy machine!

I've used both of the services linked above, and both seem excellent - detection rates and performance are far superior to ClamAV, and they Just Work(TM).

Azure Storage seems to have an integrated AV option:

https://learn.microsoft.com/en-us/azure/defender-for-cloud/d...

Mind you, last time I designed something using this we just used ClamAV - which is pretty easy to develop against even if it is a slight pain to manage on an ongoing basis.

Is using ClamAv because of some compliance checklist that forces you to include a virus scanner in there? Otherwise I don’t see how adding a ton of additional complexity with a proven track record of CVEs would add any security at all.

No compliance reason other than adding it in couldn’t hurt.

How does

> a ton of additional complexity with a proven track record of CVEs

become:

> adding it in couldn’t hurt

By running it in an ephemeral container with no network access for each file you want to check?

Thanks for the insights! I had a look at ClamAV - never heard/worked with it. This was something I was looking for.

Do you also handle large files e.g. videos with lambda? Don't you struggle with Lambda limits then?

Great question. We use spot EC2 instances for ClamAV scans. For thumbnails, we still use Lambdas but use AWS Media Convert for Video files to select a frame for the thumbnail.

It's been a while but I would benchmark how ClamAV fares. At $(day job) we implemented scanning for file uploads and after our security team tested they noticed that it detected things fairly poorly (60% was on the high side) for very well known malicious content iirc.

We ultimately scrapped it, if anyone else has any better experience I'd love to hear it.

I am not a proper developer so I don't have the technical details on this, but another thing I have seen done is to buy access to an API on VirusTotal [1] and submit file hashes to be compared against their database of about 70 or so virus scanners. Not foolproof, as malware can be recompiled and hidden from VT until the actual file is submitted for sandbox analysis which you probably dont want to do with potentially sensitive data so it really only catches known-knowns but apparently some companies find it useful. I utilize it on a windows machine using ProcessExplorer that can automatically submit hashes but the API volume is limited for free / anonymous access.

[1] - https://docs.virustotal.com/reference/overview

If I was a malware distributor, I'd just throw a couple random bytes at the end each time I upload a malicious file, it'd still work and have a different checksum/hash.

Antivirus are really stupid tools, but not that stupid. I said that from a time where I had to work around tools flagged by antivirus. Among stupid things they do are flagging a part of an executable, some nsis plugin flagged the whole package as virus as soon as you included them. I think they probably hash files by chunks, if you have too many bad chunks then you are a virus. A few bytes at the end doesn't change that.

I'm not sure if it's still true, but it used to be that ~half of all antivirus would flag an executable if you compressed it with UPX.

You could sometimes bypass this by opening the file with a hexeditor and change a meaningless value. When UPX was popular there were also alternative file compressors that could also be used to sometimes bypass this issue.

CISA is offering a malware analysis service now, requires a Login.gov account to use and the user be a US citizen. Unsure how it compares to VirusTotal, it’s in our backlog to spike a poc in our IR workflow.

https://www.cisa.gov/resources-tools/services/malware-next-g...

I have to say this user agreement sounds like TikTok privacy policy, but my oh my! The government does its job. RESPECT.

If S3 did this, what would that look like?

Reply on news.ycombinator.com