Web servers should refuse requests for random, unnecessary URLs

The findings are (nearly) all meritless, but it creates a lot of annoying work if a higher-up in the company (who doesn't know that it's a shakedown) says that the report needs to be addressed.

the rub there, though, is that validation is extremely rigid and heuristics aren't. an attacker could (ideally) change a single bit in their signature to falsely pass validation but a heuristic will probably still catch it.

Okay, here it comes. Heuristics are essentially pattern-matching, because (as sibling points out) rigorous validation is extremely expensive. That's either in time, compute cycles, or sheer complexity. Really good - and smart - heuristics are fairly close to good validation, but in the security world, false positives and cutting corners are the norm.

Vulnerability scanners of all sorts take massive short cuts. Everyone does banner grabbing, but very few try to validate that the announced version actually matches the known behaviour of that version. All kinds of scanners and end point agents collect the detected versions of software present on the system, but then don't check whether a supposedly detected vulnerability is actually present. An entire category of stupid agents broke when the point-release number for distro kernels went to four digits, and their boneheaded logic either treated the last figure as zero (.0) or truncated it to three digits. False positives everywhere, with "detected" kernel vulnerabilities all of a sudden sprouting across the fleets. Not a single one of these agents actually tried to validate the vulnerabilities they claimed to have spotted.

(Banner grabbing works both ways: my IRC client claims to run on C-64.)

Network vulnerability scanners are not any better. A particularly advanced scanner may look at the returned Content-Type header to spot crawled endpoints with interesting payload potential .. and then doesn't bother to use magic(5) to check whether the served content matches. Another network scanner detects that a host is running an LTS Debian derivative, but raises a high-severity finding because it spots network facing software using OpenSSL 1.1.1, which is getting EOL'd. The fact that the vendor provides continuing support is known and registered in various places across detection and alerting logic, but not used in this case. The heuristic used is "is this software EOL?", when the correct validation would be "is this software supported, up to date, and still receiving security fixes?".

I could go on and on, but I think I've made my point. Even the best-of-breed[tm] heuristics as used by security software industry are a scourge and the root cause of frustration of everyone even remotely competent.

But since you asked... malware is a special case. Heuristics in malware detection are a crutch, but they are nonetheless the least bad option. That's because with malware, you run into the halting problem. The only way to actually validate that a piece of suspected malware is indeed malware, is to execute it, and closely observe what it does. But there is no safe and easy way to do that. The universe of malware is highly adversarial, with counter-counter-counter-counter technologies present at every level by all sides. The best technologies use sandboxing to explode suspected payloads, but that only opens up a new vector - now the vendor has to worry about sandbox escapes too. And anti-malware agents themselves are running extremely complex operations at elevated permission levels, trying to hook or otherwise hijack software operations at the lowest possible OS levels, to prevent them from doing what the agent deems objectionable. We used to have a different name for this type of software back in the 90's. We called them rootkits.

These days the industry calls them EDR agents. And I simply call them increased attack surface.

The correct response code would be a 3XX or 4XX response code, depending on the client behavior that you want.

In this specific case, a 303 response that redirects to the homepage would be most correct.

Alternatively, you could return a 404 response alongside the default content.

It’s like saying “what makes it wrong to signal left but turn right?”

It makes the component you’re operating behave less predictably, resulting in a less stable system overall. Follow the spec.

The relevant part of the spec says that returning a 4XX code is a SHOULD [0] not a MUST. That means:

> This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course [1]

[0] https://www.rfc-editor.org/rfc/rfc9110.html#name-client-erro...

[1] https://www.rfc-editor.org/rfc/rfc2119

So yes, there is no requirement to return a 4XX message, but you should think carefully and understand the consequences of not doing so.

One consequence is creating false positives in most vulnerability scanners.

Another consequence is potentially messing up various crawlers, including those that help search engines index your content.

    location /.well-known/resource-that-should-not-exist-whose-status-code-should-not-be-200 {
        return 404;
    }

I have to admit, I'm actually not sure what a browser does if a server is sending normal content but with 404 status codes. At the very least, I guess it messes up the caching?

Too often what happens is that the receiver is liberal, which results in senders not having to bother to fix violations of whatever spec the parties purport to be following.

As the sending code continues to be developed occasionally new deviations from the actual spec and from whatever it is that current liberal receivers accept creep in, and the receivers get even more liberal to deal with them.

A few years of this and writing a receiver for what should have been a fairly simple and easy to implement format requires huge parsers that handle a bazillion weird cases.

Systems that are liberal in what they accept are paradoxically harder to develop against, undermine standards and encourage incorrect, fragile implementations. Query parameters are a classic. In practice, unrecognized query parameters almost always represent a bug in the client. Better to find out immediately with a 400.