I looked through attacks in my access logs
nishtahir.com
nishtahir.com
I've had several instances of a new server being up on a new IP address for over a week, with only a few random probing hits in access logs, but then, maybe an hour after I got a certificate from Let's Encrypt, it suddenly started getting hundreds of hits just like those listed in the article. After a few hours, it always dies down somewhat.
The take-away is, secure your new stuff as early as possible, ideally even before the service is exposed to the Internet.
I use LE wildcard certs and they're great, you can use them internally.
Wildcard certs are indeed a valuable tool, but there is no free lunch.
The individual services can still have individual non-wildcard internal-only certs signed by an internal CA. These don't need to touch an external CA or appear in CT logs - only the reverse proxy/proxies should ever hit these, and can be configured to trust the internal CA (only) explicitly.
Renewing a wildcard is also unfun when you have services which require a manual import.
More restrictive is better than less restrictive when it comes to certificates.
"I've had several instances of a new server being up on a new IP address for over a week, with only a few random probing hits in access logs, but then, maybe an hour after I got a certificate from Let's Encrypt, it suddenly started getting hundreds of hits"
Cloudflare’s Merkle Town[0] is useful for getting overviews, but I haven’t found an easy way to query CT logs. ct-woodpecker[1] seems promising, too
It uses an API provided by https://crt.sh/
Honestly it feels like you'll need at least something like basicauth in front of your stuff from the first minutes it's publicly exposed. Well, either that, or run on your own CA and use self-signed certs (with mTLS) before switching over.
For example, when some software still has initial install/setup screens where you create the admin user, connect to the DB and so on, as opposed to specifying everything initially in the environment variables, config files, or other more specialized secret management solutions.
Just SSH scanning can be a big issue.
You don't need to freak out if you see a bunch of failed ssh auth attempts in your logs. Just turn off password based authentication and rest easy.
And if I'm really, say, in China or Russia an really need to access one of my servers through SSH, I can use a jump host in one of the three countries that I allow.
So effectively: DROPping traffic from 98% of the planet.
Boom.
You want to keep these things behind multiple locked doors, not just one.
For the servers themselves, you shouldn't be able to get to sshd unless you're coming from one of the approved bastion servers.
You shouldn't be able to get to one of the approved bastion servers unless you're coming from one of the approved trusted sources, on the approved user access list, and using your short-lived sshd certificate that was signed through the use of a hardware key.
And all those approved sources should be managed by your corporate IT department, and appropriately locked down by the corporate MDM process.
And you might want to think about whether you should also be required to be on the corporate VPN. Or, to be using comparable technologies to access those approved sources.
What if there's a zero-day in your bastion service, whatever that is?
Few days after they asked to redirect an entire subnet to their rack.
And yes, you still need to remember to close password logins or at least pick serious password if you need them. Helps to have no root login over SSH and normal users that aren't defaults for some distro...
There are plenty of organizations and individuals that can competently run ssh directly on the internet (on port 22) with zero risk to "ssh scanners."
I am just pointing out (also in few other, off-site discussions), that one should not even think of exposing a port before finishing locking it down.
Because sometimes people forget, even experienced people (including myself), and sometimes that's enough (I think someone few weeks ago submitted a story which involved getting pwned through accidentally exposed postgres?).
And there's enough people who get it wrong for various reasons that lowest of low script kiddies can profit buying ready-made extortion kits on chinese forums, getting a single VM with windows to run them, and extort money from gambling/gameserver sites. Not to mention all the fun stuff if you search for open VNC/RDP.
And those people told him the install system didn't support SSH keys (hindsight: they did) and got him to make the root logins possible with passwords. Passwords that weren't particularly hard to guess because their only expected and planned use was for the other team to login for the first time and set their own, before the machines were to be exposed to internet, using BMC.
Not everyone is following that advice. Just last week I taught a friend about using tmux for long-running sessions on their lab's GPU server, and during the conversation it transpired that everyone was always sshing in using the root password. Of course plugging that hole will require everyone from the CTO downward to learn about SSH keys and start using them, so I doubt anything will change without a serious incident.
I had a similar issue 15 years ago, and tied the Linux boxes into Active Directory and authenticated via Kerberos. Worked nice, no SSH keys needed!
I wish. I use basicauth to protect all my personal servers, the problem is Safari doesn't appear to store the password! I always have to re-authenticate when I open the page. Sometimes even three seconds later.
I use multiple browsers on my MacBook. Eg I use Google Chrome to run Google Meet, even though I don't use it otherwise.
A VPN lets me access my stuff from my phone while out of the house, for example.
Original article which doesn't contain the first graphic: https://www.xmodulo.com/access-linux-server-behind-nat-rever...
Tunneling through SSH is significantly worse because you encapsulate a TCP connection inside a TCP connection and it's userspace.
The reason is privacy. I use VPN to obfuscate my IP which means I would have to VPN my entire network. Unfortunately this has proven surprisingly difficult to do properly, meaning with appropriate performance (MTU), IPv6, no blocking (exit IP reputation), etc.
Hence I switched to Argo/cloudflare tunnels for pretty much everything.
I had planned to make the move over the next day, but I moved a single service over to make sure everything was working. Next day as I'm testing moving traffic over, I find that I've been rate limited by Lets Encrypt for a week. I check the database and I had provisioned dozens of certificates for vpn.api-tools.mycompany.com, phpmyadmin.api-tools.mycompany.com, down the list of anything you can think of.
There was no security issue, but it was very annoying that I had to delay the rollout by a week and add a whitelist feature.
What? Ideally..before? Seriously? It is 2024.. and this was true even decades ago, absolutely mandatory.
(Still remembering that dev that discovered file sharing in his exposed mongo instance (yes, that!! :D) without password only hours after putting it up.. "but how could they know the host it is secret!!" :D ).
A good starting point for hardening your servers is CIS Hardening Guides and the relevant scripts.
Once you figured out what the domain is, you can easily build a list of IPs out of the cert transparency log and if there is ever an exploit for this specific type of appliances, attackers now have a bespoke list of IPs to hack, a dream come true.
I don't see a solution for this particular use case, I would argue self signed certs would be more secure in this case.
Eventually I stopped proactively reviewing logs and stopped paying for the IDS. It was a waste of time and a distraction.
It's not hard to find really useful content that summarizes common vulnerabilities and attacks, and just use that to guide your server management. There are a ton of best practices guides for any common web server technology. Just executing these best practices to 100% will put you way ahead of almost all attackers.
And then the next best use of your time and resources is to prioritize the fastest possible patching cadence, since the vast majority of attacks target disclosed vulnerabilities.
Where logs are super helpful is in diagnosing problems after they happen. We used log analysis software to store and search logs and this was helpful 2-3 times to help find (and therefore address) the root cause of attacks that succeeded. (In every case it turned out to be a known vulnerability that we had been too slow to patch.)
Just curious, do you leverage any tools to decide when to patch or is it time-interval based? We currently attempt[0] to update our packages quarterly but it would be nice to have a tool alert us of known vulnerabilities so we can take action on them immediately.
[0] "Attempt" meaning we can't always upgrade immediately if the latest version contains difficult-to-implement breaking changes or if it's a X.0.0 release that we don't yet trust
https://github.com/anchore/grype/
https://github.com/eliasgranderubio/dagda/
https://github.com/aquasecurity/trivy
So then you set up something like a cron job to scan everything for you and email the results once a week or whatever if you don't want to monitor things actively.
Integrate security code scanning tools into your CI/CD process. Tools like Dry Run Security, or something comparable.
There's much more, but that has to do with how to run your CI/CD systems and how to do your deployments in general, and less to do with security aspects thereof.
(not a fan of the effort Greenbone have gone to for hiding their community edition and promoting their commercial products)
[0]: https://greenbone.github.io/docs/latest/22.4/container/index...
Yes. This is why relying on "patching" is a bound to fail at some point. Maybe it's a 0-day, or maybe the attackers are just quicker.
The solution to this is defence in depth, and it's very easy for most services, especially when self-hosting personal things. Few tips most people can do is.
Put up a firewall in front or put it behind VPN/tailscale.
Hide it in a subfolder. The automated attacks will go for /phpmyadmin/ , putting it in /mawer/phpmyadmin/ means 99.9% of the attackers won't find it. (This is sometimes called security by obscurity and people correctly say you should not rely on it, but as a additional layer it's very useful).
Sandbox the app, and isolate the server. If the attackers get in, make it hard or impossible for them to get anywhere else.
Keep logs, they allow you to check if and how you got attacked, if the attack succeeded and so on.
Depending on the service, pick one or more of these. Add more as necessary.
The key thing is that you should not rely on any ONE defence, be it keeping it patched or firewalled, because they will all fail at some point.
I detest this phrase right up there with "fake it till you make it". All security is by definition obscurity. Just a meaningless platitude that rhymes.
I suppose un-formalized un-proofed security practices will eventually be broken or counting on hackers not to do any investigation of your system will get you hacked, doesn't roll off the tongue though.
Can you expound on this? For example, if I add a 30 second lockout after failed authentication attempts I don’t see how that comes under any non-tortured definition of “obscurity”.
A hammer is not construction, it's a tool used by construction?
I really feel dirty trying to justify this level of nerdy pedanticness. I'm sure you can poke some holes from my off the hip internet comment if you really want to I'm not trying to be academic. I mostly fueled this comment with my distaste for that other platitude.
That being said, obscuring parts of an otherwise secure system is fine as an additional layer, especially if you just want to deter script kiddies that always hammer the same endpoints
Anyone who thinks “security by obscurity” is useless should try reverse engineering some properly obfuscated executables (or even code). Obscurity is absolutely useful; definitely not a complete solution by itself, but a very useful component to a security solution.
Obscurity is a useful tool to be added on top of real security, and can help reduce the random baseline doorknob jiggling attacks, where people are just scanning the standard ports. But obscurity by itself is not enough to provide any real security beyond that.
Level 1 : I don't know what I'm doing, so I'll invent stuff only I know in the expectation that'll be enough. This person is told (with good reason) that security by obscurity is no security at all.
Level 2. They got the above message, so do everything right. Setups, firewalls, permissions, and so on. They are proud of their expertise and lecture level 1s all day long.
Level 3. Understand that all the fundamentals need to be done right. Add addional obscurity onto that because it doesn't hurt, and can filter out some useless traffic. (These folk should also lecture level 1 with the simplified message, but can explain the benefits to level 2.)
The problem with HN threads like these is that I don't know who's giving the advice. Level 1 2 or 3. Equally readers could be any level. Which might be dangerous if they are level 1.
IF you ARE level 1, learn the correct way to secure things first. THEN feel free to add obscurity onto that if you like.
Why can't we just outright say to everyone: "learn the correct way to secure things first. THEN feel free to add obscurity onto that if you like"? There's probably some way to convert that message into a catchy phrase, instead of just demonizing security by obscurity, yet having a small sect of elites in the know who break their own "rules"...
FWIW, this (rant) applies to a broader scope as well, beyond "security by obscurity".
(For example, we teach kids the earth is round, then we tell them it's an oblate spheroid, then we tell them even that is a approximation. )
In a forum like this, the message is often distilled because one doesn't know the makeup of the audience. The simple rule is a good starting point for entry-levels.
Nuance can be hard to convey, because its usually a combination of context and resources which determine how far down a rabbit hole you can go.
I agree that this happens all the time, in every field. I'm dumb when it comes to my car, so my mechanic gives me simple rules to follow. Doctors tell everyone to eat less, exercise more, but in truth you can eat too little, and exercise too much.
The -real- problem happens even a simplification becomes obsolete. We see this in security a lot. Someone wrote the company policies 20 years ago, and they're insisting on say no-paste-on-the-password field, because that's in the policy, even though its since been shown to weaken security.
What layer of security is sufficient on its own? None. Why is "obscurity" being singled out?
It always feels a little condescendant, like "oh I got a cool saying that applies to most noobs in security, let's use it", implying one is assuming the other party in the conversation knows less.
If you can come up with some good examples, maybe we can make that a thing.
Until then, we know that "Security by obscurity" is really bad, if that's the only thing you're relying on for security.
Otherwise, that would be "Security by something else plus obscurity", which might or might not be a bad thing, depending on what the "something else" happens to be.
As an American living abroad, it's not uncommon for me to not be able to access a website I need (or want) to, and I need to pop onto my VPN.
I imagine blocking North Korea, Iran, etc will probably impact 0.5% of traffic and 0.001% of revenue for most sites.
> and push determined adversaries to use proxies?
That might mean you're left with 10% of the previous attackers so it could be worth it.
It's just so unbelievably common and so frequently harmless that it's hard to take all that seriously. But you're right, it is a threat, I won't deny that.
A friend told me that someone in his very big company sold some random stuff to North Korea and now 1) they have obligatory training not to sell there 2) they have to go through obligatory training on non proliferation of nuclear weapons
Most of probing to my server comes from USA, and a far second place is Netherlands. Probably most of it comes from AWS and other data centers.
Move DNS to Cloudflare and put a few WAF rules on your site (managed challenge if bot score less than 2 / attack score == x). I doubt you'll even pay anything, and it will resolve a lot of your problems. Just test it before moving it to production please (maybe setup a test domain). Remember, a WAF is not an end-all be all, it's more of a band-aid. If you app isn't hardened to handle attacks, no amount of advanced WAF/bot protection will save it.
Message/email me if you need help.
But since you seem to have a lot of knowledge in this area. Have you manage solutions which also includes infrastructure in Azure combined with Cloudflare?
And if so, any suggestions on things people usually miss? except for the usual stuff of OWASP and what not
The Free Managed Ruleset appears to be deployed by default, and Cloudflare keeps a changelog here: https://developers.cloudflare.com/waf/change-log
It’s almost as if those saying contradictory things are actually different people despite being on the same website. But it can’t be that, surely? Truly a perplexing phenomenon that I hope someone can one day explain.
I suspect it's because hating on Google is in vogue, and so is recommending Cloudflare.
Probably not as cheap. AWS can put a WAF and CDN infront of your site too.
And migrating from one service to another isn't much more work than moving DNS records.
Just saying, it's not the same level of vendor lockin as using dynamodb or whatever.
I use Cloudflare (free tier) in front of the very few and almost entirely unused websites that I run. I believe that the service they provide is useful for protecting the IP addresses of the servers on which the content is hosted, whilst also providing some amount of protection from malicious traffic.
I also agree that centralisation of services is a big problem for the future of the internet.
My position is that, whilst there seem to be increasing voices / examples of Cloudflare's (potential in) acting against the nebulous notion of "spirit of the internet", for me they certainly haven't reached the "evil" stage. I'm also of the understanding that it's Cloudflare customers that choose to block access from Tor or VPS IP address ranges and / or add Captcha's or other bothersome verification. True Cloudflare enable it and make it possible, but the administrators of the website that you're trying to visit have made the choice to make it more difficult for you to access their content; not Cloudflare themselves.
I would prefer there to be similar-scale alternatives to Cloudflare as a kind of a middle-ground decentralisation of centralisation. I'm sure there are alternatives, but I'm not yet motivated enough to even consider starting the research process.
If Cloudflare start selling visitor analytics to data brokers, however, very fast goodbye.
We should be suggesting self hosted and decentralized solutions to website hosting and file hosting.
On that note, does anyone have any secure methods of providing serving a file from your computer to anyone with a phone/computer that doesn't require them downloading/installing something new? Just a password or something? Magic-wormhole almost seems great, but it requires the client to install wormhole (on a computer, not phone), and then type specific commands along with the password.
Is there a simple `iroh serve myfile.file` from server and then client goes to https://some.domain.iroh/a086c07f862bbe839c928fce8749 and types in a password/ticket you give them?
That would be wonderful.
That's just plain bs...
Eg
1) they have customers and their customers want protection, with minimal downsides.
2) Cloudflare is the only one with support for Tor. I'm 100% sure you didn't knew that.
What "examples" do you have to blame them for something they aren't doing? Based on what?
I'm getting tired of people blaming Cloudflare for providing a service that no one else can provide for free to small website owners => DDOS protection.
http://forums.accessroot.com/index.php?showtopic=4361&st=0
>Please wait while your request is being verified...
I can't remember any day I didn't get a Cloudflare block. Even on bare IP sometimes. WAFs are security theater.
Which circumvents the bad reputation of certain exit nodes:
> Due to the behavior of some individuals using the Tor network (spammers, distributors of malware, attackers), the IP addresses of Tor exit nodes may earn a bad reputation, elevating their Cloudflare threat score.
You're of course welcome to make your substantive points thoughtfully while staying within the rules.
If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
I reiterated over my last comments and they've been snarky lately.
Not an excuse, a lot is going on and overworked and without patience lately.
That shouldn't reflect in my comments and I'll pay more attention to it.
Have a good week.
You're welcome to make your substantive points thoughtfully but it needs to be within the rules. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
I can imagine that might be needed if some company for some reason has to run some not really up to date stuff but yeah it is just a bandaid.
Would be cool to do the same thing for the weird stuff I see in /var/log/auth.log!
It's crazy that attackers would bother with me since the code is entirely open source and there is no server-side state. The best outcome for an attacker would be root access on a $5/mo VPS, and perhaps some (temporary) defacement of the domain. A domain no-one visits!
By the way, I find it annoying that my logs get filled with this kind of trash. It has the perverse effect of making me long for something like Google Analytics since they rarely if ever bother running a javascript runtime.
If you put your ssh server or something on an uncommon subdomain how will these scripts find it?
If you are on @ or some common name sure, otherwise no.
I then used ssh to try and connect to the originator (from an external box). I went back 5 jumps until I got to a windows server box on a well known hosting service that I could not get into.
Lots of what looked like print servers and what looked like linux machines connected to devices. Maybe just the exploit at the time.
I also limit concurrent connections, which significantly reduces data usage during aggressive attacks
But in general, I’ve given up on caring about the routine “attacks” listed in all the logs. If you have good security, they don’t matter. And if you don’t, they don’t matter either.
Over the few months I've had it running I've needed to progressively create failsafes for IP addresses that I know are trustworthy so I don't lock myself out. I've also started tiering the importance of blocking based on different sets of ports which are being probed. I've also discovered that there's a significant amount of Uninvited Activity coming from "security" companies in their pro-active scanning of the entire IPv4 space - which I don't trust at all and ban with prejudice.
It's messy and I need to update and add many explanations, but it's on Github if anyone wants a laugh: https://github.com/UninvitedActivity/UninvitedActivity
(I'm aware of various limitations and footguns inherent in this un-subtle approach but, as another commenter elsewhere alluded to, "it makes me feel better". I also think that a fair bit of processing volume can be taken off IDS' if a heap of "known garbage" traffic is blocked prior - it's all about tiers).
Some attacks don't even target the IP and instead monitor domains and its subdomains, and periodically run the same scans from the exact same range of IP addresses.
For example, on a previous job we had a recurring scan made over and over again from a single static IP address located in Turkey that our team started to refer to it as "the Turkish guy", and our incident response started featuring a preliminary step to identify weird request patterns that was basically checking if it was the Turkish guy toying with our services.
It's not expensive and can significantly help you, saving you from many headaches in situations like this. While it might not block everything that arrives at your service, it can be a great help!
There were a few overzealous rules that subtly broke parts of our app.
There were request body rules that did things like block requests that contained "localhost" in the request body. There was also a rule that blocked requests without a User-Agent header, which we were not previously requiring on API requests, so we broke our entire API for a few users until we figured that out.
Complete due diligence is required to fully understand and realise the impact of the rules and should be tested like any software change by going through a testing phase.
Ideally software teams should be fully trained and be responsible for their lifecycle.
Had a fun one after turning on the AWS WAF with some default rules–a small number of users reported they couldn’t upload new logo images anymore. Turned out some Adobe product was adding XML metadata to the image files, which the WAF picked up and blocked.
This is one of the cases where you must understand what you are doing. There's no technique for doing it mindlessnessly.
As you note, it requires adjustment due to overzealous rules. OWASP has Paranoia Levels[1] which allow you to be more targeted.
[0] https://github.com/coreruleset/coreruleset
[1] https://coreruleset.org/20211028/working-with-paranoia-level...
Then you block sooner and get slammed for blocking too soon.
Repeat for 1000 web services.
But doing that would also waste my time.
It’s the same idea used by anti-spam activists back in the day with software that would flood spam website forms with fake but realistic looking info, so the real data would be buried in the noise.
I'm sure they mostly refer to the same thing, though.
--
Be very careful that fail2ban isn't actually exhausting your own resources faster than you believe you're exhausting the attackers' resources...
The attacker even went to the trouble of spoofing some custom headers in our API client.
Our eventual solution was to a) attack our own auth credentials first, identify any users with leaked creds (from other services) and force a password reset for them. b) disallow users from setting common leaked passwords. c) make the auth checking request as low-cost as possible, and scalable separately from the main application. d) when an attack request is detected, bypass the relatively-expensive real cred check but return the same failure response (including timing) as a real failure. e) build a secondary requirement in the auth flow that can be transparently enabled when under high volume attack.
This works, so far. It sheds the volume to the application, and has low-to-zero impact on legit users. This took a couple weeks away from feature development though!
How many active users (to an order of magnitude; no need for precise numbers of course) does this service have? 100k IPs sounds pretty costly to burn, so I'm curious how important one needs to be before that's considered worth it.
And could you say what type of IP addresses those were? Did it look residential such as from compromises computers (botnet), do they rent lots of IP addresses temporarily from aws/netcup/alibaba, or is it a mix such that neither category has the overwhelming majority?
If it's all server IP ranges for a service where end users normally log in, you could apply entirely different rate limits to those than to residential IPs, for example. Hence I'm wondering how these cred stuffing attacks are set up
And of course a ton of open proxies.
The linked write up on why is interesting, finds attacks for many of the same files, and recommends blocking the user agent.
A common mistake is to publish the Docker ports unknowingly to all interfaces (e.g `5432:5432`), which makes your Docker container available to everyone. It is common to see this in Docker tutorials or pre-made Docker Compose files. Coupled with UFW, it may give you a false sense of security because Docker manages its own iptables rules.
They serve different use-cases but I wouldn't say that VPN is strictly better than HTTP auth or vice versa. Recommending to double up for a self-hosted little something, not a big target like 4chan or Gmail, is overkill
Instead of exposing your applications externally, you create a private network that uses UDP hole punching.
This isn't completely self-hosted, as you need some server to auth / broadcast connection details with. Self-hosting might be possible on ZeroTier, but I'm not familiar enough to say for sure.
Really the only good answer is defense in depth and keep looking for any indicators of odd behavior, and wall out unrelated systems entirely from each other, keep the DMZ and public facing bits as simple as possible.
Cloudflare Tunnels are great, but CF doesn’t allow the TLS pass through. CF man in the middles and decrypts the traffic.
So far I have looked into Teleport, authelia, authentik, and keycloak, perhaps combined with Traefik.
Any feedback on the level of security of these tools for being exposed to the public internet?
https://learn.microsoft.com/en-us/entra/identity/app-proxy/a...
https://learn.microsoft.com/en-us/entra/identity/app-proxy/a...
A version with end to end encryption will be great!
Aah. I diverted my domain to cloudflare, allowing only traffic from 1 country in.
So instead of exposing my IP publicity through my registrar, I set the firewall to only allow traffic in from cloudflare.
I would have loved to install PfSense as well but it was out of my budget
The correct term is directory enumeration. Traversal usually means something about ../../
I somehow always imagine these types of hacks to be more clever, like, I dunno, sending harmless-looking stuff that causes the program receiving it to crash and send some instructions into unprotected parts of RAM or whatever. This all looks like "echo ; /bin/cat /etc/passwd" and somehow the server just spitting it out. Is that really the state of web security?
Someone might be trying to play with self hosting or a contractor at a company did a bad job and accidentally exposed stuff they shouldn’t.
This attacker is likely just trolling lots of IPs hoping for low hanging fruit that can be exploited with simple/well known attacks.
This is basically how things work.
For convenience, instead of itemizing each filename, the webserver root is a subdirectory and anything underneath is fair game. The webserver uses the OS "chroot" facility to enforce this restriction. What you are seeing is ancient exploitation strings from 30 years ago that haven't worked on any serious webserver since that time, but a) keeping the test in the attackers lib is essentially free, and b) there are some unserious webservers, typically in cheap consumer hardware.
Webservers pass plain text to the app server. It is the app server/framework's responsibility to understand the source of the request body and present it to the application in a clear way, possibly escaped. But the app needs to process this and sometimes through poor coding practices, fails to respect the untrusted nature of the data. This again is more typical in historical systems and low-cost consumer products where software is not a marketing advantage.
Unfortunately, there are plenty of serious (business critical) servers that _ARE_ vulnerable to these types of attacks. I've found and remediated things like this all the time. One very common example I've seen of the `.env` issue is Django servers that are exposed to the internet in with debug=True. There's probably thousands if not tens of thousands of servers leaking credentials this way on the internet now.
Beyond that, companies often have internal systems that do not meet the same security standards that external systems require, and sometimes those systems get shifted around, maybe it's moved to a new subnet, maybe a third-party needs access and the CIDR range gets fat fingered in the firewall. Regardless - now that "internal system" is exposed to the internet with all the dangerous configuration.
It's attempting to exploit a vulnerability in bash that was discovered and fixed in 2014:
There are different types of web security vulnerabilities and the attacks you see from automated scanners are likely to be far less sophisticated than targeted web attacks. Specifically these scanners are going to spam out widespread and common CVE's that might grant privileged access to the server or dump credentials in some fashion.
The more sophisticated attack you described is essentially an overflow, and most modern web servers are usually written in memory-safe languages making it very unlikely to see that type of attack on the web. More often it's the underlying OS, servers, or communication stacks (bluetooth, TCP, nginx, etc) that have these types of vulnerabilities since they are often written in low level non memory safe languages like C and C++.
Attacks that exploit the HTTP and HTTPS protocol are a little more interesting. Request smuggling lets you trick certain load balancers and webservers by sending an HTTP request "smuggled" inside of another HTTP request.
Here is a blog by James Kettle's about some request smuggling vulnerabilities and the impact they can have. https://portswigger.net/research/http2
There's really a lifetime's worth of knowledge on web security and the type of stuff you see in scans is just trying to hit the low hanging fruit. Portswigger has loads of free challenges and information about different web security topics.
* Comparison of AWS WAF vs Cloudflare vs Others?
* Many services like EC2 charge for data transfer [1], so how much of your monthly/yearly costs of hosting goes toward fending scans like these? Does AWS count any traffic blocked by the WAF toward the transfer limits?
--
Human security - PerimeterX, Haproxy WAF, Datadome, are other players to different target audience.
If you have a good control over the app exposed, maybe you only need a WAF in the sense of stop stuff outside your infra. The sql injections, weird urls trying the way to /etc/passwd or related things look from the past and only makes noise nowadays. The real issue is when someone hits you in a rate impossible to manage with your resources or when it cost you more than the securing layer.
The problem is you have to manually configure a lot, the rate limiting aspect is way worse than Cloudflare and while the AWS WAF can geolocate an IP address and block by country it does not send the country code back to you in a header where as Cloudflare does. The last one stings because it's super handy to have an accurate country code attached to each request, especially if it's something you don't need to think about or waste I/O time calling out to a 3rd party service in the background to backfill in that data later.
--
1: https://github.com/awslabs/aws-solutions-constructs/tree/mai...
2: https://constructs.dev/search?q=waf&cdk=aws-cdk&cdkver=2&lan...
- Disable password login for SSH, use keys instead.
- Limit access to known IPs (with a managed vpn)
- Use Cloudflare: Their WAF is really good
- Forward logs to an other service that can analysis logs (datadog is nice)
shameless plug: started a small honeypot service[1] if anyone would need it as a last resort[1] to catch hackers in your servers . Feedbacks appreciated!
% ssh example.com
Last failed login: Sun Jan 28 16:59:35 UTC 2024 from 180.101.88.233 on ssh:notty
There were 5385 failed login attempts since the last successful login.
Last login: Sat Jan 27 13:33:30 2024 from xxx.xxx.xxx.xxx
Ideally you don't need ssh open to the whole world anyway, and can restrict it to a certain subnet or set of addresses. Then your attacks will drop to nearly zero.
No, that does nothing if you followed best practices and disabled password login. fail2ban is just another Denial of Service risk that has the added bonus of bloating your firewall table and slowing down all your new connections
table inet filter {
chain input {
tcp dport ssh accept comment "Accept SSH"
tcp dport 22 ct state new limit rate over 2/minute drop
Also I normally set up a jump host first - a smallest instance that only runs ssh and everything else would not open ssh port to the outside at all. One nice effect is having to search just one auth log if something about ssh looks concerning.
https://cloudonaut.io/connect-to-your-ec2-instance-using-ssh...
Perhaps it might encourage people to keep their systems patched if the outcome of not doing so was being Ddos’d!
I wonder if that’s more or less of a “safe” situation.
Btw how do you make sure you are not hacked or that some frivolous attacker didn't accessed something valuable?
For example, they did an emergency update on 1/22 in response to CVE-2023-22527.
It feels to me like we're too polite, so we're letting the infected walk amongst us. It might not be their fault, but I'd be guessing it'd also be better for victim to find out sooner rather than later if they're pwned.
I suppose the slippery slope / end game of this would then concentrate all intentional malicious traffic to VPNs and proxy's and Tor and the like, with them becoming useless due to being blocked.
I block any IP address that's probed any of my ports they have no business probing. See https://news.ycombinator.com/item?id=39171782