Someone has been attempting to DDoS us for weeks and we do nothing
tableplus.com
tableplus.com
For example, there's no reasonable world where that 200 MB blob is not cached and served over CDN. I can't imagine someone would be so proud that their application server isn't reading 200 MB from disk and copying those bytes to the client on every download; it's just so obviously poor design.
The files are indeed cached by CF:
# curl -v https://files.tableplus.com/windows/5.9.2/TablePlusSetup.exec > /dev/null
...
< cache-control: max-age=691200
< cf-cache-status: HIT
< age: 2980Or is that only with the free tier?
It’s encouraging or reminding people that this style of architecture, which was once prevalent, is still an option and is still rather legit.
Or, that’s how I read it however I also am biased as I prefer this method of development too. I think what’s missing in the article is, who are they and who is the audience. As in, some acknowledge that most things are not ever going to see mass scale usage and don’t need to be developed to the specs of a faang
I do get that the volume of data and requests they are handling hardly constitutes the claim of ddos.
Using a CDN has been common practice for many years at companies big and small. Many big CDNs even have free tiers.
> Like how can claim it’s self congratulatory when huge companies regularly can’t do this?
You don’t need a FAANG company with thousands of developers to use a CDN. You can sign up for Cloudflare today and have your files distributed by CDN with some simple changes. You could have it done by end of day.
That sounds crazy, right? But yet that's where we are.
Context: I used to manage the DDoS protection at Cloudflare, and also the WAF, Firewall and some customer facing security projects, and we frequently either saw that web scraping took customers offline, or trivial and very low volume HTTP request rates took customers offline. In the early days we considered anything to be a DoS when it threatened our infra, but the threshold for customers is barely higher than a few concurrent and active users.
The big numbers always make headlines, but it's the small numbers which most people feel.
My favourite WP hosters:
- kinsta.com (for scaling & multi-sites)
- raidboxes.io (amazing customer support, usually within 15min, even on a Sunday)
Yes, the instance had docker and was in an auto scaling group to be rebuilt if anything fails. There were 3 containers running with strict mem/cpu limits. Nginx reverse proxy (all in 128mb of ram), a mariadb sql server with a minimum of ~300mb of ram and up to 512mb if available and the php/Web host with 512mb ram reserved. Mariadb was tuned and shopware was tweaked, but that's about it. Everything run fine on a 2 core, 1gb ram instance. (has, not "has been" because a year later the shop closed for other reasons). So the morale of this story is, sometimes a $5 vps or an instance is the correct answer.
Drupal, in particular, is notorious for having multiple layers of cacheing out of the box. Of course, you can always add some extra caches...
No part of a DDOS requires the throughput to be gigantic, although the big ones are typically the ones you will find in the news.
One possible aim of this attack is to either burn through the bandwidth quotum of the source servers, or to use so much bandwidth that it becomes unaffordable. This could be done very cheaply with just a single or few attacking machines. Most datacenters and hosting providers have bandwidth limits or start charging after a certain amount, and too often the company being attacked only finds out when they receive a bill they can't afford.
This is no "DDOS", it's just misconfigured bots/crawlers going through all the links and being unable to cancel downloads.
For a company, this should definitely not be something to worry about. However, if I were able to single out individual IPs that are attacking me, then I would simply block them, report them (use the abuse form from the hoster of the attacking IP), and call it a day. This way, you can at least hope that the hoster will do something about it, either by kicking the hacker off its platform or, if it is some kind of service reflection attack, inform the victim to close the security loophole on their server and remove themselves from the botnet. If your attacks originate from a vast amount of different IPs from Russia and China, consider geoblocking.
Cgnat is becoming common on home internet. You can share an IP with up to 128 other people.
If I were to be hit by such an "attack" myself I probably wouldn't even notice until Cloudflare sends me that monthly "X TB of data transferred, something close to 100% bandwidth saved" email.
I like the app btw, can recommend.
To the contrary, I wish more people do this: The more people know that their overly complex infra is sub-optimal the better!
As long as such abuse doesn't cause monetary or resource exhaustion concerns, it's quite OK to ignore it, but stories like "whelp, turns out that 80% of the capacity of our auto-scaling fleet is not doing anything useful" are depressingly common enough to at least keep an eye on things.
My annoyance with this kind of abuse revolves mostly around logging: a majority of logs just showing the same set of hosts displaying the same pointless behavior over and over again. Again, not a huge issue if your log storage is cheap and plentiful (as it should be), but having some kind of way to automatically classify certain traffic as abusive and suppress routine handling of that is definitely a good idea.
It's also a lot harder than it sounds! I can't count the number of times I've added classification logic to my inbound SMTP server that should pick up on outright, silly abuse (of which there is a lot when dealing with email), only to have it triggered by some borderline-valid scenario as well.
Spending way too much time on going down successive rabbit holes is a great way not to get any real work done -- a great reason to outsource, or, if that's too much work as well or just too expensive, indeed just ignore the abuse, annoying though it is...
8TB egress on AWS is $595 (taking into account 1TB free egress/mo), while 8TB egress on Hetzner starts at less than $10/mo. With DigitalOcean you'd pay $30 overage for 3TB on top of 5TB included in the s-4vcpu-8gb-intel, for example. 3TB overage is $15 with Linode. I think the article has a point.
But, sure, if you have public-facing services on AWS that have the ability to send large amounts of data on demand, absolutely make sure that you limit access to those! (E.g. using a unique download token that is only available from a separate rate-limited and valid-source-checking service).
Most of the cloud architecture posts on HN either focus on how k8s/%your favourite new tool% is good for scale or detrimental to keeping complexity under control. And I think it's valuable for startups to consider cloud billing abuse attacks in addition to horizontal scaling concerns and complexity, which is what I referred to when I said the article has a point. As you wrote, rate limiting and extra checks could get the job done in a scalable deployment, so there is more than one way to keep cloud bill from an attack.
Using that as a reason to assign me responsibility for the state of the internet seems... slight hyperbole?
Maybe I am overly paranoid, but seems that old russian maxima is reasonable: Fight when they coming for cent, because when they will come to take dollar it will be too late.
Sometime around the dawn of this millennium, for example, 'consumer firewalls' at just about every OSI layer became a thing, and a lot of these had the great feature where they would automatically email WHOIS contacts for domains and IP blocks (plus all of their upstreams, for good measure) every time something bad happened, like receiving a single UDP packet on port 139.
Stuff like that, predictably, put a bit of a dent in the availability of useful technical contact information, and as much as I would like to go back to the "I have the beeper number of the guy who runs the national backbone" Internet, I'm afraid that Cloudflare is the best we can do these days, sorry.
Back to the topic at hand: "fight" on the 2024 Internet means refusing service to abusive parties as much as possible. That responsibility is best outsourced (see 'Cloudflare' above...), and a hard undertaking if you want to do it yourself without causing collateral damage (which, yes, Cloudflare also does, but at least you get someone to point at!).
Expecting to somehow get in touch (or worse, 'get even') with the myriad of bulletproof hosters (who simply don't care), admins of bug-ridden/misconfigured systems (who often don't even understand the issue) and assorted detritus is unproductive. And, as with any "the ideal amount of fraud is nonzero" discussion, that can be a hard pill to swallow, but a necessary one nonetheless.
Their advice isn't bad per se, but their numbers are not a testament to it. I expect for my Go HTTP API services to handle 5k requests per second on a small to medium VPS when there is some DB activity and some JSON formatting without doing any optimizations. This is based on deploying dozens of similar services while working at a place that got multiple billions of requests per day, spiking to over 500k rps.
They’re advocating deploying a binary as preferable to using docker, fair enough, but what about the host running the binary? One of the reasons for using containers is to wrap your security hardening into your deployment so that anytime you do need to scale out you have confidence your security settings are identical across nodes.
On that, the monolith talked about here can be hosted on a single VPS, again that’s great (and cheap!), but if it crashes or the hardware fails for any reason that’s potentially substantial downtime.
The other worry I’d have is that tying everything into the monolith means losing any defence in depth in the application stack - if someone does breach your app through the frontend then they’ll be able to get right through to the backend data-store. This is one of the main reasons people put their data store behind an internal web service (so that you can security group it off in a private network away from the front-end to limit the attack surface to actions they would only have been able to perform through a web browser anyway).
There is no universe in which _increasing your attack surface_ increases your security.
At that point, why are we making a distinction when we do run 1 app on one VM? Sure, containers have some overhead, but not enough for it to be a major concern for most apps, especially if you need more than 1 VM for the app anyway (horizontal scaling). The major attack vector added by containers is the possibility of container breakout, which is very real. But if you run that 1 app outside the container on that host, they don't have to break out of the container when they get RCE.
If you’re using a typical docker host, say CoreOS, following a standard production setup, then running your app as a container on top of that (using an already hardened container that’s been audited), that whole stack has gone through a lot more review than your own custom-configured VPS. It also has several layers between the application and the host that would confine the application.
Docker would increase the attack surface, but a self-configured VPS would likely open a whole lot more windows and backdoors just by not being audited/reviewed.
I have a FreeBSD server, three open ports: SSH with cert-login only, and http/https that go to nginx. No extra ports or pages for potentially vulnerable config tools.
I guess no one knows how to harden an OS anymore so we just put everything in a container someone else made and hope for the best.
Are you suggesting that not opening the ports to any other services means they’re no longer a vulnerability concern?
That would be.. concerning.
This is false. Or so you think your host is secured by installing Docker? And when you scale, how do you get additional hosts configured?
True is, when you use Docker you need to not only ensure that your containers are secure, but also your host (the services running your containers). And when you scale up, and you need to deploy additional hosts, they need to be just as secure.
And if you're using infrastructure as code and configuration as code, it does not matter if you are deploying a binary after configuring your system, or Docker.
There are tools that make "bare metal" configuration reproducible (to varying degrees), e.g. NixOS, Ansible, building Amazon AMI images.
The important thing is making walls indestructible, not making more walls. Interfaces decrease performance and increase complexity
(Some of) the reasons why you would do this are explained (I thought clearly) above. None of this is security through obscurity.
A single core, actually, after JVM JIT kicked in.
60 seconds per minute 60 minutes per hour 24 hours per day 30 days per month ~2.59 million seconds per month
One billion requests into 2.59 million seconds is 386 requests per second.
“… => Thus, we build a monolith service for each app, which is easy to deploy and maintain. No Docker, no Kubernetes, no dependencies, no runtime environment - just a binary file that can be deployed on any newly created VPS. …”
It's fine, really. Those database and logging services you can put in a docker if you like, but if you put them anywhere else it works just the same. A Postgres in k8s or a Postgres on a dedicated server is the same as far as the client is concerned.
I'm mostly not following what is under a DDoS attack. Is it their web page mostly consisting of marketing material with static pages?
Yes.
Compare this to deploying python, node or php... Needless complexity.
If only running (and keeping running) a database server could be this straightforward!
I tried Vercel pkg, Vercel ncc, nexe, and a few other tools I can’t remember right now. They all had issues with node v20, some dependencies, or seemed to not be maintained anymore. I ended up relying on esbuild as a compromise to get a fat script containing all sources and dependencies, tarballed with some static files we rely upon we can at least get versioned, reproducible runs (modulo the node env). Still not perfect, a single binary would be preferable
Now you can use this native feature (not totally stable yet though) which I’ve been meaning to try https://nodejs.org/api/single-executable-applications.html
And with that still, you’d be much better served by using a more expressive and less painful to use language like C#. Especially if the type of use is personal.
[2] https://peps.python.org/pep-0599/#the-manylinux2014-policy
Depending on the distribution of the traffic they might have survived well on VPS's without Cloudflare anyways, doesn't seem that large. Would be interesting to see more detailed stats of rps and how much (if any) Cloudflare stopped before they got it.
Russian layer7 ddos'es that I know of targeting Swedish companies have been large enough that major providers run into capacity problems and fall over (including Verizon, Azure Frontdoor, Cloudflare, GCP's Load balancer). This strategy would absolutely not work against those volumes.
Also Java jar files give you the same benefit.
You have to explain that one a bit more.
I think the word you're after is "disingenuous"
I've got nodejs lambda code that is doing 388B/month and only at this point have we even considered changing the language for performance because the cost savings have a net positive ROI.
It took 5 years to get to this point.
You can configure the cache so that they can't do it: https://developers.cloudflare.com/cache/troubleshooting/cach...
Without knowing their specific configuration, we don't have enough info to complain about that.
Also the separate domain doesn't really change much with CF in front. It could be nice for a few reasons, but it's not really bad. If you have a reasonable CDN already, there's really not much point getting a different one.
I've had 3 situations where my place of work was under DoS attack, in the 3 cases I managed to identify an email address and reached out asking why they are doing it, and if they want to talk about our backend. In 1 case, the "attack" was a broken script by someone learning how to program, the other two were real attacks and one of them just immediately stopped once they knew we knew who they were, the other actually wanted to chat and we emailed back and forward a bit.
99.99% of the time a DoS is someone who is bored. Talking to them tends to work.
Edit: there's some questions about the situations so I'll expand:
- The first was not a real attack, and they were doing the network calls through their authenticated API key. This was early days of a YC startup so of course there was no rate limiting in place. In this case I exchanged 2 or 3 emails and after they sent me their python script I sent them back a patch and they finished their scraping without bringing us down. Never heard from them again
- The second was at a different company, we were getting targeted to distribute email spam, because at the time we'd allow people to invite their colleagues as members of their account, and some people associated with casinos based out of Macau automated a way to spam their casinos by putting the URL in the name of the account, which went out in the email notification. I contacted one of the admin emails of one of the casinos I found and they stopped and disappeared. In this case we also locked all their accounts and prevented further logins + emailed them to reach out to support if they thought it was a mistake.
- The third one was more difficult, they weren't using any account, so all we had was network. At some point on the second day though they changed how they were sending some of the calls, and by mistake or not leaked their Telegram username. I installed telegram and talked to them, they trolled me a little bit, but stopped very quickly and didn't start it again. This one was very amusing to people in my company because I had told them this approach would work but a few of the big wigs didn't want me to do it (they didnt have any reason other than "obviously won't work to just talk"). I just did it anyway.
To be clear, you shouldn't reach out with some threats or how you're so good that you found them. My approach is of genuine curiosity, and my literal first message to the telegram person was:
"Hello, how is it going? I work at <companyname> and we're seeing a load of requests originating from your user here on telegram. Does this make any sense to you or do you think I might have the wrong person?"
That's it!
RE: your tidbit on "don't threaten bad actors with good time", I disagree. If you can brag and demonstrate that their fleet of 10k machines can't touch a single service of yours and can't even make it pause then I'd say that sends pretty strong message to other potential bad actors.
Those bad actors have to be discouraged. I would get a very nice ego trip if I knew I can show the finger to a state actor, metaphorically speaking. But again, they should get the message that they are not as dangerous as they think they are.
Though I agree with other commenters that this traffic didn't seem as scary as the last 2-3 recorded attacks going through Cloudflare.
Telling them how big and strong you are will just trigger some of them to show you just how big and strong their botnet is. And there's always someone with a big enough botnet to bring you down.
And? Let them keep feeding more data to Cloudflare so they deflect them even easier next time around.
Maybe I do misunderstand their motivation. But you seem to think they can knock down anyone they want. To me that's observably false; even the record-setting DDoS attack through (or on?) Cloudflare that was something like 1.6Tb/s didn't seem to do much.
Let them flex. They provide us with valuable information for free. :)
It is also pretty good PR.
That sounds like a fascinating story. What did they want to chat about?
>99.99% of the time a DoS is someone who is bored. Talking to them tends to work.
This is an overstatement. A great number are extortionists or state sponsored attackers. They're not interested in chit chat, except if it's to negotiate a price to stop the flood. Particularly not the ones commanding resources which are sufficient enough to make a dent in the operations of a substantial commercial entity.
Seems that they had some kind of alliance.
> we keep things as minimal as possible
Wonder what's in that file that makes it need to be that large...
4TB/200mb = 5000.
Anyhow, doing the same with a high traffic application would be a very very different animal, specially when the app has 100k+ active daily users and is doing actual stuff. The advice is not bad, but it sounds so silly. From experience every time a commercial web application was build as a monolith it became very hard or even unmaintainable in a few years, specially when 15+ Teams are constantly contribution. So pick the right hammer for the problem you have, but pretending a simple marketing webpage + payment/subscription is a good example for architecture is just a bit much.
This sounds like a dream, both in the sense that it's wonderful, and that I'm not quite sure I believe it.
Anyone have the cuneiform expression for 80/20?
anything but a flogged mare
That can easily be done on a one core server if you use an efficient web language like go/rust/nodejs.
Tableplus is a simple marketing site that serves the binary via cdn.
Most of the time when a site goes down is because it is doing something very CPU intensive either at the app layer or the Database layer. E.g an expensive query.
If queries are hitting indexes and app is doing simple auth, routing and sending queries to DB, it’s hard to DDOS it easily.
With things like Cloudflare pages and functions, someone could hit it with billions of requests/month and you’d still be in standard $5/month tier.
They could download terabytes off CDN and you’d have $0 cost.
It’s pretty radical how much you can build on Cloudflare on their free tier.
Not entirely sure it's a wise approach given the deeply asymmetric infrastructure costs of DDoS attacks, especially if the attacker has access to a botnet.
[EDIT]:
in other words, there is a non-zero probability that the attacker, piqued by the boasting, might be able at the flick of a switch to increase the intensity of the attack by a factor 1M.
“Systemctl” instead of “systemd” ? Hm, do I detect reticence to publicly admit the undeniable, vast superiority of systemd by confusingly using the name of the utility?
"We do nothing..because we can."
This speaks volumes about your attitude towards security as a business. If I was your enterprise client I wouldn't really be happy reading this.
What is it and how can one learn about it.
"To start explaining the microservice style it's useful to compare it to the monolithic style: a monolithic application built as a single unit. Enterprise Applications are often built in three main parts: a client-side user interface (consisting of HTML pages and javascript running in a browser on the user's machine) a database (consisting of many tables inserted into a common, and usually relational, database management system), and a server-side application. The server-side application will handle HTTP requests, execute domain logic, retrieve and update data from the database, and select and populate HTML views to be sent to the browser. This server-side application is a monolith - a single logical executable[2]. Any changes to the system involve building and deploying a new version of the server-side application."
As someone who thinks microservices actually simplify a lot of things, especially in complex domains, the idea that a monolith is a choice makes me cringe a bit.
I mean they start out simple, but ...
In the domains in which I’ve worked, most services receive calls over the network, and go on to make database calls that also go over the network. So whether you do the routing inside or outside a monolith makes almost no difference to latency. And what’s more, with a front end like GraphQL, you can parallelise the work which reduces latency further.
Microservices have a lot of benefits relative to monoliths, but they aren’t a panacea any more than monoliths are. They’re a useful architecture for certain workloads and a poor fit for certain others.
But in my experience it’s quite a lot more difficult to maintain discipline over the long term with monolithic architectures, and that’s why I tend to prefer microservices attached to messaging architectures like NATS. YMMV, and that’s fine.
How much of each of your microservices is boilerplate code? How much is outright copy-pasted? Microservices can be an invitation to write lots of lines of code, so that management sees that you're "efficient".
Tell some others orgs I've worked at that microservices are simple and they would laugh.
But yes it depends on the complexity of your domain/org.
> If you see you could have done something simpler, can you de-complicate it?
this is why I think microservices are generally better - because they increase the friction to create coupling, and coupling creates complexity, and in particular makes it very difficult to de-complicate things. Put another way, microservices make coupling far more obvious, and the presence of strong coupling in a microservice architecture should be a massive red flag - one that's really easy to miss in a monolith.
The unfortunate thing about participating in these exchanges, however, is that it's impossible to explain ourselves in sufficient detail, so in the monolith vs microservice debate, we all end up just speaking across each other.
It's just that in my experience, the friction didn't do much to prevent coupling, because the friction was just that you had to write a lot of boilerplate code - not hard, just boring - and they were willing to do that rather than having a design meeting with me...
DDoS attacks do benefit some specific corps, for instance cloudflare.
What's very important is to build DDoS resistant infrastructure without them, to rid of the incentive to shadow-hire hackers to DDoS and force some infrastructures to move there and pay them.
There is too much suspicion in the digital world nowdays. Like current crypto is not mainly for shaddy ops and mafia? Really?
Wake me up when you have hundreds of millions of QPS of DOS load.