Comment by cookiengineer | Hacker News Reader

Comment by cookiengineer | Hacker News Reader

Parent Full thread

This is a money laundering scheme where they are trying out how far they can go per domain.

It's also a bug in the paypal API that they're abusing, where the SDK doesn't differ between example.com and www.example.com. If webshops like yours get exploited and used for money laundering, they will mix transactions from those two subdomains, while leaving the www.example.com domain as it is. The support people at paypal are dumb enough to not take care about each case, and usually they mix transactions later also via other social media services that have microtransactions (e.g. tiktok or snapchat streams where you can gift away items).

The way paypal support's workflow works is that they have to nanually identify each and every transaction separately, meaning a human will be busy for weeks on end. Not kidding you. That's how the scammers keep winning with schemes like this. Usually there's also no way to escalate this, not even for business customers, at paypal, due to how their support offices are structured organizationally.

As a mitigation I'd recommend to block ASNs that are known hosters that do this, and double check your webshop version for known vulnerabilities and fixes.

If you don't use docker already, start to virtualize your webshop software now. I can't stress how important this is. Also double check any users and passwords you are using for the services, and the rest of the filesystem for indicators on the VPS. Disable SSH passwords and use only SSH key authentication on the VPS in case this hasn't been done already.

I'm writing this because usually this kind of scheme starts to happen after the server got pwned already, and after e.g. the ssh password bruteforce scanner was successful or after the web exploit / persistence exploit was successful.

If you need a starting point to block those botnet affiliated networks, I started both a firewall and scam database project that does exactly this:

[1] https://github.com/cookiengineer/antispam

[2] https://github.com/tholian-network/firewall

Sorry for the noob question, but how can Docker help remediate the situation? I'm currently learning about DevOps.

It can't easily, Docker should not be naively treated as a security solution. It's very easy to misconfigure it:

- The Docker daemon runs as root: any user in the docker group effectively also has sudo (--privileged)

- Ports exposed by Docker punch through the firewall

- In general, you can break the security boundary towards root (not your user!) by mounting the wrong things, setting the wrong flags etc.

What Docker primarily gives you is a stupid (good!) solution for having a reproducible, re-settable environment. But containers (read: magic isolated box) are not really a good tool to reason about security in Linux imo.

If you are a beginner, instead make sure you don't run services as the sudo-capable/root user as a first step. Then, I would recommend you look into Systemd services: you can configure all the Linux sandboxing features Docker uses and more. This composes well with Podman, which gives you a reproducible environment (drop-in replacement for Docker) but contained to an unprivileged user.

I agree with what you wrote, and add that you should make sure that your service's executables and scripts also should not be owned by the user they run as.

It's unfortunately very common to install, for example, a project as the "ubuntu" user and also run it as the "ubuntu" user. But this arrangement effectively turns any kind of file-overwrite vulnerability into a remote-execution vulnerability.

Owning executables as root:root, perms 0755, and running as a separate unprivileged user, is a standard approach.

Yep, forgot to mention that.

To take this further, it's also possible to enforce that executables are confined to a few well-understood directories with good permissions by using the noexec option when mounting. Directories with executables are mounted normally (with locked down write permissions) and everything else mounted with noexec.

I use this on Nix OS. There all executables live under the read-only directory /nix/store. So you can mount / with noexec, giving the nice property that all files that are writable are by definition not executable.

> - Ports exposed by Docker punch through the firewall

I've been using ufw-docker [1] to force ufw and docker to cooperate. Without it, Docker ports do actually get exposed to to the Internet. As far as I can tell, it does its job correctly. Is there another problem I am not aware of?

[1] https://github.com/chaifeng/ufw-docker

Docker keeps well behaved programs well behaved. You can escape in one line of shell.

How? Like if I have a Debian-Slim container running it's possible to "break-out" onto the host?

Yup that's trivially easy if you have permissions to use mknod and mount. (and if the file system namespace looks like it normally does all you need is mount.)

Docker is for organizing things for yourself, just like directories are. If you want actual isolation you have to take extra steps.

EDIT: and I feel like I should add those extra steps are exactly what most server software does automatically when it chroots itself. Again docker is really just for organizing things.

For those not intimate familiar with containers (docker/podman), can you link to a brief blog post that touches on this in detail for further reading? Much appreciated.

> Docker is for organizing things for yourself, just like directories are.

Services have the following dependencies: static data files; configuration files; executable code/binaries; library dependencies.

In days of yonder, you'd need to download/install all of that ^ on each machine where "service A" needs to run. Developers would run and test "service A" on ubuntu 18.04. But production servers had to run ubuntu 16.04 because "service X" that also runs on the same server needs a library that has not been ported to 18.04 yet.

But "service A" needs a library that was never available on 16.04. Welcome to dependency hell!

Containers bundle all of those dependencies into one object that can be downloaded directly onto the host server, ready for the "service A" process to execute. Now it doesn't matter if production servers are running 16.04. Everything "service A" needs is stored inside the container blob (including some minimal ubuntu 18.04 stuff).

the magic that lets this happen -- containers re-use the host server's OS kernel. Running a new ubuntu 18.04 container does not start a new OS kernel running. the process for your container is just 'firewalled' off from all other processes using cgroups [0]. containers re-use the host's kernel, start a cgroup'd process which starts your container's services and processes (the 18.04 'OS' services and your binary/code/executable).

short/simpler version: containers share the core of the underlying operating system on the host server.

> If you want actual isolation you have to take extra steps.

unfortunately, this means containers share the core of the underlying operating system on the host server.

containers not being isolated from the host server OS can present a security risk as you can escape from the container and "do bad things to host server". [1]

In cases where that is a problem you mostly have two choices:

* use VMs instead (a completely isolated OS instance is started for each service, cannot interact with the host OS at all -- this uses a lot more memory/cpu)

* use rootless containers [2] (container processes are launched under a specific user namespace rather than kernel namespace -- escaping the container means you only get access to the user namespace)

[0]: https://en.wikipedia.org/wiki/Cgroups

[1]: by default the docker daemon service and all the container processes it starts are running as root, which means escaping out of a container in a a default docker installation is as bad as giving someone root.

[2]: https://docs.docker.com/engine/security/rootless/

> Yup that's trivially easy if you have permissions to use mknod and mount.

Docker containers don't have mount permissions by default.

Docker is not really a security boundary (unless you use something like gVisor), so it's a bit of a red herring here.

The idea is to make your app immutable and store all state in the DB. Then, with every deployment, you throw away the VM running the old version of your app and replace it with a new VM running the new version. If the VM running the old app somehow got compromised, the new VM will (hopefully) not be compromised anymore. In this regard, this approach is less vulnerable than just reusing the old VM.

Containers allow separation of access rights, because you don't have to pwn only one program/service that is running on the host system to get physical access to it.

Containers have essentially 3 advantages:

- Restart the containers after they got pwned, takes less than a second to get your business up and running again.

- Separation of concerns: database, reverse proxy, and web service run in separate containers to spread the risk, meaning that an attacker now has to successfully exploit X of the containers to have the same kind of capabilities.

- Updates in containers are much easier to deploy than on host systems (or VPSes).

> Separation of concerns

Sorta: yes the container is immutable and can be restarted, but when it does, it has the same privs and creds to phone up the same DB again or mount the same filesystem again. I'd argue touching the data is always the problem you're concerned about. If you can get an exec in that container you can own its data.

Why do you think ISOs never really took off? I feel like they solve so many issues but only ever see folks reach for containers.

Do mean VMs? ISO is a file format, commonly used for VMs and other computers.

For VMs, they did take off and essentially the entire cloud ecosystem runs on mostly VMs behind the scenes for VPS and similar hosting.

It's true though at it seems more popular for developers to reach for containers when they need to think about deployments, particularly docker containers. But VMs are still widely in use and deployed today.

yyeaaah, i built a cloud. :) I love VMs. I'm a disciple of Alex Polvi. Lets call it an "Immutable Application VM" Stack. Each application service (or a logical group of application services) is packaged directly into an immutable VM image, and the orchestration manages these VMs directly. No separate container runtime or container orchestration layer on top of the VM. So you have an Immutable, Bootable System Image, but you would use kvm plus .iso plus orchestration tech. Basically, why does nobody built a cloud on the cloud lol??

(I helped build digitalocean from zero the pre-IPO, so I'm verrry rusty, this all might be nonsense/wrong think, and happy to be told as much! :))

Just thinking about this from a proxmox pov -- applying this advice, do you see an issue with then saying: take a copy of all "final" VMs, delete the VM and clone the copy?

And, either way, do you have a thought on whether you'd still prefer a docker approach?

I have some on-prem "private cloud"-style severs with proxmox, and just curious about thinking through this advice.

There's already unix permissions and regular namespaces. Docker is very hard to secure.

Not OP, but Im assuming its because of immutability of the containers where you can redeploy from a prebuilt image very quickly. There is nothing that says you cant do the same with servers / VMs however the deployment methodology for docker is a lot quicker (in most cases).

Edit: Im aware its not truly immutable (read only) but you can reset your environment very easy and patching also becomes easier.

It can't. Also there's nothing inherently wrong with ssh password auth.

Maybe you wanna check your systemd log once in a while. It begs to differ.

If ssh password out would be no problem, then why are so many APTs "wasting" their botnets with credentials stuffing?

Your assumption is wrong, and policies for key based auth eliminate the problem quite easily. Versus on the other hand: are you checking every colleague's password for length, charset, etc? All the time? On every server?

Probably not.

You might want to back those statements up.

Not parent, but see my sibling comment re: Docker. The issue is imo that Docker is very easy to misconfigure and gives you the wrong mental model of how security on Linux works.

On SSH password auth: its secure if you use a long, random, not reused elsewhere password for every user. But it is also very easy to not do these things. SSH certs are just more convenient imo.

Using docker does not help in this specific case - if the attackers came via ssh, they will have root access as before, and if they come in through the application, they still control your application inside the container and can make it serve what they want.

For ssh, the problem does not lie within password auth itself, but with weak passwords. A good password is more secure than a keypair on a machine whose files you can't keep private.

This is not money laundering. Why would they dispute if it's ML?

Please explain the money laundering part here?

Im not op and I’m not sure they are using it for money laundering.

A money launderer can use a marketplace by creating a seller account and buying from himself. Since he’s the one buying he doesn’t need to deliver anything but he gets the money from a legit source. Usually he would use a payment method as close to money as possible so that it leaves less traces. But in OPs case, the amounts are low so he needs too many transactions to get something valuable. And because of the disputes, he’s (probably) not getting the money (?).

It could be card testing: the fraudster has a bunch of cards and doesn’t know which is valid or canceled. The best way to find out is to test in a real site. So he’ll test out each of them and the ones that go through are good to use elsewhere. The thing is that it would be better for him not to dispute the transactions so the OP would take much longer to find out about the scheme and shut it down. It’s better to use low amount transactions in this case so it doesn’t use too much of the credit available for him to defraud and probably doesn’t warn the card owner.

Another option is doing it just to hurt the OP marketplace. If you have too many disputes the brands can fine you and if you don’t solve the problem they can turn your account off. I’ve seen it happen when a competitor was trying to hurt the e-commerce. It’s a low move and rare but it happens.

One thing that might help is to analyze the sellers too. In a money laundering and even in the other settings, it could be part of the scheme. Are they new accounts? Are their volume exploding out of nowhere? Etc

> Since he’s the one buying he doesn’t need to deliver anything

This only works (in my mental model), when you produce the product you're selling in-house – like a digital product. But lots of "reselling" type businesses try to use this scheme as well. Like a restaurant might ring up more meals than they served, or less to not pay taxes. But, is this not easily spotted when the food import(?) cost doesn't match the revenue?

Maybe I just answered my own question, if the business is able to cook the books both ways, but it would also limit how much they're able to launder. Or is the import/export balance rarely/never checked?

That's why popular businesses for money laundering are car washes and nail salons. They're mostly cash based, and have very little in the way of inventory, so it's easy inflate your sales.

I'd think a video game arcade, especially one with laser tag, would be the best option.

Especially if you stick with quarters instead of using game cards like most modern arcades. Since quarters would be recycled anyways (Taken from the games and restocked into the quarter machine), it makes it easy to just deposit the cash you want to launder as if it had been fed into the quarter machine.

It's pretty easy to figure out max capacity of any of those businesses. Audit the traffic and compare against reported figures. So yeah, you can get away with it, but can't go overboard. It's safer to just waste some high margin inventory to keep. A bar I used to go to was a ML operation and they would just ring in lots of expensive drinks throughout the night and then the boss would come in and settle up the registers. No matter how much I drank my tab was always $20. The same was true for pretty much anyone that was a regular. It was great because it was definitely impressive to order a round of patrons shots for everyone at the bar. On a busy night there would be a couple instances where the DJ would bring up one of "the guys" and let him announce that the DJ was gonna play his 3 favorite songs and while they were playing everybody in the bars drinks would be on him. Very fun.

But a car wash uses water and a nail salon hires workers. Shouldn't take long to check that those numbers don't add up with what was sold at the end of a month.

Maybe. If you calim to wash a million cars but only wash a thousand that will be obvious, but 10 washes different is lost in the noise. Nail salons are easier because you can have the expensive personalized service that no real person buys but if someone investigates you will give it to them.

More likly the above are selling something illegal though. Pay for the expensive hand car wash but get drugs instead with a cheap automatic wash - nobody will know the difference.

For higher valued goods they use horses. A saddle can go for $30,000, so you buy some $1000 saddles and sell them for $30,000 and $29,000 worth of something else.

They will gladly send water down the drain if it threatens their enterprise. Besides, you have to be on the burner for huge crimes if law enforcement is going to care enough to audit water usage. Again, minor piece of circumstantial evidence in any case.

The gentleman who owns the nail salon in my Midwest suburbia strip mall drives a Lamborghini. One wonders about the immigration status and compensation structure of the nail techs.

Same here. Our nail salon often has a McLaren parked in front.

Money only has meaning as a flow. Value moves from A to B. Forensic analysis can follow this chain quite a long way, which is a problem for people trying to hide illegal activity. They're always looking for ways to break that chain. If OP is correct and this attack allows you to covertly shift money around, that can break the chain and let the bad guys use the illegally obtained funds with legitimate services.

It might look something like:

1) get funds via illegal activity (dirty funds) 2) spends funds at an ecommerce site (dirty funds) 3) secure a paypal refund WHICH GOES TO ANOTHER ACCOUNT (clean funds)

The PayPal vulnerability allows the money to move from a dirty chain to a clean one.

It wouldn't go to another account if you do a dispute, what are you talking about?

Yeah I work at a large US fintech and this isn't ML

>2) spends funds at an ecommerce site (dirty funds) 3) secure a paypal refund WHICH GOES TO ANOTHER ACCOUNT (clean funds)

How it breaks the chain?

Account1 buys for 10k USD, requests refund, receives it?

Even if it went for some reason to account2 then there is still the chain, but why would it go to other?

Unfortunately nowadays, blocking by ASN is not going to help you out much in solving this type of issue.

The reason for this is stealthy botnets.

For a brief rundown, I'd suggest this article.

https://jan.wildeboer.net/2025/04/Web-is-Broken-Botnet-Part-...

Web scraping is not the same as web scanning.

I am aware of these types of botnets, how they work, and which companies are behind them. Hence the reason for adding my spam database to the initial comment, which focuses on exactly those, combined with the ebpf firewall module that analyzes and correlates repeated bad behaviors.

It's not a new technique btw, APT28/29 and others have been doing this for around 10 years now.

Reply on news.ycombinator.com