Happy to give guidance to a fellow startup - I know you're unlikely in a position to be able to pay for a solution.
Digital goods, donations, ticketing, any sort of marketplace -- it doesn't matter your size, just having a merchant account they can transact against is enough motive for them.
- *Track anonymous user sessions*, even if you delete all anonymous sessions every 24 hours to prevent data accumulation, this will do wonders when it comes to tracking a user on their "approach" to your payment experience. It should be cheap-as-free to log some of these events so you can identify different populations of users based on how many "typical" events they skipped. With this, you are looking for users that skipped essential or common steps.
- *Get some sort of free device fingerprinting tool in place at or before your payments experience.* [https://github.com/thumbmarkjs/thumbmarkjs](https://github.com/thumbmarkjs/thumbmarkjs) ← this is a MIT fork of Fingerprint.js after they changed their license. It's a great starting point, and while these can be blocked or manipulated, it does a TON to raise the bar on would-be attackers. With this, you are looking for users with the same device hitting your payments experience over and over, and people who are blocking this script from running.
- *Some IP reputation vendors have a free tier API (e.g. IPQualityScore)* that might be helpful at the volumes you are working with, just be sure to cache lookups so you aren't making a bunch of API calls to get the reputation of the same IP over and over. With this, you are looking for IPs that either have a poor reputation or are classified as VPNs/Residential IP Proxies/Cloud Proxies.
- *Lastly, keep a running rate for your payment failure ratio over the last 20 + 60 minutes.* This lets you put logic in place that automatically puts more strict controls in place if your payment acceptance rate dips below 90% and transactions are above a certain minimum threshold - this should let you sleep a little easier knowing that your mitigations automatically shift to "battlestations" if an attack goes off while you are sleeping or out with your family.
*Being clear:* cookies, device fingerprints, and IPs are pretty easily manipulated by a motivated attacker - checking behavior across all three at the same time significantly raises the bar for a would-be attacker.
This should keep you out of the hottest water until you get to a size & scale of attack pressure where you might want to consider using my startup's platform to proactively classify and honeypot malicious user behavior.