I (and the entire Radar team) are on hand to answer any questions you may have!
The other point I bring about is rather rhetorical - There are no open standards, model baselines and datasets in the Fraud domain. Compare building a model for fraud detection to building a model for image recognition or object detection There is a standard baseline, standard datasets and your model competes against that baseline. Because of the open nature of image recognition, the models have improved astronomically. I feel that a lack of such openness is fraud is holding back on innovation. I could be wrong in this assessment so please correct me if so.
(And on class imbalance: we spent quite a bit of time experimenting/analyzing how to deal with it—we found that sampling rate has a marginal impact on performance but not a huge one.)
https://www.youtube.com/watch?v=4inIBmY8dQI&feature=youtu.be
Is there a way to tell when this happens in the Dashboard? There wasn't at the time, but I'm hoping this is maybe now visible somehow? It's obviously helpful to know when trying to help a customer resolve the situation.
This means that if your business model involves "try before you buy" or usage-based billing, you'd better be sure to make an initial charge, otherwise the customer might incur costs before Radar decides to block the charges.
Even if you do require an initial charge, if you allow customers to change their credit card between recurring charges, the new card could be extra risky and "fly under the Radar" until the first charge attempt.
Are there any plans to offer fraud risk and blocking when attaching a card to a customer, or will still be limited to just blocking charges? With Stripe's new emphasis on recurring billing, it seems like this would be important.
We currently see Radar as a liability for us. It might block the occasional fraud and avoid a chargeback, but it also allows customers to incur costs with dodgy cards before we know they're dodgy, and then blocks charges outright before we know.
In software sold on a free-trial model, you assume most trials don’t convert (overwhelmingly due to declining to pay but with a bit of fraud) and then the cost to provision the service (COGS) is, effectively, a marketing expense. COGS in SaaS are typically negligible to low; this is why the industry is OK with providing services on, basically, a digital handshake. If you want to allow users to try out high-COGS services (or highly-abused services) prior to verifying capacity/willingness to pay, you’d need some way to credit score potential customers outside the context of a particular payment.
To date, we’ve generally focused the bulk of our ML efforts on things which apply to the majority of our users, but as we get better at customizing these technologies to specific industries at scale and even on a per-account basis, we could certainly imagine applying them in contexts that are more relevant in your model. I’d love to hear more detail about your use case; feel free to email me (my HN username at stripe.com). If we get closer to shipping something that is probably interesting, we’d be happy to give you a heads up.
- compute a huge number of features, many of which are quite complex (involving data collected from throughout the payment process), in real-time: e.g. how many distinct IP addresses have we seen this card from over its entire history on Stripe, how many distinct cards have we seen from the IP address over its history, and do payments from this card usually come from this IP address?
- train custom models for all Stripe users who have enough data to make this feasible, necessitating the ability to train large numbers of models in parallel,
- provide human-readable explanations as to why we think a payment has the score that it does (which involves building simpler “explanation models”—which are themselves machine learning models—on top of the core fraud models),
- surface model performance and history in the Radar dashboard,
- allow users to customize the risk score thresholds at which we action payments in Radar for Fraud teams,
- and so forth.
We found that getting everything exactly right on the data-ML-product interactions necessitated our building most of the stack ourselves.
That said, we do use a number of open source tools—we use TensorFlow and pytorch for our deep learning work, xgboost for training boosted trees, and Scalding and Hadoop for our core data processing, among others.
It's a little tough to talk about this in an HN comment but please feel free to shoot me an e-mail (mlm@stripe.com) if you'd like to talk more.
While we can’t fix the previous experience you had, we’ve rebuilt almost every component of our fraud detection stack over the past year. We’ve added hundreds of new signals to improve accuracy, each payment is now scored using thousands of signals, and we retrain models every day.
We hope these improvements will help. We want our customers to be able to provide you services; that’s what keeps the lights on here. We’d be happy to look into what happened if you have specific websites in mind—feel free to shoot me a note at mlm@stripe.com.
It would be pretty useful information to have, since you're collecting it already.
Edit: no polemic intended