Ex-Googlers Launch Sift Science (YC S11), a Fraud-Fighting System For Websites
techcrunch.com
techcrunch.com
The article mentions a 90% detection success rate, but it doesn't go into rates of false positives (people who aren't scammers who are flagged as scammers) or false negatives; and it doesn't mention if there's any path for people marked as scammers to become not scammers.
Also, do you have any plans to feed information to law enforcement in various jurisdictions?
I agree that the rate of false positives is extremely important here.
The false positive is, out of the ones they found, how many weren't fraud.
So if there are 10 real fraud attempts and they identity 9 of them, but find 90 others as well, then they still have a 90% fraud detection level, but they just hampered 90 customers who did nothing wrong. This would be an extreme case, but it should make the point. This applies in medical science at lot when false positives can mean lots of time and money spent on patients who don't really have the disease that was tested for.
The parent's point was that he wanted to know the false negative rate when they found 90% of fraud attempts. I was asking if that 90% number meant that by definition there was a 10% false negative rate.
False Negative: Identify person as safe when they are committing fraud.
False negatives lead to losses for large corporations and small companies growing. False positives destroy individuals lives in extreme cases.
If the company becomes large, they should have a way to deal with false positives. Hopefully a number that people can call and talk to someone. Making people wait days to complete a transaction or worse yet, not allowing them to make it at all is bad. If they become really big, this becomes really bad as it could cut people off to services and products that they may really need.
Obviously, these are long-term problems and if they arise, the team is doing a lot of other things right and I'm sure they will fix it.
Then...
pc86: "Maybe I'm misunderstanding your question, but doesn't 90% detection imply a 10% false negative rate"
Then...
atwebb: explains difference between false positive and false negative.
Then...
pc86: "The question was about false negatives, not false positives. The parent's point was that he wanted to know the false negative rate when they found 90% of fraud attempts. I was asking if that 90% number meant that by definition there was a 10% false negative rate."
But... the parent's point was about false positives, not false negatives. In order to clear up the dialogue I put the definition of each, then re-asked the question which was not previously answered.
The question still stands: What are they going to do about people who are detected as fraudsters who are actually not trying to commit fraud?
Pc86 pointed that we actually know the rate for false negatives, if we assume that 90% success means that the system did not detect 10% of real frauds. So he was asking if he misunderstood the meaning of "90% success".
edit: typo
Since the false negative question was answered, why are you focusing on it? The false positive question is far more interesting as it will impact individuals, rather than the buyers of the service who will work to understand the false negatives and the risk implications.
To me, the false positives will be the same as someone having their identity stolen and not being able to get credit. This has wide reaching implications for those individuals. Its likely not an issue for them as a company, but I hope they are thinking about the implications so they are not harming individuals.
[added] I assume everyone knows what false positives, false negatives are, the point of my original comment was to redefine the thread since the false negative problem is addressed in the TC article.
DanBC said the article did not mention FPs or FNs. I agreed with the FP point so from here out lets ignore anything related to FPs. My question/point was that as far as I understood it that 90% detection rate by definition means there is a 10% rate of false negatives.
FNs to me are not interesting because they will be studied by the service buyer so there is someone obviously interested in researching it and the market will make sure that FNs are low enough or the service will not make any money. FNs will automatically be solved. Asking about them doesn't even make sense unless you are a buyer.
FPs on the other hand are not important to anyone but potential 'victims' (too strong of a word, but still) and those victims do not have any way to negotiate their position in the process.
By forcing people to use the system by signing up service providers, the company is creating a situation where FPs are a very big deal for those that are flagged as potential fraudsters.
I could care less about FNs since those will be naturally resolved by the market place.
In general, a level of FPs has to be tolerated, even by the most aggressive activists if the company can reduce a lot of fraud, but FPs are still serious.
That being said, do you differentiate on the type of fraud? Fraud is not fraud. I guess my question revolves around real fraud (stolen cc data being used), and friendly fraud (me charging back a transaction fraudulently claiming I never received a service). I'd be curious to know if you are handling that, and how. Is it merely a case of asking the people using your system to rate a particular transaction a certain way and putting into the pool of data to be processed? Or do you do something more?
Also, you mention skipping verification steps. Do you do things to recommend specific extra steps, or do you just provide a score and then let the client figure it out? For example, a score of 90 might be reached in different ways. In some ways, you an automated phone call could alleviate much of the concern, while it in other cases, it wouldn't. I guess the question really comes down to whether you provide guidance on which verification steps to take.
Google search detection system that identify a real user from a bot has always failed when the user travels through the Tor system. Thus, it looks to me that any machine learning system, even those produced by someone with immense amount of data, will always have a high false-alarm rate. This is also obvious if one look at gmail when it wrongly mark legit mails as spam, and at IDS/virus market. How will Sift Science will deal with the issues of false-alarms, and how should website owner act regarding the score. Should they automatically follow the score given, or just flag suspected users. What is the suggested best practice? If its a webstore, should they block, delay, or just observe the transaction?
Most of our customers review each user we flag, and have a human make a final go/no-go decision about whether to accept the transaction. I'd recommend starting with that configuration.
Once that's working, then I think it makes sense to automatically block users who have exceptionally high thresholds. We can help identify a threshold with a low rate of false positives!
> “But as we talked with people, we realized that it wasn’t solved at all,” he says. “Nobody was using them, and there were a couple of holes – things that existing products didn’t do. And one was ease-of-use.”
FWIW, the reason I am not using any of these providers is not because they require talking to sales people (I generally consider having dedicated sales staff and account managers a feature), and not because they require SOAP (I do not understand why that is a big deal... there are many libraries, and I'm a software developer ;P), but because I did not know that this class of company existed: when I have asked other companies that are similar in any way to mine how they handled fraud (such as Etsy), they always tell me they are using an internal solution (often even one that they say "we are considering licensing to others"); I thereby have been planning my way down that same path. I then wonder whether there is some other systemic reason why this is the case? (Such as, maybe the economics don't work out? Maybe there are legal concerns?)
On that last point: I guess I'd have some reservations about sending personal user data to you in a way that you aggregate, that I'd then have go through legal review on; so, even if I were sold that your service would be epic, it might be that after review I feel like I cannot be legally compliant with various privacy laws or security certifications I have to maintain at the same time as use your product. (I notice you have nothing on your website touching on this specific problem.)
> Beyond the first 5,000, it’s 10 cents per user per month.
What kind of customer are you targeting as your primary customer? I see the mention that you are getting large marketplaces, payment processors, and "top 500 retail websites", but you also seem to be going down the "freemium and no sales people" route (which I associate with trying to pick up the long tail, ignoring the larger players). Then, as you have an actual sticker price (which is maybe something I'm reading too much into, but if you are going to provide a price there's probably some reason behind it), I go to review it, and I find an inconsistency as I can't imagine that many payment processors--companies that generally deal in tiny percentages of transactions--would be able to afford that per transaction: that's as much as the entire fee charged by many of these companies (even with a 20% discount).
As a more personal example (as I am interested in your service), I process tens of thousands of dollars a day in transactions, and I would absolutely love to be able to outsource fraud detection (and despite having reservations about some of the other benefits listed by this TechCrunch article, most of which seem to be about picking up smaller customers, your "we use machine learning, others don't" is enticing to me); but, my margins are low enough (on a $3-$4 average ticket, 70% upstream licensing, then various transaction fees, costs on the disbursement of the 70% itself, various costs related to the various forms of sales tax compliances I have) that I doubt I could handle $0.10 to do a fraud score on every single transaction (although I haven't analyzed this fully yet). (This might, however, be because a good brunt of my fraud management problem is already currently either outsourced to a third-party payment network or simply absorbed as a cost by them, such as the case with Amazon Payments, which does not charge for chargebacks, despite having an insanely large number of them.)
> Fraudsters often operate in far-away timezones, so activity late at night is correlated with fraud.
How account-specific are your rules? I ask this, because what is going to look like fraud for one site is not going to look like fraud for another site. As an example, I have a worldwide userbase: I would be concerned about even a theoretical increase in the false-positive rate on users from Europe, or somewhere like Saudi Arabia (which has an inordinate number of my paying customers).
> Every site has some unique fraud patterns. You can train Sift Science by sending $label events, and you can also explicitly provide detailed feedback via a trainer page:
Is there a way to get large amounts of old information stored into your system for rapid training? Information on chargebacks often takes over a month to receive, so attempting to start scoring things based on transactions that only begin as of right now is going to require a rather long time before any real training can begin.
> The other big issue with the existing systems, Ballinger continues, is that they use a fixed set of rules. That is, they would flag transactions over a certain dollar amount, or every transaction from Nigeria, for example. But using fixed rules is a problem because Internet fraudsters don’t play by a fixed set of rules – they’re always adapting their tricks to stay ahead of the preventative techniques.
Do you know what the actual effectiveness gap is? Like, are we talking "the alternatives are simply not effective" (which seems highly unlikely)? If not, I presume it would be something vaguely quantifiable like "we catch X% more fraud than alternative solutions", which would be useful for attempting to compare your product to alternatives (as if you are 5% better at catching fraud but cost twice as much, then maybe it isn't worth it if your business model is setup in a way where fraud isn't that costly to you).
How do you gather data in that case - can fraudsters not just go "under the radar" ?
Maybe there is more to this than the techcrunch article states which is "Businesses can integrate Sift Science’s technology by copying and pasting a small snippet of Javascript code to their sites, the company says."
According to the article, the client JS is just used to get some extra metrics, they rely on the server talking to it as well.
You can see the steps for integration here: https://siftscience.com/quickstart
I wrote one of the first bigtime fraud detection web systems (hence my name...system is also still in use today and finally adapted to an approach my team and I architected over 5 years ago! Yay to shortsighted founders/VC!) and am interested to see what they do.
BTW, the newly adapted approach previously mentioned? Yeah, it looks alot like this. Best of luck!