Comment by ekanes | Hacker News Reader

Comment by ekanes | Hacker News Reader

Parent Full thread

This is awesome stuff. Theoretically. But Sift doesn't actually make these functional/actionable right away through their service (even though they could). We signed up, love (LOVE!) the idea, but they keep asking for more data before returning meaningful results.

Their home page says, "Get going in minutes: Integrate in just three steps: paste a Javascript snippet onto your site, log transactions from your servers to our REST API, and send examples of banned users." But it isn't so. They require a long-term in-depth model of your site/usage including multiple fraudulent examples (what if you've mostly solved fraud?) before returning meaningful results.

According to this post and previous posts, they should be able to return meaningful results with very basic things: time of transaction, email address, etc. They should start off with: "Here's our recommendation, but it's based on limited information so we feel X strongly about it." Instead they say: "Give us more information, we can't help you yet."

Hey Aaron, I'm the OP, and CEO of Sift Science. Sorry that your experience has been subpar. I'll follow up offline.

I think there's some confusion -- we actually do have a global machine learning model, which is what customers will start with if we haven't received any labels (training examples to learn from). But, your mileage may vary with the global model. We think it's a starting point, but that fraud differs from site to site in subtle ways, and to be truly effective you need to have your own model. This is why training examples are so critical. Once we receive enough training examples, we're able to build a model specific to your site.

That said, I'll be the first to agree with you -- we haven't done a great job setting the right expectations and messaging, especially on the point above (that your mileage may vary with the global model). We've been working hard to improve this, and there are now several reminders in the console, and an introductory tour, that emphasize the importance of a thorough integration with labels to achieve best results. With machine learning especially, bad data in = bad results out. Also, our first UI/UX designer started a month ago, and improving this experience is a top priority.

With regards to making fraud facts more functional and actionable -- we actually do provide reasons we think a user is suspicious in our web console and API (see https://siftscience.com/docs/getting-scores). But, there isn't an "aggregate learnings" page like what's presented in the blog post -- that's on the 3-month roadmap.

Again, sorry for the subpar experience, and I'll follow up with you separately. We have a very technical product -- we're working hard to abstract the complexity, while making sure we set customers up for success. We've also been overwhelmed with customer demand, and have been scaling the team to keep up (7 FT employees two months ago, 15 now). That said, no excuses. Just know that we're not happy to hear about experiences like yours, and are working hard to make it right.

Thanks, Jason

(EDIT) Aaron and I just had a good phone call -- we're going to make things right.

Thanks for the call, for listening, and keep up the great work!

Yeah we work with one of their competitors and it is not simple. It takes an investment and thoughtful analysis to get use out of these services. No surprise. It's an inexact science.

But, they really shouldn't advertise it as easy to use.

I disagree: Sift can be easy to use. One of the features of Sift is its flexibility. You can make it easy to use and get some additional anti-fraud protection. Or you can make it more complicated but get better results.

Disclosure: I'm a happy current customer whose model isn't quite right yet, but is getting there.

>You can make it easy to use and get some additional anti-fraud protection

Well, that's kind of the point: sure, you can get it to function easily, but getting it to actually provide you with solid coverage takes some effort.

>Disclosure: I'm a happy current customer whose model isn't quite right yet, but is getting there.

That pretty much seals my point.

I mean, who wants semi-effective fraud protection? That's where the extra work comes in. And what you will find is that fraudsters change their game and you have to evolve your model, even once you get it customized. So, if you are simply using the "easy-to-use" out of the box signals, you are going to find that fraudsters have already been trained by other Sift customers. Sure, it might take them a moment to realize that you have implemented these measures, but they will adapt quickly when they do. If not, then you are dealing with amateurs and you really don't have much of a fraud problem.

And, that's one of the reasons it surprises me when these services publish fraud signals (as in the subject post), even if it's only a few simple ones. You will find that one of your most effective weapons against fraudsters is secrecy and misdirection. Never "train" them by letting on how you are trying to detect them and never give immediate feedback when you bust them. For instance (and I will give only a couple of simple examples), make them believe there is a problem with the site or that their order is on the way (just don't send it). The longer it takes for them to realize that their game isn't working, the better. And the more confused they are about whether it actually is working, the better. You want them confused, frustrated, and ready to move on to an easier target.

Again, my point is that for anyone who has a real fraud problem and needs a service like this, devising an *effective" strategy for using it is not simple. Instead of simplicity, Sift should advertise that fraud is tough, but they are tougher. They should talk up their models, machine learning, etc., but also their experience in and commitment to understanding customers and leading them to an effective use of their tools and mitigation strategies. This stuff isn't turnkey. It is very much a customer service and consulting business, whether they would prefer it to be pure self-serve SaaS or not.

In any event, good luck.

> what you will find is that fraudsters change their game and you have to evolve your model, even once you get it customized.

That is exactly why I want a machine learning model, so that it can constantly be modifying and improving itself. Even better that it has a network of many clients, so that it can see lots of fraud examples and IP addresses where fraud is happening in real time.

> you are going to find that fraudsters have already been trained by other Sift customers.

The opposite should be true: fraud attempts on other Sift customers have helped improve your fraud protection.

> who wants semi-effective fraud protection?

If I can get immediate marginal improvement inexpensively with a probability that the fraud protection becomes very effective, then yeah...sounds good. Sign me up.

Reply on news.ycombinator.com