Plenty Of Fish Hacked – Chris Russo’s explains how he did it | Hacker News Reader

Plenty Of Fish Hacked – Chris Russo’s explains how he did it

grumomedia.com

144 pointsdomino100 comments

Wherever the truth lies, which in my mind is much closer to Frind's side, an SQL injection attack, in 2011, on an actively developed site, using modern day technology, is pathetic.

I've tried to come up with analogies that explain how people can write systems without actually knowing how to program. Something like, just because I know how to make nachos or pancakes doesn't mean I'm a cook. This is the type of stuff I'm talking about though. You can know about if statements, variables and even fancy classes...you can put it all together and build a system and be hugely popular, but it doesn't mean you are any good at programming.

Duct tape programmers indeed.

So... blame the victim? Really?

Why not, when they are being completely irresponsible with millions of people's data?

Nothing like hearing reports of security issues to make one remember the adage "There but for the grace of God go I." Humility may be in order: substantially none of us are capable of delivering a system without at least one game-over bug in it.

There but for the grace of contracting security professionals go I.

Incidentally, their notification emails routinely contain your password in cleartext as a "reminder". I signed up for an account several years ago but was put off by the incredibly ugly design. Here's an excerpt from an old email from them (note, redacted by me!) This one is from a few months ago:

-- Hello REDACTED,

Thank you for signing up on 10/12/REDACTED 4:08:52 PM. Remember your password is REDACTED. --

The most recent one from had an empty string as my password, as in "Remember your password is ."

That is standard behavior for many web sites whose purpose includes no important personal information. vBulletin and some other forum engines do that by default. These site owners figure, probably rightfully so, that the support burden for a forgotten password exceeds the expected value of some black hat actually intercepting the plaintext email (low) times the meaningful impact of any ensuing activity (also low). The chief risk is in compromising a password that this user also uses for applications of high security impact, but it is not the responsibility of this particular site owner to protect a user from generally dumb behavior.

More generally: security best practice is not always about enforcing as tightly as you possibly can. Security has real costs and it's a cost-benefit tradeoff against many other factors.

That may be their assumption (clearly is, given the evidence), but I think it's a pretty poor one and it's certainly off-putting as a potential user.

A dating site contains, practically by definition, a fair bit of personal information. It's not online banking, but there's a lot of ugly stuff that an attacker could do if they could break into a large number of user accounts, and particularly if they could de-anonymize those accounts.

POF was pretty clearly sacrificing security -- which in this context means the potential privacy of their users -- in order to get more engagement and build userbase. Bluntly: they were taking risks with their users' data in order to build their business.

That's not terribly cool in my book, even though I can see why they might have made the decision. The fact that it's understandable doesn't mean that it's right.

They do have personal information, your password for that site and your email address. Lots of people reuse passwords, and that might be their email password. Once you have email & password, you have access to 90% of their online identities.

vBulletin does not store passwords in the clear.

Nor does it transmit passwords in the clear when you go to recover an account.

Same thing with IP.Board (Invision).

The only one that I know of that has done so is phpBB, and I am not sure if it has been fixed or not because with their security track record I don't even want to try them.

The "victim" framing here isn't helping anyone. This is a silly thing to argue about.

The victims are the users who mostly lack the technical fortitude to realize the perils of using a single user identifier/password combination across many different sites. The amount of identity theft that could occur by dint of PlentyOfFish's absolute carelessness renders them just as complicit in wrongdoing as the attackers who shed light on the problem. To call POF a 'victim' is absolutely wrong.

They're not the victim - they're guiltier than anyone who grabs their customer data.

When you provide an online service that gathers consumer information - particularly sensitive consumer information (such as that which you might find on, say, a dating site), it's your responsibility to secure it. If you get hacked, it's your fuck-up, your responsibility, your culpability.

And no, that's not my opinion, it's objective fact - read, for instance, about PCI DSS, and the associated very steep penalties for information leakage.

I find it interesting that my original comment started at -3 and is now +7..and that the "so blame the victim" went to +3 and is now at -1. Not sure why I find it interesting, but dang it'd be nice to see who up/down votes.

"an SQL injection attack, in 2011, on an actively developed site, using modern day technology, is pathetic"

You left out -- stores passwords in plain text.

Pof has to store passwords in plaintext, sine they include it intentionally in a weekly email reminding you to login.. Yes. I know.

At any rate, anyone who has actually used pof can tell its been hacked together by substandard developers. No polish, a bizarre ui, and just general weirdness. This is the kind of shodiness I'd expect.

You can encrypt a password in the database and still keep it retrievable (for things like this). Storing it in plain text, however, is inexcusable.

Storing passwords using reversible encryption is not much better than plaintext.

Not much better is still better.

And it can be significantly better. In a past situation I stored sensitive information using public/private keys. Even if you got complete access to the webservers and database, you still could not decrypt that information. It would only be useful if you had access to the private key, which was much more securely controlled.

Iirc okcupid would let you login with one click via email via a token. I'm fairly sure they stopped doing this a while ago. A much better solution, with a better experience as well.

Neither way is ideal, but making it easier to login probably helps viral growth.

At any rate, anyone who has actually used pof can tell its been hacked together by substandard developers.

Developer, not developers.

It started as a part time side project, and until 2007 Marcus Frind was the sole employee.

2007 is over three years ago.

Running a site as large as plenty of fish solo likely doesn't leave one with a ton of time for new development, unless he completely ignores customer service emails.

The grandparent is saying that POF should be considered to have been written by one developer, because that's what it was until 2007.

No they don't have to. They can email users a link with an embedded hash that automatically logs them in. These links can expire. This is similar to how an email password reset works.

But the user could've still forgotten it though..

The system quellhorst describes does not require the user to know the password, or even their own username. They click the link in the email and they are automatically logged in; it's an alternate authentication scheme. If they have lost their password that's a separate issue that should be handled separately but similarly by emailing them a link, again with an 'embedded hash', that allows them to reset their password.

The assumption of course is that they and only they can access the email address specified in their account profile. If that's not the case then this all becomes a bit more problematic.

Yeah, I agree it's unbelievable stupid. that was meant as a tongue in cheek statement, hence the 'yeah, I know' at the end.

OKCupid used to use tokens fwiw.

Hey, now ... I bet a lot of banks do it, so it must be okay. :-/

Between that, and mandating short passwords, which a LOT of banks do. :-(

There are lots of programmers out there who do a fine job, they are just clueless about security.

For companies that have programmers that work great, but aren't security experts, it's time to hire security experts.

Suggesting that all the work the programmers did is worthless because they aren't security experts is seeing the world as black and white. It's gray my friend.

> There are lots of programmers out there who do a fine job, they are just clueless about security.

Yeah, there are lots of mechanics out there who do a fine job. They are just clueless about breaks ;)

Stupid analogy aside: If you as a programmer who develops stuff for production are not aware of rules like "NEVER EVER FUCKING TRUST ANY USER INPUT" then you're just wrong for the job.

I don't agree because I work in the industry.

Some are programmers who manage the User Database. They deal with all user accounts data. But they aren't the same guys who are responsible for server security.

I'm talking Enterprise Apps that support tens of thousands of users.

This guy was responsible for everything, being the sole developer.

There are also lots of programmers who are clueful about security, but work for clueless managers or organizations. I've had many a conversation that went like this:

"Okay I put together this prototype and it's working. I should check it over for SQL injection spots and---"

"StoptalkingnotimeforthatSHIPITNOW!"

The problem now is that with all this noise, POF is certainly going to be hacked. Others will want to follow Chris path.

POF founder habilities are going to be tested in the media arena and in the technical arena at the same time. Also lets see how he manages the customers & the legal side. Hard!

Calling author(s?) of POF `duct tape programmer(s)' does harm to jwz (http://www.joelonsoftware.com/items/2009/09/23.html)

They (people who don't actully know how to program) can build a large system on a shoestring budget exactly because they don't know how to build it properly, or how hard it is to. No distractions over those pesky little matters like ACID properties, security, accessibility, etc. etc..

Because those are a major part of what slows programs developed by professional developers.

No distractions, no slowdown, until a security consultant mails you a proposition you can't easily refuse. The hackers Mr Chris Russo alleges were exploiting the hole did it quietly (if at all), for obvious reasons. Nobody knew, nobody cared until now. Several years of good business, right? And free publicity now, isn't it?

To call these people Duct tape programmers because of one SQL injection weakness is somewhat unfair. Writing code isn't hard. Writing code that works is harder. Writing code that's bug free is very hard. Writing code that is not only functionally bug free but non-functionally bug free (including security as a non-functional requirement) is ridiculously and ludicrously hard.

It's only because some frameworks do 90% of the heavy lifting (and parameterised queries for the 5 extra percent that's not covered) that we don't see SQL injection as much as we used to.

This week it was PoF. Previously it's been Twitter, Facebook, even HN. Are the people behind these Duct tape programmers?

I'm going to touch on how hard security can be in my talk at http://www.meetup.com/HNLondon/ - if there are any London-based HN readers that are going, I look forward to seeing you all there.

Frankly, Marcus does not sound like he is very computer Saavy. He sounds like one of these people who knows how to write some PHP scripts, but does not really understand the details of what goes on behind it - so assumes that what this guy is doing is some terribly complicated hack.

I think the argentine guy probably searched the website for vulnerabilities as a way to get business. He found one and contacted Marcus. Marcus then freaked out and panicked, leading to this.

Marcus taught himself ASP.net http://plentyoffish.wordpress.com/2006/06/14/how-i-started-a... His story is pretty remarkable considering POF was a one man operation until recently.

ASP.net is just Visual Basic. It's not the most complicated language to learn, and you don't need to know much about computers to develop in ASP or VB.

That isn't true. Asp is the web piece of the .net framework and can use VB, C#, python, ruby or any other language supported by the framework.

But is that how he used it in this case?

Saying 'ASP.NET is VB' is sort of like saying "General Motors is a Buick Skylark." It just doesn't make any sense. ASP.NET is a framework for developing web applications. You can use any number of supported languages - C#, VB.NET, J#, Ruby, Python, F#, etc. I can only assume that he conflated the late 90s scripting combo of ASP/VBscript with ASP.NET.

It's obviously possible to write terrible code in any language, with the aid of any framework. It sounds like POF did just that.

The original site was in ASP, then he migrated it to ASP.net. I doubt that he paid any attention to the 'framework' part of things. He just did a VB syntax migration.

Rails is just Ruby. It's not the most complicated language to learn, and you don't need to know much about computers to START developing in ruby or Rails.

See? This has nothing to do with platforms, and everything to do with realizing your strengths and weaknesses, and improving your platform when hundreds, thousands, or even millions of users trust you with their data. You can't just chalk this up to POF being on a platform you think is somehow inferior to your platform of choice.

Ruby is difficult to learn compared to learning ASP.net if you went from VB6 to ASP to ASP.net. I'm familiar with those languages.

The OP posted that he learned ASP.net as if it proved that he's really good at programming.

I don't care if he taught a cat to fish for tuna, he fucked up big, and in a bad and trivial way.

> I don't care if he taught a cat to fish for tuna

I'm going to try and remember that phrase; it sounds like it has 1,001 fun uses. :-)

POF is hardly a great example of superior software engineering. The whole point of POF is really affiliate lead generation for other off-network dating sites.

I'm pretty amazed that in this day and age companies still store sensitive information like user passwords and credit card numbers in plain text.

IMO storing this kind of information in plain text was never acceptable. It requires the right combination of arrogance and incompetence for this to happen.

In some ways it's no wonder POF was able to bootstrap and run this amazingly large system all written by one guy with limited hardware. Dunno what their stats (or headcount) are now, but years ago POF was heralded as some genius site because it was all put together by one guy and running on a few load balanced servers.

I'm not saying it takes multiple people to make something secure, but if one person either doesn't have the experience or knowledge to make something secure, and there's only that one person, there's no one else to even determine there's a problem.

This appears, unfortunately, to be the norm and not the exception. It comes to mind every time I sign up on a new service.

Obligatory: http://codahale.com/how-to-safely-store-a-password/

Not only does POF store it in plaintext but they email it to you every few days when they send you their "new matches" email. So basically they're sending probably about 1/4 of their member base's passwords in the clear over the internet every day through those emails.

I know how and why you should store hashed passwords, but surely you'd have to store credit card numbers in plain text? If you store a hashed credit card number, you'll be unable to charge it again? How do you use a hashed credit card number?

Ideally, you don't store credit card numbers at all. If you need to do recurring billing, there are platforms that will do it for you, otherwise, process the transaction and remove the number from your system. Storing this kind of info securely and in PCI compliance is not trivial, and takes engineering resources that are better used on something that will distinguish your product from competition.

I'm not surprised PlentyOfFish was hacked in the least. I signed up at one point and the password entry form wasn't even obscured. That was my first red flag. So I made sure to use a throwaway password.

But it got worse. Every few days, POF started sending me newsletters with my password in plaintext! "in case you forgot, your password is:...." in addition to whatever else the newetter said. So I knew they were storing passwords in plaintext.

Couple those things with the complete unusability of the site…well I deleted my account (which is unsurprisingly difficult to do as well).

because there was a serial killer, murdering people from the website.

err what?

I have to say, from the writing alone Chris Russo sounds a lot more confidence inspiring than the POF people. That long rambling blog post by POF did not come across as very professional.

Both posts are rather comical and infantile. Drama all the way, serial killers and Russians. Techno soap.

The serial killers are a quote from the mail of the wife of POF founder, I think?

The Russo side does sound "insecure", as he admits himself. Presumably they have not been in business for very long. But that is not the same as infantile, it's just inexperienced.

Finger-pointing is always infantile.

You've got a point there.

I'm tempted to give Russo some leeway though as English probably isn't his first language.

Definitely isn't. His English looks exactly like an Argentinian writing English.

Yeah its sounds like Frind in his panic over the active exploitation of the vulnerabilities by other hackers felt like Russo was trying to blackmail him because of his lack of command of the English language.

He was probably trying to communicate that he was looking for a consulting position after finding the exploit trying to be helpful

http://www.gather.com/viewArticle.action?articleId=281474976...

I wonder where the discussion of "murderer" or "Russian mafia" first came up, seems totally unrelated to the incidence and random for anybody to bring it up first. Could be some kind of pronunciation miscommunication like "member" -> "murderer"

Am I reading this right? Are both sides of this story intimating that the security guys found a vulnerability in the site, published it, and then pitched them a consulting gig? Go live with something or not, pitch a gig or not, but I'm not sure you can have it both ways without looking skeezy.

hmmm this really looks like a complicated matter to judge with the available information. Really there are several plausible scenarios where POF or Chris could be at fault. The information available to the public is not enough to make an informed judgement. Everything else is pure speculation or pure DH0 (as per Paul Graham's disagreement hierarchy).

For what we see now, it is clear that POF people were scared shitless from the report (obviously) and tried to limit the damage and gain time by manipulating Chris whom they thought has downloaded all their users data and is extorting them against the disclosure of this data. I think anyone in their shoes would think the same (at least primarily).

Now wether Chris really intended to extort them or not is another matter we cant judge from what we are seeing. He does seem like a nice guys from his emails, and kind of excited about the "opportunity" but really that is not enough. Nice people sound nice... so do most criminals.

One more thing that is unclear ! Was POF actually hacked (in the sense that data was leaked) or was it just that a vulnerability got exposed? I think this will tell us a lot as well about the real motivation of the "hacker".

To discover a security vulnerability (Chris Russo's POV) in a site and to hack a site (POF's POV) are different things.

  POV - point of view

You are right and I changed the title of the post to "Plenty Of Fish Hacked - Chris Russo explains what happened" instead.

> By the nightfall of Sunday 30, Mr. Markus Frind sent me an email accussing us to steal his whole user database without a single proof, based on supposed information that "20 employees of him told him.

This fellow is either a fool or a terrible liar. If he didn't take the data then he would have said "I didn't steal the data". He instead accuses Markus of a lack of proof.

Lack of proof == credible, since the burden of proof is on the accuser

"I didn't steal the data" == my word against yours

There is no burden of proof in the court of public opinion. There is no 5th Amendment either.

Imagine this scenario, your boss says to you: "20 of our employees are telling me you piss on the toilet seats."

You respond, "You have no proof".

Good luck with that.

     there is no burden of proof in the court of public opinion

Except that article by Markus Frind walks a fine line between libel and opinion.

And you can get your ass sued to stratosphere and lose if you fail to produce reasonable proof when making public accusations.

I agree with you. Defamation / libel laws are far more strict in Canada than in the US. This is clearly going to interfere with Chris Russo's ability to earn a living as a security researcher. I have a feeling he is going to get paid to go away at some point.

So then you look at whose word seems more reliable. It would be very weird if all of Markus' employees plus one of his competitors made this crap up. It would be much less so if Russo were trying to avoid the repercussions of his actions.

... unless "lack of proof" is a common phrase in his native language. He may of meant something closer to "has no basis" for all we know. Nuances shouldn't be analyzed with English this bad.

Wow. I just read the Inc Magazine article about this guy from 2009: http://www.inc.com/magazine/20090101/and-the-money-comes-rol...

Highlights:

1. The site makes $10 million/year. 2. He has three employees, all customer service. He does everything else. 3. He pays himself $5 million/year.

This quote is particularly telling: "At other sites, when one thing goes slightly wrong, the reaction is to buy more servers or hire a Ph.D.," he says. "It's almost unbelievable -- it's like people are trying to justify their jobs by spending money. This isn't rocket science."

This guy still sounds like a scumbag. He's complaining they responded to someone threatening to post people's private info by saying they'd post info on the person doing it? Boo-hoo.

If Chris wasn't going to post the info publically (ie, extort POF), then he had nothing to worry about.

As an observer this certainly doesn't make Mr Russo seem any better.

Edit: per below, removed work extortion as posting people's private data might be done for other reasons rather than to make money.

I don't read anything about an extortion attempt (in this account), what am I missing?

I guess it is normal for a security firm to hope to gain some business by exposing security flaws (hey, we found a hole in your site, hire us to fix it). As this incident shows, it is not an easy business to be in...

That's true, the guy might just post the data to be an asshole, rather than asking for money.

Either way, the guy who was cracked is saying if the cracker posts the stolen data, he will post the cracker's contact details to every account. Which is completely reasonable.

I didn't read anything about him posting the stolen data. I think he specifically wrote that he didn't even steal any data.

I don't know whose account is true, but I don't think it is reasonable to threaten somebody who approaches you with information about a security hole, just in case that somebody has evil intentions. In general it seems reasonable in human interactions to not issue death threats upon the first encounter.

> I didn't read anything about him posting the stolen data.

Read the post again - the owner of the site is saying he will publish the guys details if he posts the data.

If he doesn't post people's personal data, fine. The site owner won't publish his address.

If he does, then fair enough.

Hi, nice to meet you. Btw, try anything funny on me, and I'll set the Russian mob on you.

How to explain? Yes, strictly logically speaking it is "OK" (no funny actions, no Russion mob). But psychologically I don't think it would be a good approach to human interaction. It establishes an atmosphere of distrust right from the start. You approach somebody and say "you might be a criminal".

Or imagine you meet somebody on a party and he says "Hi, my name is soandso. By the way, I carry a gun". How would that make you feel?

I understand what you're saying, but do you have any information that this is how the site owner began the conversation?

I imagine we're seeing a small snippet of something larger.

I also see the security company guy doesn't dispute that his revelation the site was cracked began with a push for payment to remedy the situation.

No idea, really. It sounds as if Russo denies having any data, in which case I struggle coming up with a good reason for POF to issue the warning. But ultimately, I really only know what is written in the two articles. I guess time will tell.

> It sounds as if Russo denies having any data

I don't get that impression - I imagine that if he didn't have any data, he'd say so explicitly. If he didn't have data, though, I'd totally agree with your observations.

> I guess time will tell.

Indeed.

     If Chris wasn't going to post the info publically (ie, extort POF), 
     then he had nothing to worry about.

Considering that anybody can post that info anonymously and that the vulnerability used was there all along, if Chris is in fact innocent then he had plenty to worry about.

My understanding from reading the post is that the site owner would only publish Chris' private details if Chris published the private data, not someone else.

And how would the site owner know if it's Chris or not who published the data?

Chris may publish people's private details with his own name attached. It would be stupid but my expectations are low.

If, in a short period:

* Chris' personal account was used to crack the site

* Chris asked for money and was rejected

* Chris potentially threatened to leak the data

* The data was subsequently leaked anonymously

I'd say that there is also a good chance Chris was directly responsible.

Unfortunately on the web it is hard to establish the chronological order of things.

I.e. that data could have already been available to you if you paid the right price since 2 months ago.

Correlation is not causation, and we of all people should realize that.

I'm not defending Chris, it's just that Mark sounded like a jerk in that blog post and gave a more complicated explanation of the events than Chris; which means there's a good chance Mark got really defensive and blamed the person trying to help.

Nothing, including all of the justifications for the terrible security practices, should excuse Marcus Frind for the death threats and sheer hatred being thrown at the messenger.

Chris mentioned that this was actively being exploited by malicious hackers. Taking your rage out on the guy trying to help is not appropriate by any means.

Maybe relevant: http://www.reddit.com/r/netsec/comments/f647p/i_found_a_seri...

Sorta what I asked for the original thread, but not any great details. More or less what I expected though.

I guess that'll have to sate my curiosity for now.

in any case epic fail at damage limitation by all involved

Wow, I don't know whom to believe. But it's 1a drama. And I love drama :)

Reply on news.ycombinator.com