Help me figure out how I lost this bet to an engineer

I'm the engineer he's referring to. I still get a laugh seeing Justin's amazement when we told him her name. To answer a few questions...

- This isn't the actual photo but looks similar, except that she was alone. Justin was afraid of reposting the same photo after the results of handing over a photo last time :)

- There was no useful exif data in the image.

- We knew her high school and approximate age in Austin, but were unable to find any yearbooks / class rosters online. (Idea not pursued: try to acquire a yearbook offline).

- Facebook Graph Search and LinkedIn with the criteria we knew didn't help.

- Google Image search didn't help.

> "I'm sure there's a lot of information missing from this story"

Quite true.

Ultimately we had several methods going at once to try to figure it out.

The one that came back with results first relies on some information Justin still doesn't know we had (though he would still be impressed), but other approaches we also expected to work would be possible with only what's in the story, and haven't yet been mentioned here.

EDIT: shared more at https://news.ycombinator.com/item?id=6369751

  At this point he could not see my screen (he was standing directly in
  front of me), but offered me a simple bet – He said if I sent him the 
  picture of the girl he could find out her name.

Reminds me of:

  Sky Masterson: One of these days in your travels, a guy is going to show you a
  brand-new deck of cards on which the seal is not yet broken. Then this guy
  is going to offer to bet you that he can make the jack of spades jump out of
  this brand-new deck of cards and squirt cider in your ear. But, son, do not
  accept this bet, because as sure as you stand there, you're going to wind
  up with an ear full of cider.

http://www.youtube.com/watch?v=d8Wvgs9q3js

Simplest explanation: He knew before he made the bet.

Wow. It is stunning how oblivious nearly everyone on this thread is to how douchey it is to BET YOUR BROGRAMMER PALS THEY CAN CYBERSTALK A GIRL.

Advice to woman: Run.

Advice to startups: Don't let your sales and engineering teams be giant douche canoes and if they are certainly don't publicly brag about it.

This is obviously a shill post setup to advertise close.io and HN fell right for it.

Why would this post be hosted on the close.io blog and not on a personal blog? It has nothing to do with close.io.

Why would he mention close.io so many times?

Why would he ask about the quality of the "close.io engineers" instead of just his coworker or friend?

Smells fishy to me.

If you know the high school and year(s), you go thru the friends list of every person on Facebook who did post their high school from that year (or nearby years), and does show their friends, until you see her face.

That's only a couple hundred kids, each with a couple hundred friends.

If you still get nothing, start doing friend requests for the people that don't show their friends. I'll bet you'd get another 25-30 kids that way.

I don't see why you'd need facial recognition software to go thru 20,000 pictures.

3/4ths of them would be the wrong gender/age and be immediately disqualified.

But if you're a programmer, you can probably scrape and aggregate to speed things up - drop out the males, other high schools, only show pictures once, etc.

I think what makes this feel creepy instead of cool is that you've just described an exploit without giving any thought to mitigating the threat.

So you've found an attack vector where you can get a woman's name and address off the 'net with a picture and a small amount of information. Maybe it doesn't work all the time, but it worked at least once. What are you going to do with that attack vector? Are you going to make an iPhone app, so that others can snap pictures of random women and recreate your exploit automatically? Or are you going to come up with ways the attack vector can be shut down, things that either individuals (eg, the woman herself) or organizations (like Facebook or Google) can do to block the attack?

One of the things that looks like a clue to me is that this the lead engineer of close.io. Browsing through their site, it looks like they do "telecom in an API"; it reminds me of Twilio and OneBox.

So, here's a thought. Let's say the Engineer has access to the SMS logs (NSA-style) for Justin's phone. (Maybe his phone goes through their system. But even if not, maybe it's company-provided, so there's an admin interface provided by the telecom that lists this information.)

Justin probably doesn't regularly text people in Texas, since the team is based out of Palo Alto. So look through his logs for a number in a Texas area code (http://en.wikipedia.org/wiki/List_of_Texas_area_codes) that he's been texting more often than usual.

That gets you the girl's phone number; probably a cell phone. Now, you need to go from the cell phone number to a last name. Reverse Phone Lookup probably won't work for this, although it's certainly worth a first try. More likely candidates: looking through the users table at close.io for a matching phone number, or doing a google search for the phone number. Or, try calling in the middle of the night and seeing what the voicemail says. Any of these approaches might work.

Once you've got a last name, grab a copy of the white pages. Most public schools publish their district boundaries, so go grab a copy of that. Look through all entries that match the last name, and see how many of them fall within (or close to, in the case politics made the boundary change) the district boundary. You probably will get a handful (maybe three or four). If you only get one, bingo!

If you get more than one, I'd call the cell phone to see if I can get a first name somehow. Then, I'd go through the landlines in the White Pages and call them one at a time, "Hello, may I speak to X? Sorry, wrong number." until one matches.

As someone who happens to be good at finding people on the internet (true confessions), the approach I would have taken would have been related to FB likes, or log-ins at the bar that they met on Facebook. I.e. they were all at the same bar in Austin, so search for people who have checked-in at the bar. And perhaps the other photo the engineers had was another location that she liked, or another piece of information about the bar in question. Then, it's just a matter of narrowing down results based on age and location. The fact that they knew the high school would have made this quest even easier. I once found an individual from a first name and the name of the bar they work at in (comically) downtown Austin. It took forever, but I had patience. :) The internet has all sorts of shortcuts to finding people if you know how to look!

If they knew her high school and were doing lots of batch processing, I wonder if they found digital copies of the yearbook and tried facial recognition. For a single subject, it doesn't even need to be particularly accurate- they can easily sift through twenty possible matches with their own eyes.

She looks pretty; At least for guys pretty = memorable. When I was single I was pretty bold with the ladies and I'd have flirted with her first (perhaps getting shot down, oh well), remembered at least some demographic information, and then used it to completely F with my coworker's mind for awhile. When I was younger I was quite the practical joker.

Note the convoluted written responses about not talking to anyone he knew and so forth which technically does not exclude the girl, herself, saying her own name.

The other alternative is she's somebody's buddy/family member so it would be trivial to know before hand that she's so and so's ex or sister or whatever and given that intel, figure it out.

Here is how I think it was done:

I assume the author is not a Facebook friend with the girl. Otherwise it would have been too easy. But friends of friends is not the only privacy leak on Facebook. There is a very old feature (which is about to get deprecated) called Networks.

Networks are created for schools, universities and other big organizations. They had the name of the high school so they have to list it as their own high school in their Facebook profiles.

This will automatically put them in the same network as the girl. I am assuming here that she has listed that on her Facebook profile.

Then performing a graph search for women who has gone to that high school and live in Austin with an age between X and Y will narrow the search a lot. Even if some of that information is not public on Facebook it would be public to them if they are not same network if she has the default settings. This is a neat trick which is still usable.

I don't think looking through the pictures would take more than half an hour.

Is it just me or is this spectacularly creepy? "I bet you can't cyber stalk my friend! I bet you we can!" I could understand if this was at defcon or a security/privacy related conference and all parties involved were in the field, but just cyber stalking some random woman? What's the point?

Assuming that she didn't show up in a reverse Google Image Search, maybe they used some sort of facial recognition API (like http://www.lambdal.com/), perhaps along with the other people in the picture, against scraped pictures from LinkedIn and Facebook based on her appx age, location (Austin), gender (F), and high school.

At least that's what I might have done.

    >My Ask

Nails on a freaking chalkboard. What is so bad about the word "question"?

I think the key is knowing what high school she went to. From there, you could possibly get yearbook pictures and student lists. Knowing an approximate age would narrow the possibilities significantly as well.

So basically the engineers knew something specific about the girl already which narrowed their data set. Unless they tell you what that was, we can go all day long guessing how they "figured" it out. In fact with this information I am interested in the actual time taken to find the girl. They could have found her in five minutes after receiving the picture and you wouldn't know. Why? From the article:

1) They were running software on a computer for hours

They are engineers!

2) The first couple of tries didn’t work

Are you sure they weren't debugging?

3) They knew what high school she went too prior to searching

If they know the high school they only have to find one or two established teachers who were around for a long period of time. It is not unlikely they can call a name from a picture after all those years.

This smells like viral marketing to me (too bad I still don't know what close.io is, should have added some information to the article. Or are you looking for new engineers? "Hey look! At close.io we seal bets with engineers. About girls! Come work for us!").

Possibilities: - they looked at signups at close.io after the evening, thus narrowing a list of female names for image searching. They could further limit this to IP's from the austin area. - They bought the ad words 'justin gold', and 'justinbgold'. Then waited for her to google you, thus revealing IP addresses for further googling.

Saying she "doesn't exist on the internet" is a big assumption. First, she is on Facebook, so she's on the internet, easily accessible or no. Second, most people leave some internet trail through online forums, newspaper articles and newsletters.

I find it very likely that she was found on the internet somewhere.

I'm guessing it has to be something to do with facial recognition, since he asked for the picture without knowing its contents (other than that it has her face in it).

From there, if he had an idea of where they went that night, a club or bar, perhaps he could do some Facebook/Twitter searching magic to search for pictures of that night, taken by other people. Then they could look for her face in the background.

Not sure where to go from there... perhaps they got lucky with one of the pictures being tagged with name. Or perhaps they identified everyone she was shown to be associating with, and found intersections in their Facebook friends list? Does it have to do with Facebook?

How are you surprised that they did it when they knew what high school she went to? Two possibilities:

Option 1: Step 1) Find copies of yearbooks going back a few years, 2) find her face in the yearbook.

Option 2: Step 1) Search facebook for people who went to the same high school as she did in the probable time frame. 2) Look for her in that set, or in the set of friends that the people who matched the search have.

Feel free to sprinkle in face detection on the yearbook/facebook photos to make it more engineer-ey.

As a matter of fact, how did they find out what high school she went to? I can't imagine you telling them that if you were not providing much information about her to anyone else.

jbg331 - while we wait for people to ferret out the right answer, can you tell us how the woman in question has responded to the "game"? Genuinely curious.

Hm. Did you guys use the Prism API by any chance?

Although this most probably isn't the case here, note how making a post impersonating the person who'd lose the bet would be a great way to solicit methods to win the bet.

This thread kind of exists as it does based on the fact that people were assuming there was something clever and software related that lead to the information because that's what the sales guy thinks, but the incomplete information we have from the developers suggests that they simply had a second photo that probably had some kind of identifiable information in it like her name that they saw via ZOOM,ENHANCE and a bit of Google which anyone could have done, even the sales guy.

FWIW, as others have pointed out, I do find this a bit creepy too, on two levels. First, the initial bet is a bit creepy. Guy involved said girl laughed it off, but that doesn't necessarily mean she didn't find it creepy that some guy she recently met was betting other people they couldn't track her down Enemy of the State style. And then on top of that, inviting the entire Internet into the game via the second obscured photo adds a much bigger layer of creep on to it, especially since in the process a lot of secondary information on her was leaked. Probably enough along with the photo of her face missing for others to find her, and you've kind of made that a public challenge indirectly. (Not one I have any interest in pursuing because like I said, the whole thing seems a bit creepy to me).

Well everyone's on his/her phone these days, so let's assume a second picture caught her busy with her phone. Highly doubtful a photo would be high-res enough to get text-level detail but certainly the general layout and colors of whatever she was looking at would show up. So if this led to a successful ID, she must have been looking at an app or page that is user-customizable to some extent like facebook with a unique profile and cover. If you have even a blurry image of a website, the RGB colors are still going to be pretty close so if you have some sort of way of automatically iterating through user pages on the website in question (GraphAPI), doing direct page capture to image, scaling down that image, doing the appropriate rotation, shear, etc., and then running some simple image match criteria, that "might" work. Facebook has billions of users though which surely would take more than 24 hours to iterate through, so maybe you could prune the search space by only searching friends of people claiming the high school graduation year in question.

Some ideas...

- using tools like http://www.pictriev.com/ or similar offline solutions to get a pretty good estimate of her age.

- using image analysis to "fingerprint" the camera used to take the picture. Then crawling sites where she would be likely to post pictures (twitter, etc.) based on the suspected area and comparing to find matches (i.e. pictures taken with the same camera). I'm not sure what the current state-of-the-art is for those algorithms, but if for instance the camera displayed an obvious visible flaw (darker spot), it would have made it easier. On a perfectly clean camera without visible defects I assume that more than one picture is necessary for effective camera fingerprinting.

- using tools like http://graphics.cs.cmu.edu/projects/im2gps/ to try and determine the location just based on visual features.

Why not ask them?

If your engineers as self-satisfied about finding her as it sounds, then they'll be happy to describe their process over a beer or two. I've never known an engineer who doesn't like discussing how they solved some riddle.

Where was the photo taken? What was in the photo other than the girl?

If the photo was taken at her home and there is any landscape in the background, the obvious initial point of attack would be to identify the location the photograph was taken.

Made an ad campaign on Facebook targeting her high school and girls of her age saying in the ad something like "Click on this link darling, need to see this, Justin Gold ;)" and redirecting her to some fake (made by you) rosy website with hearts on them where she is socially engineered to leave her name. Probably not this, but I would try it for the fun.

Really people? You are going to give ideas to track people to the general crowd. It's not even an exceptionally technical problem.

Philfreo > I think that the girl did share something with you that Justin doesn't know, such as something that she did accomplish in that school, like winning some kind of award, winning a competition or something similar. Then, you used that information along with the name of the school in Google.

tl;dr -- fiverr

1. Compose tactful email text explaining the bet. This is optional if you don't care about tact or a potential negative impact on your or your company image.

2. Go to fiverr and get a person or three to find teachers, administrators, and students who were at the same high school around the same time. Have them send the letter you composed to these potential teachers, staff, and peers. Track with close.io.

3. Get answers and try to confirm results.

4. With new data, iterate text of email and/or task request for fiverr if appropriate.

If you are skilled at chatting up gatekeepers (many sales folks are good at this), then you can call the school admin, explain the situation to her, and the answer would be yours in a few minutes.

There are some other options that are of questionable legality and/or ethics, but I will stay away from those.

I wonder what the resolution of the original image was?

The inside of her thumb is visible through the glass, so with enough detail it might be possible to record her thumb print. Even if you could do that, though, odds are against her print being present in any publicly-available database.

Did the girl leak the information? Like mentioning his name/twitter handle or something like that?

The bet is fine but writing a blog post about it on your company blog is in bad taste in my opinion.

The answer is probably simple. Social engineering. Find someone who went to the high school, then send them the picture and ask who it is. Concoct some reasonable non-creepy story like "I found this person's wallet and I want to return it".

Knowing the high school and approx age in Austin...

You probably know the year she graduated, you can run subsequent queries using +-1 based off of that graduation year. What you're after at first are rosters. The end goal is to place a picture for each name discovered.

Over 10 years since high school graduation? I'd be looking at one of the classmate or facebook groups regarding reunions. Less than 10 years since graduation? I'd still look trying to compile a list of names from these places. Old high school sports and club rosters maybe? Build a graph of people who did go to that high school during those years. Start building 1 degree of separation.

Once you've gotten a list of names, start cross referencing local groups, meet ups, colleges starting with the University of Texas -- possibly even spamming people using generated you've collected )lastname_firstletter@utexas@edu). Perhaps a phishing Email as follows "information regarding blah blah, looking for the person in this pic, we think they left their purse/cell phone etc"? Or post something in a local Austin Reddit.

I'd also do simple searches based on individual names and generated college emails. Start cross referencing all college clubs past/present with names and/or generated emails. If the person is/was in graduate school they will probably have even more school related information online.

If the person has graduated, start working the job route. Online career profiles monster, linkedIn, facebook. Have they done any sort of volunteering? Lot's of pics associated with volunteer work are out there.

5K results are a treasure trove, location, name and age ripe for searching. Some 5K's have corresponding flickr sets where you can match name to race number to a face. Once you have faces either look at them or run them through some CV for possible matches. Even if the name you've found isn't the person you're looking for such pictures might lead you to them anyway, if that person appears in a picture with your target. If you know that the person you found knows your target, start looking at that person's information. Who do they follow on pinterest, twitter, etc...

Does this person own a house do searches against the tax databases for full address of home. Even if the address isn't useful you might learn their middle name - re search with middle initial and middle names in lieu of first names.

Look at old social networking sites like myspace - the target might have old profiles up?

Did you recently add her on any social network? Even if her FB is inaccessible, yours isn't... maybe a friend recommendation engine would eventually suggest her if someone just friended and unfriended you repeatedly?

Did she sign up for a free trial of close.io?

Did anyone mention twitter ? Maybe seing her tweet at some time and looking for tweets happening at the same time in some region lead you to her tweet account. Then from a tweet account to a blog, then to her name ?

I can confirm there is no GPS coordinates embedded in the photo

Hate to sound cynical, but this seems like an attempt to solicit responses to identify good engineers to hire or promote some upcoming feature of the web app.

Did you use Tineye or Google Image search to see if there were other photos from the event? A combination of her dress and the time of her arriving there?

To all those people who are complaining about stalking:

If the girl and the engineers have no complain about each other, who the fuck are you to decide?

My bet is people who gets it right will be contacted by close.io recruiter

Justin had another photo of her in his tumblr with metadata?

She checked in with Foursquare? Some people still use it.

Reverse image search? Facial matching image search?

Since the reply is buried down below, here's how they did it:

https://news.ycombinator.com/item?id=6369859

He sent the picture to his buddy at the NSA who performed an image search with facial recognition on all images ever transferred over the internet.

I'm sure there's a lot of information missing from this story, which leads to a lot of possible solutions that aren't negated by the lack of limiting circumstances.

More than likely the author is an idiot and the image had EXIF data. Considering they already knew her highschool, it wouldn't had been to hard to go through the year books in a few hours.

This is dumb. You can get a private investigator to do this for $100 - $200, with almost surely a better method that Phil used.

Given what Phil does, the most cost effective (rational) method he could have used was to spend those hours working, and then pay a private investigator.

If he wanted to do it himself, the easiest way is to facebook search for people who went to her high school in the right date range, then ask them who is in the photo. Given the start time and end time, Phil probably didn't do that, he probably browsed their photos & friends until he found her.

Not a very good use of time, but it would work.

I think guessing the exact method used isn't interesting, because the problem isn't a particularly hard one. He's done the social engineering equivalent of picking the lock on your filing cabinet.