Facebook intern visualizes friendships, draws world map | Hacker News Reader

Facebook intern visualizes friendships, draws world map

facebook.com

430 pointspsawaya116 comments

Someone outside Facebook tried to do the same with a US map and was sued by Facebook. Data in, data out only for Facebook.

http://petewarden.typepad.com/searchbrowser/2010/02/how-to-s...

I think FB disliked his map because it showed how poor the connections between areas where. FB's map pretends that all areas are equal and focuses on the global nature of their network.

No, I believe it was because he tried to publish his dataset. As Loic said, "Data out only for Facebook."

The worst part of the whole debacle was that Facebook's robots.txt file explicitly allowed the crawling of this data.

I don't think your reading into the situation deep enough. FB changed their robots.txt file becaue they reolized they wanted to keep that information private. But, what caused them to want to make it private?

Well the reality is: Looking at the network of US cities, it's been remarkable to see how groups of them form clusters, with strong connections locally but few contacts outside the cluster.

vs this PR peice that says:

It's not just a pretty picture, it's a reaffirmation of the impact we have in connecting people, even across oceans and borders.

PS: Visualizing data is like photography. Instead of starting with a blank canvas, you manipulate the lens used to present the data from a certain angle. aka, To make something with the right look takes not just data but the correct manipulations of that data.

If you're interested: Russia is missing because we've got one very similar social network[1][2] that's got arguably more features and arrived to the market a year or so earlier than facebook got its russian localization.

[1] http://vk.com

[2] http://en.wikipedia.org/wiki/Vkontakte

I’m pretty sure that this is a development that happened all over the World. Germany had its own Facebook clone [1] early on and it absolutely dominated before Facebook entered the German market [2]. That’s now no more the case.

To quote StudiVZ CEO Riedl: “There is now no more direct battle against the global player Facebook.” [3]

It’s interesting to see that the Facebook clone worked in Russia but didn’t in other places.

[1] StudiVZ: http://www.studivz.net/

[2] StudiVZ was founded in November 2005 (a whole year earlier than Vkontakte), Facebook was localized in early 2008

[3] A week ago, in an interview (German): http://www.handelsblatt.com/unternehmen/it-medien/studivz-ch...

That's an interesting case. I might make some wild guesses based on what I see:

1. vk ports features from fb 3 to 6 months after they appear (share-widget, like-widget, connect-widget, possible friends)

2. vk's got a comprehensive search engine with lots of filters by age, city, gender, current status (married, single, in active search etc.). Lots of people I know use vk for dating. I might be missing something, but I didn't find such a thing for facebook.

3. vk's got native music and video upload. By native I mean that you don't use any third party application/service to do this. Everything's hosted on vk's servers. That's the reason copyright owners sued vk several times (RIAA even blacklisted the guys). However, they [copyright holders] lost in all the court cases.

4. VK offers cheap contextual ads. While not exactly a killer feature, it is very nifty for business, -- they only charge per-click and if you get the targeting and ad text right, you'll get more visitors than you need.

5. Another random guess is that it's easier for a German than for a Russian to migrate to English service.

5. Another random guess is that it's easier for a German than for a Russian to migrate to English service.

In principle, Facebook is localised. I could imagine that few apps are translated to German though, and even fewer to Russian, except those built by native developers; (I don't use any apps, so I don't know) that native developer pool ought to be the same as the one building VK (or VZ) apps. Assuming Facebook does a decent job of filtering localised apps, I can't imagine this to be a massive issue for users.

I can only assume the media upload is what keeps people coming back.

All of the other reasons you mention seem to be reasons why businesses would prefer VK over Facebook, but surely it's the users that are the key?

"I can only assume the media upload is what keeps people coming back."

People keep coming back because that's where their friends are.

See, that’s what I would have thought about StudiVZ and yet Facebook could spread in Germany like a wildfire. StudiVZ still has millions of users but Facebook has those same users, too. (This is only anecdotal evidence but Facebook also seems like the busier place to me, activity has transitioned from StudiVZ to Facebook.)

This tells me that even if a social network seems to have a particular market cornered, a competitor still has a chance. (Which, I think, is quite nice to know.) Network effects are obviously important and have to be considered by anyone who wants to build a social network but I think there are certain properties of social networks that dampen network effects: Signing up is easy and free, for one, and rebuilding the social graph, while some work, is far from tedious. You only have to add some friends and recommendations get you the rest of the way.

It might be just a Russian phenomenon. Are there any other countries where Google is not the dominant search engine?

Baidu-Baidu in China.

And for a bit of a non-obvious one, Seznam in the Czech Republic.

I wonder if Seznam really is more popular than Google in ČR. Do you have any data to support this? Also I wonder what the deal is with Jyxo.cz. How much behind Seznam is it nowadays? Wasn't it technically superior?

My citation for the Seznam claim is "I read it on English Wikipedia". I have no idea if it's accurate. Sorry.

Naver in South Korea. Google's been struggling to get its market share for the last few years without much success.

The most important factor is that number 3: music and video uploads. You just didn't present it as it is. Facebook also has an option to upload video. But the real difference is that VK became practically the alternative for torrents: all the movies and all the music is there. On their servers. Free to search, watch and enjoy. Facebook doesn't have and will never have it.

So if you have all your friend on VK AND you have all the entertainment stuff there - why switch? Unless Facebook comes with some interesting strategies to pull the users, the status quo will remain as is. And from what I hear and see, FB is putting a significant effort into this. It'll be interesting to watch in the nearest 1-2 years.

Well, StudiVZ just screwed up with some good money in the back. They had downtimes and they developed no new features. StudiVZ is too basic to be competitive. Now they have apps and stuff liek that, but now it's too late. I always wondered what they did with their money, because the really developed very very slow. And had major problems with downtimes for months.

I guess that this is not the case with vk.com and they may as well survive.

> we've got one very similar social network that's got arguably more features and arrived to the market a year or so earlier than facebook got its russian localization.

Except that it's an incomplete relatively ugly clone of Facebook, which is why quite a few people are jumping ship to Facebook. It does have its niceties in the form of music uploads, but that's more of a quirk of the general state of regard for copyright laws in the country.

Surprisingly, I found vk's interface much more appealing and intuitive than fb's. Besides, I can't find a single reason to migrate to facebook and so do most people I know, and as the map shows, most people in Russia.

The users really started to trickle over to Facebook during the year. I see even my less technically inclined friends appearing there now.

Give it another year or two.

Sounds like you two should make a public bet?

All three of us, sure.

Yes, indeed. I miscounted the first time.

I think Zuckerberg makes it very clear that is not about the platform but about where most of your friends are. Which is why it would be very difficult to displace people from facebook.

Same happens in Spain with Tuenti http://tuenti.com

Also, VK and FB have the same investor DST.

Others have mentioned the media uploads before, that's definitely the killer feature.

The other great thing is the lack of censorship. Vkontakte doesn't censor posts and doesn't have Facebook's policy of selectively banning links to outside websites.

You can't post a picture of a titty on your FB wall without the Facebook Nazis coming down with threats on you. Vkontakte doesn't ban adult content or adult-related groups.

The two things they do actively fight against are spammers and pedophiles.

About respecting copyright law, Facebook does nothing about people posting stolen photographs. It's just more socially acceptable to pirate those.

The other thing about Vkontakte is it's very popular with CIS diaspora and exchange student communities abroad. Lots of people keep accounts in both Facebook and Vkontakte, but the latter gets used more for things like organizing social outings and events.

backgroundify! (removed branding, resized to 1920x1200): http://s3.amazonaws.com/fb-world-image/fb-worldmap.jpg

The US / Canadian border in the West is interesting. It seems between west of the great lakes and east of Seattle there isn't much connection across the border.

zoomed http://zoom.it/9qx1

Could just be a graphing anomaly, or that there aren't many people living on either side of the border in those areas.

On the other hand, if you were to think of the kind of people in each country who wouldn't have any friends at all in the other country, wouldn't they probably live around there?

I've been in that area. I think it is real.

It is a lot easier to move around and work on one side of the border or the other, which leads to people naturally not meeting people on the other side.

In the east there are so many more people and there is so much cross border traffic due to trade (particularly from the auto industry) that a lot more cross border connections exist.

I've been to the Eastern part of the border: Stanstead, Quebec/Vermont. Over there, some of the buildings are right on the border. One of the guys I worked with lived in apartments which had their parking lot technically in the US.

http://en.wikipedia.org/wiki/Stanstead,_Quebec#Geography

Check out the library:

http://en.wikipedia.org/wiki/Haskell_Free_Library_and_Opera_...

http://www.time.com/time/printout/0,8816,948520,00.html

As for the west coast, let's not forget about the geographical anomaly known as "Point Roberts":

http://en.wikipedia.org/wiki/Point_Roberts#Geography

The other transnational gap I can notice is the India-Pakistan border.

That big gap between the two is actually the Thar Desert.

Indeed but remember that gaps only appear on the if there's a lack strong lack of social connected between the regions. So it's entirely geographic.

Partially this is likely due to the relatively low populations - Manitoba and Saskatchewan have a combined population of ~2.5 million.

Alberta bumps up to ~3.7 million, and the connections start to increase.

Added Countries: http://ow.ly/3pLxP

I really wanted to look at the map in the context of where actual countries were so I threw together a Google Maps overlay. I couldn't believe how accurately the connections followed country lines, it was amazing!

It's the same Paul Butler that posted about A/B testing a CV http://paulbutler.org/archives/experiment-in-testing-my-resu...

That kid's going places.

I wonder if google found him by posting this A/B Resume test.

You mean Facebook?

It's really interesting that you can clearly see the differences between West Germany from East Germany, even 20 years after the reunification. Also the borders of the Austro-Hungarian Empire, 80 years after the end of the World War I.

I also wonder who those users in North Korea are.

Those are all Kim Jong-il

Kim Jong Il is friend to all Korean people, but the map does not reflect that?

He is a true friend to all those Koreans. But the map only shows those who have computers. AND access to internet. Which would be his family, government and more or less that's it. The rest would love to show their friendship and devotion to Kim but can't - lack of computers is killing them...

> It's really interesting that you can clearly see the differences between West Germany from East Germany, even 20 years after the reunification.

That might also be an artefact of young people leaving East Germany for West Germany and Berlin. The facebook population density in East Germany seems to be quite low by that picture.

I might be mistaken, but I believe he took the destinations from the current city profile field. I'm sure there aren't that many Facebook users in North Korea.

What I want to know is, were the 10MM pairs all people who make their friendship data public? For people in urban areas, this is sufficiently anonymous. But for those faint dots in the middle of nowhere, is it?

I think very carefully before I reveal any information publicly about my users. Sometimes seemingly aggregate information isn't.

Seems to comply pretty well with this http://www.vincos.it/world-map-of-social-networks/

Notice the conspicuous absence of China, Russia and Brazil.

Why auspicious?

Because I'm an idiot and meant conspicuous. I'll be editing that now.

Am I reading it right? R generated that?

You're surprised? Why?

I can't speak for the poster you're responding to, but if it's easy to make things that pretty in R, I suddenly want to learn it.

We're all wondering what he used to visualize his data set.

My experience with R is that it generates much more... utilitarian output.

The visualization definitely wasn't done in R. I'm guessing he used Gephi (see, for example, http://gephi.org/2010/map-geocoded-data-with-gephi/).

I bet you Gephi cannot handle more than tens of thousands of node.

How can you tell that it definitely was not done in R? He says

> As a sanity check, I plotted points at some of the latitude and longitude coordinates. To my relief, what I saw was roughly an outline of the world.

His plot doesn't "know" about the world - he is just drawing curves between points in a plane (lat. and long.). I haven't used R, but I am sure that would be possible.

Because I use R quite often, and R just doesn't produce graphs that look like that :).

But yes, it's probably possible to draw an uglier version of the visualization. (I know you can draw the individual points on the plane, so drawing lines between points is probably available as well, though I'm not so sure about the great circles part.)

Ok. I believe you :) It just seemed simple enough that anything that could do graphics would be able to do it.

Yes, it was R. The author explains: http://paulbutler.org/archives/visualizing-facebook-friends/

the lack of china is pretty interesting. russia is practically non-existent too, but it's much larger (especially with this map projection) with a much smaller population.

I'd suspect it's because people in China use proxies to evade the filter, and hence have non-Chinese IPs.

the data used is not based on IPs, but on user-submitted "current cities" instead, as mentioned in the blog post

Oh, huh. That's weird.

I wonder if Chinese Internet users for some reason think they'd be caught evading the filter if they put their actual city in their profile?

Or maybe his sampling method was just abysmal? ;)

Or maybe Chinese people just don't use Facebook.

Yes, I made that post before I saw what smokinn said about QQ. (And what others said about other Chinese social networking companies, and American companies in China, in general.)

The lack of China on the map was interesting to me as well. I believe that Facebook being blocked in China has quite a bit to do with it.

Not really. The real reason is 2 letters long: QQ

EDIT: To explain and not be snarky, QQ is an IM/portal/social network/web mail/social game platform that is actually larger than Facebook. (Over 600 million active last I checked.) They completely dominate every aspect of Chinese web software.

There would be many more using Facebook if it wasn't blocked. I'm not sure how it's doing now but I remember lots of students last year were joining xiaonei which is just a facebook duplicate for Chinese students only.

IMO many of these sites are blocked in China so Chinese companies can dominate the domestic market rather than for political reasons.

Not at all true. QQ is more akin to MSN than to Facebook (even though it tried to do social networking features, it did them late and poorly).

Facebook would be on this list if it weren't blocked in China: http://venturebeat.com/2010/04/07/china%E2%80%99s-top-4-soci...

China already has 2 indigenous Facebooks, Kaixin (targeted towards professional workers) and Renren (targeted towards college students). It also has QQ, which is more like MSN Messenger + MySpace + gaming.

Filtered or not, US web companies don't tend to do well in China. Amazon and eBay aren't filtered, for example, but they both completely flopped here. A lot of it has to do with US tech companies not understanding how (or being unwilling) to do business like Chinese.

How do the Chinese do business? And how is it different from how U.S. Tech companies do business?

Amazon and ebay need a lot more infrastructure than youtube/google/twitter and have therefore never really put in a big effort on China. I'm sure they aren't very successful in most developing countries. Those other ones would be competitive like they are everywhere else in the world if they had a chance.

Google was the one that really invested in China and though not in the same league as Baidu yet, it was definitely moving up in market share until the government turned hostile on them. They definitely "understood" how to do business in China: they had a big Chinese staff, they even had a music search like Baidu which surprised me because it seems like something RIAA would go nuts at, and they made their own very good pinyin input system.

I'm guessing if Amazon or ebay got successful and competitive with their players, China would suddenly notice some pro-Tibet book on there and use it as an excuse to give them the boot.

eBay does not actually need a lot of infrastructure in a new market (relatively)- int's all server based. Amazon is a another story, with their inventory approach requiring critical mass of warehousing close to market.

eBay are famous down here in New Zealand (at least with those that care) for believing that serving a dial-up market with heavy picture laden HTML from 200-300ms away and no local presence was going to work. A local start-up (Trade Me) destroyed eBay pretty quickly and has gone on to post better per person numbers than (I believe) any eBay affiliate. Same story in China. I say there are/were too many MBAs in eBay and not enough (especially local) tech-heads.

Meanwhile Skype is doing great things post eBay ownership.

L an ex Trade Me MBA. The only one at the time, and part time at that.

Russia primarily uses Vkontakte.

also, brazil is still holding out for the most part. i wonder what the penetration rate of a social network needs to be in order for users to not jump ship to facebook?

I believe Brazil is heavily invested into orkut.

I wouldn't say Brazil is holding out, it's just that population is heavily concentrated on the southern coast. The northern parts of the country have a smaller, poorer population. It would probably be brighter if Orkut didn't exist, but the Amazon basin would still be completely dark.

How about a real hi-res version, or a vectorized version, or dare I dream, the raw data?

This looks pretty, but it'd be more interesting to me if I could explore it some.

All that dark may be attributed to numerous things as alluded to already: lack of development, lack of human settlement, poverty, simply not on fb

The border region between the Canadian prairie provinces and the American plains may be because the cities aren't smack against the border as in BC/WASH. or Ontario/NY etc.

The few dots in NK could be those few expats or SK/foreign tourists, or NK operatives logging in. NK does have a twitter account.

Anyway, the map is stunning and i think its a great visual of the inequalities, especially between Africa and the global North. It might also highlight the irrelevance of fb in regions where community (not the same as digitized "communication") is more necessary, strong and relevant to the life and survival mechanisms therein than costly technologies elsewhere.

> The few dots in NK could be those few expats or SK/foreign tourists, or NK operatives logging in. NK does have a twitter account.

It's not based on IP, but based on self-stated resident city.

ah, thanks. i guess we can exclude tourists then. could also be fans of the regime from outside of the country.. who knows.

Interesting. In Germany you can still distinct clearly the former eastern part, which has less facebook activity. You see the 'island' of Berlin, though. For your informaion: The union of Germany was 20 years ago...

Anyway, I'm not a fan of facebook, google and the like, as it's not obvious that the consumer pays the big bill in the end. People think of great, free services but all the billions those company make come secretly from our pockets. (Via created consumer trends, higher prices that other companies can take due to the consumer information they buy from facebook)

Weird: Sardinia is much brighter than Corsica (density for Corsica (34 hab. per km2) is about half the density of Sardinia (70 hab. per km2), so I guess that explains it?)

Weirder: Portugal (11M people, 114 hab. per km2) is much brighter than Spain (44M people, 93 hab. per km2): the difference in density does not seem to explain the difference on the map. Are Spaniards less social than the Portuguese?

In Spain, Tuenti is considerably more popular (Spanish equivalent of Facebook). This probably explains the density difference.

this might not represent "real human relationships".. people playing games like mafia wars (45 million users) keep adding hundreds of other players as friends..

Nice piece of data visualization there. +1 for using great circles instead of straight lines.

Looks a little like that brief orbital shot of the Borg future Earth, doesn't it?

Or the Taelon Commonality, for Earth, Final Conflict fans.

Brazil is almost totally dark; Orkut effect?

I'm pretty sure that's just the Amazon.

It's both effects. If we had an Orkut map it would be like this but with the brightness much amplified. The coast and the South would be as bright as (say) Mexico.

Its interesting to see many connections within India. Also India and countries in Western Asia seem more connected than India and US (which is what I was expecting assuming the FB population in India)

>Its interesting to see many connections within India.

The map is quite predictable to some extent too. Delhi, Mumbai and Kolkata are clearly visible, and I think I see Hyderabad, Bangalore and Chennai as well, albeit as less brighter dots. Bangalore not being a bright dot is curious in itself - is it that people here are more networked with other cities and countries rather than within the city?

>Also India and countries in Western Asia seem more connected than India and US (which is what I was expecting assuming the FB population in India)

How do you see something like that? I found it pretty difficult to keep track of the lines. If that were true, it would be pretty interesting indeed.

i see some bright lines between indian cities esp mumbai and the gulf region.

btw, i wonder who you are....as you are also a Sundar

Beautiful. This belongs in the MoMA.

It's interesting how sub-Saharan Africa is still a mostly dark continent. Also interesting are the connections between the Kurds from Turkey and those from present-day Iraq, and how people from Baghdad are still more or less connected to those from the northern parts of Iraq (there's still a major Sunni presence in there as far as I know).

Really shows how entrenched the established players are in the emerging markets. Russia has VK (VKontakte), China has QQ (and supposedly blocks FB too), Brazil has Orkut. Makes me kind of admire Facebook for being able to take so much market share in Western Europe, despite home-grown alternatives there.

Neat. It is interesting how similar this rendering is to Chris Harrison's city-to-city map: http://www.chrisharrison.net/projects/InternetMap/medium/wor...

I think the giant bright line through central Mexico into Arizona is pretty funny.

That "giant bright line" tracks along the coastline of the Pacific Ocean/Gulf of California. Why is it funny? There is a major highway there, and many populous cities along the shore. As you go inland a ways there is a giant desert and then a giant mountain range – it seems reasonable that such areas are thus less populated. Many parts of that map have "giant bright lines" along coastlines (for example, Spain, Australia, northwest Africa).

Were it possible to gather the data, it'd be interesting to see a breakdown of how the connections were made. I'm curious whether more were a result of travel/temporary visits or relocation/permanent moves.

They don't use Facebook in Russia?

Jokes aside, this is an amazing marketing/hiring tool for Facebook. I hope they print this on billboards.

why use euclidean distance in the function for defining weights? Doesn't it mean cities that are farther in distance with fewer friend connections would get a higher weight than cities that are close by? Not sure how it helps visualize or how to interpret it.

Presumably they're using euclidean distance to scale down the brightness of long-distance links -- to ensure that long-distance relationships don't overpower short ones in the visualization. Otherwise, a pair of friends thousands of miles apart would be contributing the same "energy" to the image as a thousand pairs a few miles apart... You'd never be able to see local detail in the areas heavily crossed by global connections (e.g. the Caribbean).

looks familiar http://www.masterresource.org/wp-content/uploads/2009/12/ear...

Russians don't use Facbook.

edit. I love this map. China and Russia are black holes.

It would be interesting to see this with full spectral progression

Oh, China, you are so dark...

So beautiful

So where's Kevin Bacon?

This is really amazing. Nice!

Reply on news.ycombinator.com