Youporn.com is now a 100% Redis Site
groups.google.com
groups.google.com
The way they use MySQL is also interesting IMHO: they populate a relational database in order to be able to build new indexes in the Redis side, using the relational DB for the stuff it is best at, generating new "views" of the data easily.
(Relational DBs are also good to do a zillion more things of course.)
Never did manage to see the final product though!
He said that he was confident that DB2 and Oracle could handle the load. They responded that SQL Server was much more capable than he thought and asked him about the workload.
Their response to his answer was "we can't even simulate that, maybe we're not ready".
I assume that they got better.
Though, oddly, they've been late to the game on social.
reviews, 'discreet' dating sites, sugar-daddy match-making sites, etc.
Porn does "social", but only anonymous social, which is naturally handicapped.
If someone created a social network which allowed better privacy controls, the porn sites would love it.
And I suppose one could argue that war is often just a manifestation of a desire for sex or power/money as an intermediary to sex.
Edit: Otherwise I don't see what the goal does to make the butchery of your fellow man any more or less obscene than it inherently is.
In Prometheus Rising RAW mentions that the reason monks take a vow of chastity is because it instantly frees them from one of the major hamster-wheels of the human condition.
Of course, I'm not planning to become a monk :) But food for thought...
The legal definition of obscenity requires that a work is meant to be arousing and has patently offensive sexual conduct, in addition to having no redeeming artistic value.
Therefore, adding sex into something offensive makes it more obscene.
http://www.wikisummaries.org/Prometheus_Rising
It fucked my mind when I read it :) It's a bit far out sometimes though, but it's a cool read.
It amused me that you were testing a porn site -- the homepage / potentially-misused zrange-issue.
I remember seeing YouTube storing videos using some sort of specialized hardware from NetApp or something and they still got hit by the limitation of Linux filesystem (number of nodes?)
* http://code.google.com/p/redis/wiki/Benchmarks
* http://redis.io/topics/benchmarks
I love how they use sInterStore as a replacement for a fat SQL query.
Tomorrow our site, with over 100 million page views per day, will be relaunching running on Symfony2. This is a complete rewrite. Prior to that it was running the Catalyst Engine written in Perl.I suppose if each video's metadata causes a redis lookup and there are 300 videos displayed per page that would do it, but you'd think you could batch those lookups. It seems a fairly inefficient.
Either that or they have some background jobs causing redis lookups that aren't a result of page views.
There were 76 requests made as observed from the net tab in Firebug. The majority of these requests seemed to be for those video images. Note that I'm using an adblocker and refuse to check out the site without it.
Note also that they said 300k queries. There could be constantly-running background jobs doing conversion, pulling those preview frames, and crunching numbers on the metadata.
This was all being discussed in the context of a fairly formal business meeting.
Some very interesting insights from the hotels side of things.
If I kept track of the porn watching habits of my neighbors, that would be considered strange and maybe rather pervy. And I don't see how it makes a difference in that evaluation whether they are or aren't renting the tv they use from me.
In any case, they probably have no need for individual statistics: the aggregated watching habits of guests is enough to provide useful information.
the guy from pinklab http://www.pinklab.com/reddit.html talked around there, but can't find the link. there were also some others, even a guy who ran a 2g1c kind of site.
And i am happy that my months of lobbying for Redis at the office paid :D
Youporn is somewhat quite a simple site I'd imagine. It's just categorized videos with a bunch of stats (views, related video...), and users. There's almost no social interaction between users (i've just registered an account just to confirm that).
That's a very clear (and good) reason why 100% redis suits for this website, apart from the goodness of redis like sorted sets are perfect for the "top rated" or "being watched" sections
Imagine the site now wants to add a little bit of social interaction, like allowing users to post video, follow each others... I'd say 100% redis wouldn't work any more
As far as I know, Youporn is all about users posting videos (whereby amateur porn is probably only a small part of all uploads, I guess)
Follow each other? I'm not sure you know what the point of a porn website is.
Say I'm a fan of a porn star XXX and I'd like to see a simple notification that there's recently a new video tagging her. Same thing if I like some particular user's taste for porn and would like to get noticed if he/she has posted a video. Basically if it gets more social, my wall or dashboard or whatever, would be more sorta relevant.
/briefly worked at last.fm
Or the good old' NFS?
How do you handle where to put the files on which disks? do you write custom software or there's some sort of off-the-shelf software to do this?
The reason I asked about this is because it looked like a few people I talked to seem to prefer to use Distributed File System such as HBase, GlusterFS or something else so that they don't have to write extra code.
And I'm guessing you back them all up on daily base as well?
Like I said, nothing fancy but it works without any issues (apart from a small delay when you have to copy the file to 2 or 3 different servers (redundancy) before it becomes available, but it's not a big problem but I have few ideas how to solve that if it becomes one :)).
Wish you all the best to hit and solve that problem (meaning you'll be successful by that time I'm guessing (^_^)).
What I'm saying is that there is more than it meets the eye.
Pretty much the exact opposite actually. Porn industry guys tend to be 30 - 40 on average and super cool and laid back, though in an LA sort of way. I'm sure there's a lot of crap going on beneath the surface, but at least from the outside it seems like a much nicer work culture than silicon valley.
Wow!
Mine would probably hear the reasoning out ... then ask for a less "unsavory" example he can use when talking with everyone else.
NB: The HN article link hits an expired job post. Here is original job ad via WayBackMachine: http://web.archive.org/web/20090418113245/http://jobs.perl.o...
Can someone explain why there's no Ordered Map in Redis? Basically to achieve and ordered index, one has to use Sorted Set and a classic Hash Map, and pair these two when ordered data are needed, which is an extra level of indirection and wasted space. Am I missing something?
Among other things, this indicates Redis' impending adoption as a standard infrastructure component across the industry in general and as a replacement to standard databases in some of those places.
Unless you ask how they get porn, that is.
Side-note: I also heard that the founders of Akamai started up after seeing how the porn industry handled all the traffic from geographical different places.