How YouPorn Uses Redis: SFW Edition
blog.togo.io
blog.togo.io
Nowadays the vast majority of sites are built using Java services and a PHP frontend. There is still some legacy Perl about running some fairly critical systems, but there are very few people in the place who could regard themselves as full time perl devs.
Source: http://www.gabrielweinberg.com/blog/2009/03/duck-duck-go-arc...
They've been optimizing the heck out of it, and as much as my kneejerk reaction to their infrastructure is "a collage of rubberbands and paperclips waiting to break in unexpected and unpredictable ways", I've gotta hand it to them for not giving in to the temptation to prematurely optimize. Clearly it's been working really well for them, and so I guess it's worth not "fixing" it until it starts breaking.
The first question the rest of the board asked me was "Well, are we going to have trouble hiring?" To which, I truthfully replied "No." The interesting part about non-popular languages, is that you're going to have fewer unqualified candidates. (And yes, in my local market, there are still a lot of perl programmers, maybe not as many as Ruby programmers, but the average perl programmer I come into contact with has more years of experience than the average Ruby programmer I come into contact with.)
I think the argument "we can't hire people that know X" is nearly always the wrong one. Several years ago, when we looked to re-build core services for a product (replacing perl with Erlang, natch!) - we couldn't hire a person that knew Erlang to save our lives. So, we hired good developers in other languages and trained them. If you've got the right technology and the the right tools, you can always find the right people.
I think the YP people weren't just going for some senior developers, they wanted a huge pool, probably with easy enough replacement if someone is going. Considering that not everyone wants to have a porn site on their CV, or might want to quit once they start a family and would have to explain where daddy/mommy works, that seems like a good idea anyway. So you need the largest pool possible, which generally boils down to either PHP or Java, with the latter being a bit more expensive.
Curiously, anyone who's been involved in the Perl community for any length of time will see "PERL" there and dismiss the poster out-of-hand. The decade-or-two old catchphrase goes something like:
"Perl" is the name of the language. "perl" is the implementation/interpreter. There's no such thing as "PERL".
People who type LUA insta-brand themselves as outsiders.
Anyway we use Perl pretty much exclusively at Booking.com, and I'm pretty sure we're at least a couple of orders of magnitude bigger than the likes of Crowdtilt. There's a bunch of other interesting companies using Perl as well, a lot of them just don't make as much of a fuss about it as the companies using say Ruby.
Last I knew e.g. Morgan Stanley used it for most everything, including new developments, and it alone probably employs more programmers using it than the combined hobbyist contributor base of some newer emerging languages.
It's pretty hard to claim that Perl is "dead" and square that with the ongoing activity on the CPAN (https://metacpan.org/recent). This is not the kind of activity a dead language gets.
I think what a lot of people mix up is that just because something isn't as big as it was before doesn't mean it's dead.
C isn't as big relatively as it was in the 80s, Perl isn't relatively as ubiquitous for scripting and web development as it was in the 90s, but neither of those languages are dead.
If you look at the total amount of contributions / software written for any of the languages that aren't currently in the spotlight you'll most likely find that we're at the high water mark of the number of the amount of software written in them, and the number of people that have been employed to write software in them, just because the industry as a whole is getting bigger every day.
Maybe they're not bigger relatively compared to some other languages. But that has little bearing on how good they are when it comes to using them in your toolchain. If anything they're better than they've ever been before.
It's actually laid out in perlfaq1 ($ perldoc perlfaq1)
What's the difference between "perl" and "Perl"?
One bit. Oh, you weren't talking ASCII? :-) Larry now uses "Perl" to
signify the language proper and "perl" the implementation of it, i.e.
the current interpreter. Hence Tom's quip that "Nothing but perl can
parse Perl."
Before the first edition of Programming perl, people commonly referred
to the language as "perl", and its name appeared that way in the title
because it referred to the interpreter. In the book, Randal Schwartz
capitalised the language's name to make it stand out better when
typeset. This convention was adopted by the community, and the second
edition became Programming Perl, using the capitalized version of the
name to refer to the language.
You may or may not choose to follow this usage. For example,
parallelism means "awk and perl" and "Python and Perl" look good, while
"awk and Perl" and "Python and perl" do not. But never write "PERL",
because perl is not an acronym, apocryphal folklore and post-facto
expansions notwithstanding.
I wonder if the Perl6 chasm tainted outsiders' perspectives on perl and the community
Larry Wall has told me that the fact that he had two acronyms that he liked for it was one of the reasons that he chose the name Perl.
Alternatively, they might just be a refugee from FORTRAN, the language which predates the invention of capital letters :)
The comments you've made about its usage do not address the comment from the article that I highlighted. It's becoming increasingly difficult to find senior engineers with a lot of Perl experience. I think it's unquestionable that there is a stronger desire in the market for Python and Ruby. Just consider Ycombinator. In my batch of 66, there were only 2 companies (mine included) which used Perl in any way. The vast majority used Python or Ruby. Does this not strike you as an important trend? This is not a rhetorical question, I mean it sincerely, since you also seem to be familiar with the language and love it too.
Correcting someone on a pedantic issue outside their main point IS considered if not an attack, then surely quite rude, in regular conversation.
Even more so when you add: "People who refer to it in all-caps instantly give themselves away as outsiders to the language" trying to expose them as some kind of wannabe poser.
What the fuck does this guy knows about the parent's history and expertise in Perl?
I thought that the rest of avar's comment was a reasoned response to the claim that perl is "fading, however slowly, into obsolescence".
p.s. He is also a former Google Engineer who was part of the small team that built GOOGLE NEWS. http://dealupa.com/about
Something that may cause confusion is "acceleration" vs "velocity". Languages like Ruby are still accelerating, Go is accelerating a lot, even Javascript is accelerating from an already massive user base. Languages like Perl and Tcl are still widely used, but their adaption has peaked and is not accelerating.
this is entirely incorrect. They've been running with a plethora of different languages for a while now: Scala in particular, but theres a lot of work in C for HFT stuff.
That's what "dead" in casual discussion means though.
Because in the stricter sense that "someone, somewhere is using it", even SNOBOL is not dead.
Are you suggesting that when Mats created Ruby, he was thinking to himself, "I can't wait for the all the first graders to grow up and use my language?"
I rather think that the creators of computer languages think in terms of present, rather than future computer programmers, and not in terms of "My language will make all the Beliebers want to become software engineers."
Or some people just don't fucking care about "standard spellings" and prefer their own.
If you look at a old JAVA/Java logo it actually is stylised as JAVA.
In fact my first JAVA textbook stylised it in all caps also so ever since I just copied that.
Ps. I dont actually give a shit about JAVA or what anyone thinks it says about my skills because of the way I write it
I think this is important, and worth spelling out. The entire pool of programmers is much larger, and while C may not have as much of a share of the total market, the total number of C developers is still probably larger than in the 80s. In that respect, it's hard to say C itself is in decline, just it's position of dominance.
I'm not sure the same can be said of Perl, as there has been a fairly common theme of former perlers moving on to Ruby or Python, but then again I've heard stories about people returning to Perl after a while due to the Perl renaissance (which is why you'll get a much different opinion from someone active in the community than outside; inside we see a lot of cool stuff happening).
high-traffic websites that use Perl extensively include Amazon.com, bbc.co.uk, Priceline.com, Craigslist, IMDb, LiveJournal, Slashdot and Ticketmaster.
One thing to know is that Manwin (youporn, pornhub, etc parent company) is located in Montreal. For a reason I can't really explain, you mostly find PHP and C# developers here, so most Montreal startups are PHP based.
I also interviewed a couple Manwin alumnis, and they told me that they were a couple hundred developers and that there is a huge turnover (you often see a 6 month / 1 year experience at Manwin when you read a PHP dev resume).
So it's easy to understand that they prefer to use the local dominant language, it do not mean that Perl is globally dying, just that they have specific constraints. And to be fair Perl is not the only small community in MTL, Ruby and Python also have a strangely small community here.
Same here in Vancouver; I always figured the C# part was just our proximity to Redmond (and then PHP because it's the only language that doesn't impinge upon C#'s niche) but I guess it's more of a Canada-wide thing. Weird.
tl;dr Perl devs seem to be in short supply country wide, along with jobs that would need their skillset.
When I worked in online education in the .com 1.0 days, a lot of the big tools were built on Perl (both Blackboard and WebCT I think). I'm sure some of them still use it.
At the very least, I know EMC and Qualcomm both rely heavily on Perl internally. At EMC there has been some shift to Python, but Perl is still the clear winner. Startups are more likely to use the new shiny, but Perl is still kicking in the enterprise.
I think in general its because my friends who work at start-ups doesn't really stop gushing about how MongoDB or Go or Redis is amazing and fixed all their problems, whereas porn sites/porn industry in general is talked about far less.
1) Extremely high levels of traffic. Furthermore, when it's a video site like YP, you've got people engaged and watching videos for a longer time.
2) Common cloud infrastructure is normally off limits. Running adult content from AWS or Linode is forbidden by the TOS, so you have to roll your own servers.
The guy in the interview, Eric Pickup, has a very good talk about his software stack and the amount of traffic that YP gets. https://www.youtube.com/watch?v=RlkCdM_f3p4
Of course, the photo might have some subtle implications. For instance, where's the guys left hand????
Where do you get this? I've worked in the porn industry for 12 years and have ran porn sites on both Linode AND AWS... They allow it.
That being said, we do things very differently in porn than in the startup world I have come to notice. Most startups scoff PHP, whereas 99% of the porn industry is based on PHP. Not to mention we handle HUGE traffic loads on single servers, and the only time we ever have more than one server (outside of the sites sub 1,000 Alexa) is just for media such as pictures and videos, which are only offloaded to handle as static content while the rest is dynamic.
Did you guys use any of the crazy file systems, like MogileFS?
Only the vocal ones. I find that PHP is used a lot more than what the chatter on HN might lead one to think.
My main problem with PHP is that I'm primarily a Perl dev (but have done some heavy PHP work in the past), and it's syntax is just close enough to make me always aware of how unintuitive it is in comparison, and all the extra steps I have to take to accomplish something simple. It's maddening.
Disclaimer: I wrote the above article and work for RedisToGo. But that article is unbiased :)
If on the other hand by 'query' you mean, for example, using operations like diff, intersect, and union on sets that store keys (for the purpose of knowing what data to pull) -- I actually haven't ran into performance problems yet. If what you're saying is "it's important to spend time to think about HOW you're going to be determining what data to access", then yeah, I'll completely agree that this is crucial. However, I will also point out that this will be true even if you're using a relational database -- though those are expected to be "queried" so perhaps designing a feasible solution is a bit less demanding.
I find myself continuing to use it for those things just to keep my redis config cleaner and optimized for the things redis is (seemingly) better for like persistent storage and message queues.
I'm now using Redis clusters to distribute cache reads so that each webserver has its own local copy of the cache which it reads from, but there's a single master that accepts writes. This does occassionally mean a double-write, but since we're talking about cache data and redis is atomic, this isn't a problem, and it splits the load of my reads, so that I don't end up bottlenecking if the master is being slow.
I read this paragraph and was a bit baffled. It sounds like they had merely been toying around with Varnish. I would think that heavy edge-level caching was essential to keep that kind of site up. Considering most content probably isn't personalised, it's basically a matter of serving a lot of static content fast.
In Summary: we took a perl site, changed it to php and implemented Redis because it is good at lists. I don't really have any specific tips on Redis or any details of clever implementation approaches.