Foursquare.com & Scala/Lift [slides] | Hacker News Reader

Foursquare.com & Scala/Lift [slides]

docs.google.com

44 pointsTrevorBurnham17 comments

This is a good example of what I call 'Guitar Center development' - a disproportionate fascination with tools at the expense of the actual thing you're building.

Agreed! In fact, it seems an obsession with obfuscation and esoteric syntax. When he gets to any real principles of software engineering (scale, reliability, etc.) he admittedly falls short. Cool, though, eh?

I saw the talk in NYC. From the questions asked, I'm under the impression the system scales quite well. According to Crunchbase, they now have about 400k users.

http://www.crunchbase.com/company/foursquare

@idelwords - It's worth noting that this talk was given at the Bay Area Sala Enthusiasts Meetup hence the focus on the tool (scala) and not the product.

@wavesplash - It's true that there are potential scaling issues when you have stateful servers, but the vast majority of our requests are API requests which do not have this issue significantly mitigating the risk.

-harryh

Why not directly reply to them? It's less likely they'll read this here.

We read it here, and I really appreciate that @harryh is engaging in the dialogue.

Ok, I didn't mean for my comment to seem a bit snarky, which I think it did. He made some valid points and I just wanted to see the discussion continue.

It's not the focus of the talk that I take issue with, but that the choice of tools (fun, new, poorly tested) seems to have been made without regard to the needs of the product.

Personally, I think it's better to innovate at the product layer and use the most boring technologies you can get away with below that.

Sometimes there are sound reasons to go with a new technical approach, but by your own admission this was not the case here.

And so the classic symptoms appear - full rewrite, unexpected delays, interoperability problems in the new layer requiring a change in an unrelated part of the app (mysql to postgres migration in this case), hacking on an ORM, the joys of being the first people to discover a new class of bugs at scale.

All of this effort could have been more productively channeled into making the product unique and awesome, however boring the backend.

Unfortunately the rewrite was necessary no matter what tools were used (even if we had stuck with PHP). The existing codebase was bad. Very very bad. And, all things considered 3-4 months for a full rewrite was pretty good.

Whatever downsides there are to the scala/lift decision we have not experienced them yet. Though certainly we may run into them in the future.

If you want to save some clicking:

- Author picks Lift/Scala not on the best future interests of the company (ease of hiring, scalability, agility, maturity) but on fit for his world view about type safety and it would be fun for him.

- Author commits startup to stateful servers / scaling issues from day 1 of the rewrite.

that's not how i read it at all. he's experienced with java, prefers statically typed languages, and has had a lot of success so far with scala. for accusing someone of a heavy bias you're doing a pretty good job yourself

That's a very narrow definition of success.

There are at least 3 areas of technical focus at a startup after you reach product/market fit: 1) ease of feature development 2) ease of hiring 3) ease of scale under load

The author sacrificed 2, needlessly created well-known scale issue that could be avoided in 3 (basic distributed systems class), for no substantial improvement over the other options for 1.

Yes, I have a bias towards startup team members factoring in their future impact on their coworkers and the company.

If you only pick tools that everyone else knows so your hiring pool is large, then new tools never stand a chance of being used. You would stagnate all progress in the industry!

It's funny because when Paul Graham writes articles about how the web lets us use whatever language we want (see http://www.paulgraham.com/avg.html), everyone celebrates the possibilities of writing their web site in their favorite lisp dialect. Yet here we have this evil engineer who picks a technology he likes, successfully improves a product and meets business requirements with it, and is ridiculed for picking a relatively obscure technology!

Check out tiobe. I wouldn't be surprised if in a year Scala passes the combination of all lisp/scheme dialects in the rankings.

Startups aren't about language diversity for the sake of diversity, they're about unfair leverage. In this case the author made a choice that didn't add any leverage to Foursquare. If anything, the author admits he boxed the company into a corner because of a rookie scaling decision.

In contrast, the Asana guys are writing their own language+framework and if it delivers on it's early promise they'll be outgunning the competition with a 10x improvement on development time and code size.

http://asana.com/luna

That's the sort of advantage Paul Graham had by using Lisp in ViaWeb vs. his competition that was mostly writing c++.

My criticism wasn't of your comment that Lift was a poor choice because it doesn't scale well. It was of your comments that the technology shouldn't be used because there aren't that many developers to hire that know/are willing to learn scala/lift or that it isn't as mature as some other technologies.

As for your argument on scalability, while I have dabbled in Lift, I am too ignorant to know how much work it would take to scale Lift versus alternative technologies. I find it hard to believe that choosing Lift is really going to make it that hard to scale. David Pollak and the creators of Lift seem like they are pretty smart people to me. However, I have no evidence to prove you wrong and therefore did not even address that point.

I don't believe I made that argument.

Here's my point as a startup conjecture:

"Sacrifice the network effects of established languages/frameworks only when it gives your startup an unfair (10x) advantage."

It will be interesting seeing how they deal with stateful servers as they continue to grow.

Does anyone know if they are currently doing session pinning?

Reply on news.ycombinator.com