Hacker News Reader

Hacker News

Comment by pajju | Hacker News Reader

Full thread

pajju

They have a very Interesting Stack as seen on Quora[1]:

-Python + heavily-modified Django at the application layer

-Tornado and (very selectively) node.js as web-servers.

-Memcached and membase/redis for object and logical-caching

-RabbitMQ as a message queue.

-Nginx, HAproxy and Varnish for static-delivery and load-balancing.

-Persistent data storage using MySQL.

-MrJob on EMR for map-reduce.

[1] http://www.quora.com/Pinterest/What-technologies-were-used-t...

View on HN

Pardon my ignorance: what is interesting about Pinterest stack?

I think by now most startups, YC or not, in Silicon Valley will pretty much have a similar setup:

1) Choose the main web-stack (Rails or Django)

2) Choose the API framework (node.js, or something else)

3) Memcached for caching (or some NoSQL)

4) A message queue (ZeroMQ, RabbitMQ, or something ...)

5) nginx, HAProxy, Varnish, (or similar technology)

6) Hadoop for map-reduce (only if you need to go that path...)

Not trying to be a nit-pick, but wanted to mention that ZeroMQ is not a message queue in the sense that RabbitMQ (and Kestrel, Beanstalkd, SQS, etc) is. It's a socket library that you'd use to implement those technologies.

And while this is truly unimportant, my experience with bay area startup stacks is that Redis has very thoroughly supplanted Memcached for most use cases. Though most still have a memcached cluster or two.

That being said: 90% of software development in this world isn't done at tech companies and startups. So from that perspective, that stack is interesting.

Can you elaborate why Redis has supplanted Memcached? Is it because Redis occupies a space in between Memcached and MemcacheDB? (in the sense of Redis can be both in-memory and/or persisted on disk?)

(Not that I care about starting the holy war of X vs Y, just out of curiosity why)

You have a point regarding your last statement. Perhaps I've been here far too long to notice more of the 10% and less on the 90% :)

... which is true ... last I read someone is making almost $1M revenue developing Windows desktop app in 2012 and another company is making $60k/monthly revenue developing BB apps using in-apps Ads. I think I'm going to sign-off on HN and go to the other side... :D :D :D

Redis is far more feature-ful, and can serve the purpose as memcache. Why have two parts when you only need one?

Redis offers persistence, set operations, namespacing, you can delete multiple entries with wildcards...

I agree. I think the interesting aspect is that they built a service that serves an impressive number of users on fairly common technologies.

There isn't anything very revolutionary about their stack by itself. FB didn't buy them for their technology... they could have replicated that on their own.

If they want to be talking about infrastructure, the story of using common tools at high volume is more interesting that just what tools...

Oh, I wish it was that simple.

You can't mix and match (you can theoretically)

Let's say you began with Django, right? Or there was something already ready in Django.

And then you begin to see the warts. But ok, you keep churning along.

And the more you churn the more of a specialist in the deficiencies of each technology you become. Like the fact that Django's ORM runs like a dog.

Or see all the "we moved to MongoDB and we regret it" discussions

From that list, there are two technologies I would recommend strongly (if you know what you are doing): nginx and Redis

The rest needs to be handled with care. And Redis is very powerful if you know how to use, but it's easy to go the "lazy way" with it, and it will be fast, but not as fast as it can.

Yes, but the Django ORM helped them launch faster. There is nothing wrong using it to start with and then writing your own SQL queries when you need to scale or even changing your data model to accommodate scaling.

I agree 100% (even though I don't know what or if they changed the Django ORM - most likely yes)

Use the tools to launch faster, then optimize.

My company is going through exactly this scenario right now. We launched on and use Rails for most stuff, but there are a couple of important cases where we have to do direct sql.

Though I've had to work around some of Rails defaults for my use cases, there is no doubt that without it we'd not have been able to launch and iterate so quickly.

I hate the Django ORM and I don't know why Django doesn't just deprecate it and switch to SQLAlchemy as the new default, which is a much, much better ORM and used primarily by non-Django Python apps.

Why don't YOU try?

Why don't I try what? Deprecating the Django ORM? I've already done so on a personal level (in fact, I avoid Django wherever possible), but since I don't control the Django project, I can't deprecate it for everyone, as I suggested in the parent comment. :)

Excuse me but couldn't they just optimize the way they use Django ORM?

... as in: check the output of SQL queries constructed by Django ORM and tweaked it in your Django Model layer such that the resulted SQL query will be tuned for performance?

Or at the very least, shouldn't Django ORM at least provide a way to retrieve data with "filtering" as optional parameters? (get only a few columns, but not all..)

Let me tell you something

If you're reading "just tables" without, or with simple relations and conditions, it's ok (because, as you pointed out, that's basically SELECT with)

The problem is when for example you have inheritance, or moderately complex relations.

Then you'll see Django spamming your DB with requests and the DB may be fast but the operation will be slow because of the sheer number of requests.

So yeah, in a sense you can optimize for it, or use SqlAlchemy (which is good, but it has a different philosophy than the Django ORM in the way it's used)

Inheritance is a different beast that ... well, let's just say that it may creates and endless debate whether one should or should not have inheritance in the model...

My experience is heavily in Java (JPA2 and Hibernate) and less of Rails (simple/moderate apps). Typically an ORM provides a way to handle custom queries but still help maps the result to the Java object model (which helps a lot vs writing getter/setter :)).

I don't know how people down south in Silicon Valley design their model (especially young scrappy startup) but I have a slightest doubt that they go that far (advanced/complex). It's not like startup start with "analyzing requirements" and move on to "modelling session" :) => they just go ahead and write code.

So in theory, the data model shouldn't be complex enough to warrant edge-case.

I didn't say that it is simple to come up with such setup. But whenever I see such list, it reminds me of some sort of blueprint architecture (or template) that I've seen somewhere else.

Seems like a sensible stack.

However, is it not reasonable that would someone not consider node.js, as web stack as well?

I will agree the focus is more towards backend and asynchronous API's traditionally, but the amount of front-end/UX orientated modules are growing every day.

Sure why not. Some people don't even mind the lack of polished modules and instead spend time to write their own.

Not a big fan of node.js (or JavaScript for that matter) and probably never will so I can't comment much.

I would think that Nginx + Django + SQL, and the rest is premature optization...

Even simpler : Django + SQL in Heroku.

Reply on news.ycombinator.com