Ask HN: Scalable framework for web apps? | Hacker News Reader

Ask HN: Scalable framework for web apps?

25 pointsSingAlong33 comments

I'm planning to build on my new idea and would like to know which web framework is scalable to build a web app on. My choices of languages are PHP, Python and Ruby. I wouldn't mind a long learning curve.

I have worked on CakePHP for a couple of projects and tried Django on my localhost. But would like to know if you guys find anything else that's interesting and better. How about RoR?

View on HN

I'll give my top recommendation for each language you named. I have used all 3 of these with great success. I have also used competitors for each of them.

* PHP - Kohana

* Python - Django

* Ruby - Merb

I used many PHP frameworks a while back and eventually settled on CodeIgniter (which is very good). After Kohana spun off from CI, it began to gain advantages and has continued to.

I've used Pylons and TurboGears before settling on Django. It's quite fast, and can be made faster when paired with psyco.

I used Rails for a long time before approaching Merb. I have followed its development since shortly after the pastie that Ezra made of the initial code. Merb is making a lot of great choices and is and will be my framework of choice.

I don't know why but I just like using variety of languages. I enjoy the learning different stuff. After 3 years with PHP, I developed an app for a contest a while ago on Google App Engine using web.py. That was my first shot with developing a python web app. I'll give Django a try later. But since my web app deals with developing for plugins go blog engines, I guess I have to deal with some Python blog engines too.

Now since you mentioned Merb, I'm going with Merb. Just tried Ruby online on http://tryruby.hobix.com Seemed cool, so I'm sticking with Ruby this time.

Merb seems to have some problem with Windows XP. I get an error while installing it. But it's just the docs that generate the error. The main package installed. merb 1.0.3

P.S: Does trying out too many languages often make me a bad programmer? Should I stick to one stack of technology and learn new stuff as the trend changes or do I continue to do what I love to do?

IMHO, knowledge of multiple languages is usually a sign of a _good_ programmer. Keep at it.

make sure to learn the canonical style of the languages, not just the syntax and grammar. it is the difference in attitudes and approaches that yields insights that will carry with you from one situation to the next.

I too think it is important to work with a variety of languages, but it is even more important to be good with at least one. Basically, don't spread yourself too thin.

Coolio!

Wow! I've now got Merb on my http://localhost:4000

I installed Ruby On Rails too. Will see which one of them I'm gonna love.

Now it's playtime to get some code of my own.

I also read a lot of reviews about merb. All being great. Incase if someone else wants to know why merb is good for web apps that expect to handle a lot of processing and traffic even for a small amount of users. It seems that merb has a smaller core called the merb-core while RoR has a slightly bigger core. While RoR has some features built-in merb has those too has plugins/helpers. So I don't think there's much difference as along as you want your app to be a bit faster on the processing side.

As you mentioned, the main difference (to me, at least) is agnosticism and hackability. Merb forces almost nothing on you in terms of the framework stack. Also, the core of merb is much more approachable for people wanting to modify it for their own needs.

Thanks for pointing out Kohana! I was looking at PHP MVC Frameworks, and really liked CodeIgniter, but wasn't convinced about the PHP4 thing. Kohana seems to be what I was looking for.

Scalability is the last thing you need to be worried about. Worry about the business model. If you get the business model right, and you have scalability problems (i.e. "Oh shucks there are too many people trying to pay me money!"), then you can just pay to get them resolved.

Just make sure you hire the right people. This appears to have been Twitter's problem, and it took them awhile to fix it.

Experience counts.

Lack of scalability will nag at you and may prevent from making some choices. My current product is paid, and I have contemplated making it free while trying to catch the revenue off the ads, but given that scaling the web site will take work now whereas the ad payback may or may not come later I chose not to do this.

"... Lack of scalability will nag at you and may prevent from making some choices ..."

To the point where you have trouble getting more users. MySpace is the classic study here. How do they scale all those windows servers? ~ http://www.baselinemag.com/index2.php?option=content&tas...

This is only a problem if your business model is based around selling poorly performing lower-than-remnant-inventory CPM advertising to users who are largely a) poor and b) disdainful of spending money except on iPods, which famously do not need CPM advertising to reach your users.

Not to toot my own horn here, but my (very, very boring) website makes about ~$80 for 1,000 pageviews. It is built on Rails. Supposing I totally screwed the pooch and it was structurally incapable of doing more than 100k page views a day without a top-to-bottom rewrite.

Suppose I got to that $8,000 revenues a day and figured "You know what? This is too darn constraining. I WANT MORE, BWAHAHAHA". I'd be able to hire whomever I pleased to do the rewrite (pay $5k a day for my own ops team, who cares, I can afford it). And it would be vastly easier scaling my site than Myspace/Twitter can scale theirs, because I would not have to set world engineering records to do so.

The first response any engineer is going to make is "Erm, you should probably think of putting multiple web servers in front of a beefy database rather than running both web server and database out of a machine with 256 MB of RAM." This is a pretty easy engineering problem compared with "OK, innovate technologically to be able to handle data on a scale that 5 years ago would have required a major multi-national corporation or minor nation state to generate."

Then I'd be back to the boring-as-dust business of, you know, providing features to convince my customers to continue paying me money.

I modded you up because I agree with the spirit of what you're saying, but I think you're being a bit unfair.

Assuming your $80 CPM site is the one in your profile (if it is, what does it use Rails for? edit: nevermind, just read your blog post about your cron bug :p), then it's not very surprising, since it offers a product for download. What would happen if your product made more sense as an online service? If you charged the same price and got the same number of sales, how much would it make per 1,000 pageviews? It would still be a lot more than advertising revenue, but nowhere near $80 CPM.

That may be right, but if you build expecting the scalability problem AND you get masses of users, you can swim in the money instead of using it to solve the problem.

McScrooge approves this message.

This is true at some point but can be dangerous if you neglect it entirely. I have worked as server admin as a part-timer in a company that stalled because they don't use grid technology for their shared plan right from beginning. We have clients complaining the server response and lots of problems with I/O etc. Converting this 500~ shared servers into grid system is painful to do when we have limited time and resources. But, this problem would less occur if we have chosen a slightly expensive but reliable for the long run grid system or the like.

>Scalability is the last thing you need to be worried about.

That's what friendster thought too.

Friendster became very popular but couldn't handle the scalability part and ended up dying a slow and painful death...

+1 for Pylons. We're currently serving around 20 million dynamic requests a day via pylons from a handful of servers; pylons probably won't be your bottleneck.

What I like about it: python, promotes good code organization, relatively small code base (you can just read the source code if you're not sure how something works), and for the most part it stays out of your way.

If you're looking for flexibility, give Pylons a shot. It's often paired with SQLAlchemy, which is an amazing ORM that can be configured to do a lot of things which will make scaling easier in the long run. (Whereas Django doesn't have multiple database support, for instance; SQLAlchemy can even handle sharded data transparently.)

Scalability isn't a magic pixie dust you find in some boxes of cereals and not others.

Scalability is extremely application specific. As your application grows, what will be the bottleneck? Maybe it's page-serving performance. Maybe it's the database. Maybe it's disk storage. Maybe it's synchronization of data. Maybe it's raw bandwidth. With no specific knowledge of your constraints, any specific advice is worthless and potentially harmful.

The question you need to ask yourself is not "which framework should I use?" but "how should I architect my application to ensure that the most likely bottlenecks can be resolved simply when I get there?"

If your likely bottleneck is bandwidth, for example, building your application with eventual use of a CDN in mind is probably very smart. If your likely bottleneck is disk storage, make sure you choose a host/infrastructure that allows you to scale that up practically indefinitely. If your likely bottleneck is page rendering performance, make sure you write very clean, unoptimised code, so that when the bottlenecks show up it's a simple matter to boost them without impacting the rest of the application.

"Scalability isn't a magic pixie dust you find in some boxes of cereals and not others. Scalability is extremely application specific."

Absolutely correct. I have an app that is not scalable in most of the traditional ways because one live memory based database needs to be fast, have very large capacity, and be instantly accessible to all users concurrently. This appears to require a custom coded C program whereas the front-end web servers are a minuscule issue in comparison.

If you really want to play... you could use erlang

Nitrogen - http://github.com/rklophaus/nitrogen/wikis <- quite new

Erlang web - http://www.erlang-web.org/ <- serious enterprisey guys.

Erlyweb - http://erlyweb.org/ <- ROR for Erlang

And one more for the scalability fetish - http://www.metabrew.com/article/a-million-user-comet-applica...

You are focussing too much on the language and scalability rather than focusing on just building something.

Every major language/framework out there has been used and scaled for large websites. Pick one that you would like to learn, provides you with the flexibility you are looking for, and run with it.

Worrying about how you will scale an app that you have not built yet will be a major drawback. Also keep in mind performance and scalability as you're building the site, but do not spend much time worrying about it before hand. Just take an idea and run with it, tweak the app to scale as you go.

Hi,

I just blogged about the issue of choosing the right software stack to build your web app a few days ago. You can read it here : http://thibauld.com/2008/11/web-application-implementation-s...

And I'm currently writing the next post about code architecture... I'll probably post it today. Cheers,

Thibauld

As much as I like to push the envelope, I'm going to go out on a leg here and suggest Ruby on Rails. I love it, a lot of people love it, that makes it easy to learn and deploy.

Also with Passenger, it scales out just as easily as any framework or language out there. And if you need to jump to multiple servers, you ought to be making enough money to support Engine Yard which solves your scaling problems for you.

You might want to revise your choice of languages. Common Lisp is as expressive as Python or Ruby, but many times faster. Haskell and OCaml are even better.

Clojure is another option; theoretically you have all of Java's web frameworks, and Java is also faster than Python or Ruby. I can't recommend it, since I haven't used it yet, though. (I don't want to be a Clojure fanboi ;)

If you want a framework with scalability "baked in", so to speak, you should take a look at GigaSpaces XAP (http://www.gigaspaces.com/). Its not the same model as the others mentioned on this thread but will prevent that "big rewrite" in the event that you do indeed become successful enough to need it.

Scalability has nothing to do with your choice of framework or language. In terms of scalability, Cake, Django, and RoR are more or less equivalent (the first two are RoR clones after all).

I disagree that scalability has nothing to do with choice of framework. It happens that none of these frameworks address common scaling issues to a great extent, but that doesn't mean a framework could not do so.

Regarding the "RoR clones" claim, citation, please. Django was publicly released very shortly after RoR, having been developed privately for a couple years. I don't know about CakePHP.

PHP - CodeIgniter

If you absolutely need scalability and don't mind a Google lock-in, check out Google App Engine. It can run Django, app-engine-patch makes this extremely easy.

Yup! I've tried it. But they don't yet have a payment system. Their product roadmap says that the pay-as-you-go system will kick-in sometime in March 09.

EDIT: AppEngine product roadmap: http://code.google.com/appengine/docs/roadmap.html

Due to this I'll have to stick with some host like Slicehost or Linode for now and then maybe switch later when the requirements increase.

Google reps hinted on the appengine user group that they are very likely to introduce some kind of payment system this year.

In the mean time, you can ask them for a quota increase if your app reaches the limit.

Reply on news.ycombinator.com