Comment by rbranson | Hacker News Reader

Comment by rbranson | Hacker News Reader

Parent Full thread

The real crux of the issue isn't that SQL won't scale, it's that it won't scale efficiently or effectively. SQL scales, but you need expensive, exotic hardware (InfiniBand switches, SAN storage) that requires niche engineering skills just to get up and going. By the time a startup-type company purchases all of this hardware and has figured out how to use it, the window of opportunity will have closed. In addition, you're then pretty much locked into support from the vendor for performance issues. With solutions like Cassandra, you've got the source and can solve your own problems, which, to a company rich in software engineering talent, is exactly what they want.

Even if you look past all that, RAC and ASE will run into limits as they're still relational, which is not a truly scalable approach to data modeling. There are real limits to their scalability as well. Oracle RAC will only scale to 100 nodes. That's a lot of dough for a 100 node limit.

So there it is. SQL doesn't scale (effectively or efficiently).

"So there it is. SQL doesn't scale (effectively or efficiently)"

I think you just re-iterated the author's reason for writing perfectly.

Instead of saying 'SQL' (the query language) doesn't scale, perhaps you can say full ACID doesn't scale cheaply.

We spent a bunch of time looking at large scale distribution at Berkeley for the Mariposa project. The short version: if you want to break ACID then you have to take the semantics of the data into consideration. Transactional financial data needs full ACID, many other types of data have looser needs (status updates, etc) and those looser needs can be taken advantage of.

As it turns out, even common DB's like Oracle, Postgres and MySQL can be used differently to get much cheaper scale for certain types of data. See the FriendFeed article on MySQL as a docdb (also the Friendly ruby mapping on github).

To paraphrase the author:

"ACID scales just fine if you need it (and if you need it you most likely have budget for it). If you don't need full ACID, there are plenty of other cost effective solutions you can run on low bandwidth disks/memory limited CPUs. Mix and match to fit your needs.

Please, enough with the religion."

You're arguing semantics. NoSQL = No"ACID-Compliant Relational Database with Normalized Table Design"

Uh, that is a bit of a leap...

It seems like every NEXT BIG THING comes in two parts.

1) The provocation: "Throw away design and do everything with testing" - "never use SQL again, just key-values" "Objects are Worthless!"

2) The sensible fall-back position you can use when challenged "Test are useful" - "You might not need full ACID", "We can benefit from shorter inheritance trees"...

I suppose every new idea needs a way to make a splash...

This is what as the triad of thesis, antithesis and synthesis. A lot of progress works this way. [http://en.wikipedia.org/wiki/Thesis,_antithesis,_synthesis]

Thesis (established view): SQL/ACID solves all you data problems

Antithesis: Nobody needs SQL/ACID, let's throw it all out

Synthesis: By carefully considering data integrity constraints we can find a more optimal data management solution for a particular problem (alternatively: let's go to the pub!)

Excellent plan. Alcohol always helps people get over arguments.

Bones heal faster than internet battles :)

This is simply not true. I work for a major internet company that has millions of users. We use SQL for all of our data (images and SWFs excepted, of course). We do all logging to SQL tables. We use key-value cache servers for hot, non-relational data, but we don't have a lot of that.

We built most of our data center with cheap hardware. It is quite possible that we splurged on a load balancer and some other funky networking gadgets; I don't know. We did, however build quite a lot of custom software to abstract away the fact that we have split our data among dozens of machines. This is essentially the work of two people (one hardware, one software). Finding the people who can build such a system is difficult, to put it mildly, but once you find them, you don't need deep pockets.

Reddit is another example of a site that runs on SQL. Part of the problem with the "SQL won't scale" argument is that essentially all of the live queries function like key-value lookups (or lookups against a very small number of rows). If you're tempted to do something insane like service a request with a join, then you're doing it wrong and you need to cache.

Lookups by index will have the same performance characteristics no matter what. It doesn't matter if you're using a Python dict, an index in MySQL, or a key-value store.

THEY ARE ALL O(1).

> So there it is. SQL doesn't scale (effectively or efficiently).

You might just want to say "cheaply".

Reply on news.ycombinator.com