Google App Engine to support SQL | Hacker News Reader

Google App Engine to support SQL

groups.google.com

120 pointsscorpion03232 comments

Reading between the lines, this shows that App Engine is a growing & responsive project, and makes me less nervous about building on it.

I'm trying to feel that way too... but the fact is there are people on the app engine team who could write SQL support for app engine in a few days in their spare time.

The problems with app engine and the datastore have shown that there was a high level misunderstanding underlying the core design of the datastore... and how it would function in the context of lots of simultaneous users.

So I'd feel a lot more confident if Google addressed this concern other than with the high availability datastore...

Look at this from the perspective of Google Apps corporate users. Imagine that the IT intern has just completed the corporate cafeteria's lunch menu system, and it's time to deploy it. While it's then least significant application the IT department manages, there's still a high fixed labor cost for infrastructure roll-out and management. The DBA must provision instances across dev, QA, and production environments. Onto which servers will the application be deployed? Must they be procured? Even if the lunch menu app is deployed on a VM instance, that OS image still requires security patches and other upgrades. The cost of deploying the intern's work likely exceeds his/her salary for the summer.

With SQL on AppEngine, corporate developers can build apps like this without learning how to use NoSQL for persistence. Even if GAE's SQL doesn't allow scale-out sufficient for SaaS vendors and such, it will fill an important niche in Google's apparent goal of moving corps onto their cloud. As another commenter here suggested, the bar for perf/scale in many cases may be no higher than that provided by a MSAccess DB hosted on a Windows fileshare.

We are living weird times. Multitasking, copy/paste, SQL support, these things are old, but still newsworthy when added to new platforms/services.

There is a difference between humans speaking and monkeys speaking. For hundreds of thousands (if not millions) of people that use the datastore API, this is a big thing.

There is still no reasonable competitor to SQL. It's an entrenched standard, that NoSQL doesn't completely replace.

Luckily, it's an open standard (like email) and adding in support for an open, strongly-desired, useful stamdard as a feature should be applauded.

SQL isn't special.

Scalable SQL is.

This could help out those who are building projects using Django &/or Django-nonrel.

Kind of kills one of the main reasons for the nonrel branch to exist, no? Wasn't django-nonrel an outgrowth of efforts to put django on appengine?

I think that the nonrel project has utility on other places (redis, mongo, etc...) but ABP (all buttons pressed, creators of nonrel) was pushing django on GAE alot, IIRC.

I agree. Don't forget that work has begun for a Cassandra driver as well.

Truth be told however, Django and a SQL based system aren't all that bad.

No matter what, it will be a long time before Django gets full 1.0 support for NoSql in any shape or form. That means, 100% Django coverage.

It depends on who you ask. :) We've created Django-nonrel and djangoappengine for people who need GAE's scalability or at least don't want to deal with scalability as an afterthought. Of course using the GAE datastore for everything can be really painful (and dbindexer is supposed to help in this regard - at least a little bit), but now you can have the best of both worlds and pick the right tool for the task.

I'm looking forward to testing out porting one of my Django apps to GAE, particularly since the Heroku clones for Django are not yet ready for public consumption.

So long as the performance is reasonable, this will go a long way towards removing lock-in barriers with AppEngine.

"The datastore is high-latency and extremely unreliable" - http://www-cs-students.stanford.edu/~silver/gae.html

I've found that, if you want to make responsive apps on GAE, you must use memcache. At any scale. Personally, I gave up and left for better options. The generous free quotas aren't worth the trouble.

I hear they've improved latency since I left, so take this with a grain of salt. I'm still not going back any time in the next few years, though.

Which is where this could help in the first place, by reducing the load on the datastore.

and the HN response: http://news.ycombinator.com/item?id=2264517

Hopefully this is the real deal and not an abstraction on the current datastore. When using the datastore in a relational way, you get some pretty horrid performance.

It is the real deal and it will scale, and I quote, "as well as SQL scales".

The "free scaling" using appengine datastore is nothing more than theoretical at this point. The service has severe datastore latency problems and non trivial downtime. Looks like using RDBMS, solves both these issues for now, while Google can ponder around long term solution for the bigtable access.

Member of the App Engine team here. Yes, to confirm, this is not an abstraction on the current Datastore but a separate "real-deal" service.

I rather got the feeling that the datastore was much better for certain kinds of things than SQL. However, unless you are an expert in both SQL and the use of BigTable, how would you know when to use the one over the other? Okay, so obviously SQL is 'designed' to be good at transactions (whether it is or not is an exercise for the readers), and BigTable is really good for fuzzy search.

I'm guessing that SQL is actually going to be more useful for anyone doing e-commerce (hence transactions) or anyone with familiarity with traditional SQL based persistence (even - shock horror - JDBC).

SQL is bland, BigTable is sexy. But another word for bland is mature...

Nice feature, but not super important for new projects. I suppose it will make porting some existing projects over.

I didn't remember seeing SQL on the roadmap however so I had another look ( http://code.google.com/appengine/docs/roadmap.html ) and I'm very excited to see it as been revised with a few awesome enhancements. I've had several projects I wanted to try on app engine which were impractical because of GAE limitations like no full text search, no programmatic access to blobstore, no bulk export/backup, no mapreduce. The future is looking a lot brighter for GAE.

Why isn't it important for new projects? I have a ton of projects that I can't do on GAE precisely because of this. GQL is extremely limited.

At the risk of re-hashing a number of previous discussions, GQL appears limited when compared to SQL because Google's database is designed to scale to extremely large datasets and massive numbers of concurrent users. The features that are missing were removed intentionally because they don't scale. If you can re-think your database use to be consistent with this new paradigm, then you'll have a much more scaleable app when you're done.

The end goal for every app doesn't have to be scaling on the order that GAE's original design provides. People have lived for years creating apps that use Excel or Access as a datastore.

Being able to use SQL on GAE will allow more widespread use of GAE for things besides "the next twitter".

If the first user waits 2 seconds per response and the millionth user waits 2 seconds per response, that does indeed meet one definition of scaling. The problem is you aren’t going to find a million users willing to wait that long. So why punish the first user?

The flip-side is that anyone who needs to do reporting, aggregation, etc. is going to spend a lot of time reinventing stock behaviour while their competitors focus on building new features. Those competitors are more likely to get to the point where they have the relative luxury of deciding how to handle large numbers of users because they actually have them.

It is extremely limited. I just accepted this as a result of the limitations of a distributed datastore and disqualified GAE as a candidate for any query intensive apps I want to do.

The other limitations I listed didn't seem to have architectural causes so I find them more frustrating.

I was halfway a weekend project to find out GAE can't do LIKE-statements or an equivalent thereof, only with some hacking this is sort of possible. Moved the project to RoR and got it working in no time. Another project I had couldn't be done because the datastore was to slow. I love GAE, and use it for quite some little projects but you have to know its limitations. SQL (and hopefully the speed that comes with it) are certainly a step in the right direction.

It's a very big deal for new projects, if it means Django will run out of the box. GAE is a pretty awesome platform for small projects, a clear migration path in/out will make it much more attractive for bigger projects.

Google should buy a SQL engine like Postgres now that mySQL is in oracle's hands. People will need SQL databases till the end of times, and coupled with BigTable that can be a killer in the database space.

Reply on news.ycombinator.com