Backend-as-a-service for web apps? | Hacker News Reader

Backend-as-a-service for web apps?

davewasmer.tumblr.com

80 pointsdavewasmer63 comments

Pedantically, RESTful APIs do not maintain session state on the server. This is important for scalability. (Whether "pure" REST is appropriate for a given API, or will be adopted if it is, is another question.)

If too much is done on the client-side, it loses some of the benefits of webapps, such as easier to pirate the working code; so the competitive advantage needs to be in the data (or in the server side processing) rather than the client app itself.

EDIT There's a general problem with RESTful APIs: they are fixed and can't be adapted to every use. You might end up doing a lot of transformation on the client side to assemble the info you want, and it might take several calls, causing latency. What's needed is something like relations and SQL, so you can extract the data in whatever structure you need, taking into account latency (assembled on the server, the message short though including oft-needed related data, and minimizing slow network trips). An obvious solution is for the server to just accept SQL directly - i.e. for that to be your RESTful API.

This doesn't sound right - it's certainly not a conventional RESTful API - but what exactly is the problem with it?

I think we don't have it yet, because there isn't a need for it in these early days - because, so far, it's common for one server to be used by just one client application, so that they are specifically customized for that one use-case. But what will happen when several different client applications want to use the same server? Secondly, these APIs are new, and there has been little time for them to change much, so the problem of compatibility with old clients hasn't arisen much.

SQL solves both these problems, of different application and of changes, but these problem don't exist so far. And I guess it's not certain that they will exist - if clients are always silently upgraded; if for a second app you just create a second RESTful API for it (so the integration occurs further back, at the database, instead of at the API).

> There's a general problem with RESTful APIs: they are fixed and can't be adapted to every use. You might end up doing a lot of transformation on the client side to assemble the info you want, and it might take several calls, causing latency

This isn't REST as such, in fact you have the same issue with any service API.

Solutions include creating composite resources, see Subbu's (excellent) REST/HTTP book which discusses it.

Its also worth noting that more granular resources can have advantages particularly when mutability is considered. So if you combine mutable data with immutable data, and in addition the mutable data changes to very different schedules, then taking advantage of HTTP caching becomes more difficult. So if you create more granular resources but some of them are very cacheable then you can actually in some cases get less latency.

> SQL solves both these problems, of different application and of changes, but these problem don't exist so far.

I haven't had time to play with it but ql.io is designed to address these issues and might be worth a look:

http://www.infoq.com/news/2011/11/ql-io-release

Nailed it. Composite resources and good use of caching with fine-grained resources can go a long way.

HTTP is a pretty sophisticated protocol, when all is said and done. That means there is a learning curve in using it effectively. However, a good BaaS takes care of that for you, including providing default client libraries that know how to take advantage of the API properly.

(Full-disclosure: my company, spire.io, has recently launched a REST API BaaS that attempts to address exactly these concerns.)

I don't think I was clear enough on that point - I didn't mean that the session state itself would be accessed as a REST resource.

I imagine the session state would consist of client side Javascript (since this service would be ideal for the type of single page apps served well by Backbone.js and the like). The javascript would probably submit time-expiring authentication tokens with each request. That way, session could be persisted client side, while maintaining permissioned access to the CRUD actions on the service side.

EDIT (to respond to 6ren's edit):

I agree that, in a production environment, the flexibility of the rapid prototyping could, in many cases, turn into rigidity when faced with just a RESTful API.

But not necessarily in all cases, especially when implementing a single page app style site, where you are more likely to persist data client side in the background for when you need it.

Ultimately, I think the tradeoff might be worth in some circumstances. And given the apparent success of the many mobile BaaS solutions, it seems to be often enough to capitalize on.

We are actually working on this at ChronicDB http://chronicdb.com

On 6ren's rigidity argument, relations and SQL are still not enough to guarantee you'll always be able to get the data you need; notably, when the data model changes. Backward compatibility is important and we might have they only solution that guarantees it.

> relations and SQL are still not enough to guarantee you'll always be able to get the data you need; notably, when the data model changes

What data model changes are you thinking of? Relations and SQL will cope with reorganizations/restructurings of the data, provided the information content is the same. But won't cope if the information is different (e.g. info is deleted or added).

> "Only ChronicDB can run new applications and unmodified old applications against a master database without breaking anything." [from your link]

I like your points about undoable DELETE (do modern RDB's really lack that?); but the above quote is striking because it was the stated motivation for relational databases in Codd's 1970 paper: http://www.seas.upenn.edu/~zives/03f/cis550/codd.pdf

[from abstract] Activities of users at terminals and most application programs should remain unaffected when the internal representation of data is changed and even when some aspects of the external representation are changed.

[from conclusion] In Section 1 a relational model of data is proposed as a basis for protecting users of formatted data systems from the potentially disruptive changes in data representation caused by growth in the data bank and changes in traffic.

The paper discusses derivability, and how some relations (such as those needed by old applications) can be derived from named relations using a high level language (now SQL). The idea was that you can change the database schema yet still provide the old relations (this mechanism is now called "views").

I'm not saying ChronicDB has no role, just that this one particular aspect, in isolation, doesn't seem new. But if it facilitates back-compatibility in contexts where RDB doesn't work well, it could be a big win. Back-compatibility, like integration, is always in demand.

EDIT looking at the "schema migration" example (http://chronicdb.com/system/files/images/agility_full_snap_g...), it looks like a join, with renaming, and a type change... is the new thing offered in this example that updates are bidirectional (especially for the type change)? Or is it that both old and new relations have the same name, "customers"?

You are right, this aspect is not new. More like half-baked.

I'm not sure what you mean by bidirectional. If this suggests having a separate copy for the old data and the new data then, in ChronicDB at least, no they are not bidirectional. Which is what one wants ideally: data consistency.

If you mean that updates on the old version are immediately reflected in the new version, then yes.

Basically, there's a single version of the truth, while old and new relations can have different names, for all DML operations.

Yes, the second one. So that from the old app's point of view, nothing has changed: it can create, read, update, delete as before. A single version of the truth etc.

SQL's "CREATE VIEW" does this already (note: not all views can support CRUD, e.g. if you changed an aggregate column, like COUNT, there is no clear meaning for the affect on the underlying tables. If you increased count, what rows should it add? If you decreased count, which ones should it delete? The problem with aggregates is that the mapping from the underlying tables to the view isn't 1:1, you can't run the mapping backwards to reconstruct the original).

1. How is SQL's solution "half-baked"?

2. What does ChronicDB add?

I mean, back-compatibility is valuable, but what is the problem you see with the existing solution?

1. Thank you for the link on views being updateable with some (natural) restrictions (http://news.ycombinator.com/item?id=3406952). Given this feature, SQL offers the primitives needed. Even if this feature is missing from various databases at the moment, it should be available eventually.

But updateable views are still not what you want! See below.

2. What ChronicDB adds in terms of backward compatibility is, effectively, automatic creation and management of these views for every schema change.

Rather than manually write compatibility views and rewrite/review an application to ensure it always uses these views, programmers instead access plain tables, not views. In fact ChronicDB creates no views at all!

You want to issue SQL queries that don't break. You don't want to manage views to do so. It's a subtle, yet major point. No database to date offers such automated versioning.

ChronicDB of course adds other features we wanted to have from our database (reversibility, data versioning, zero-downtime relocation) again in the spirit of saving the programmer/DBA from doing anything manually.

2. Yes, I agree, nice to have the "view" created automatically when you change schema.

Codd's idea was also that applications needn't be aware of the change (a view appears as a plain table)... but (in contrast to your approach) the views were managed by the DBA who was making the schema changes. So I think this feature was present from the very beginning, being the initial motivation for relational databases. [I looked up MySQL because that's the DB I have access to, but it's a general feature: http://en.wikipedia.org/wiki/View_(database)#Read-only_vs._u... ]

I would expect automatic versioning to be available too, perhaps only in the older and more expensive RDBs. I've come across quite a few research papers on schema evolution in relational databases - some from big RDB vendors. I don't recall any on automated versioning, but I wasn't looking for that. So this is just my guess.

However, if you're not seeing databases with automated versioning, then (even if some highend RDBs do have it) there will be customers who also aren't seeing it (or it's not applicable e.g. it's too expensive). A market for you. And if you can make it truly painless (or even just less painful), that's really something.

I don't see any problem with your other features. The one feature of mapping for back-compatibility is of interest to me, which lead me to study Codd's original paper in great detail over a few weeks. That's why I've been hassling you on this issue.

This problem has already been solved; it's called OData (http://www.odata.org/). Personally, I've been using it with Microsoft's new Web Api to great benefit.

It provides a very rich query API on top of basic http, and as you can see works naturally with REST services. I'm not aware of how other frameworks provide OData support, but some quick googling reveals that most web frameworks have odata libraries available.

It's an incredibly bad idea to allow SQL directly, for obvious reasons (would require executing user provided SQL among other things.)

I believe that having a single REST api that serves multiple front-ends (web, mobile, thick client etc) is the future as it removes the need to maintain an entirely different codebase just for a web interface. Indeed, many major websites have moved to this model.

> It's an incredibly bad idea to allow SQL directly, for obvious reasons (would require executing user provided SQL among other things.)

Could you elaborate on this? I've been thinking about providing SQL access to a data-heavy service, but I keep hearing that you never should.

(Almost) all servers have granular access-control, views can further provide limited views of the data, SQL itself is mostly declarative, which makes it possible to analyze the queries before running them, and logging and setting limits on long-running queries is standard.

Well, the first problem you'll run into is, which SQL implementation do you use? I think pretty much all discussion of the topic stops here since there are so many differences between the actual SQL standard and what the varios modern RDBMS' actually use.

The only real way past this hurdle is to create an intermediate SQL parser that uses your own interpretation of the standard, and at this point you may as well just use OData.

I'm not sure how valuable a discussion about just using the SQL implementation that comes with your RDMBS is, as doing this defeats all the effort we put into making our front ends ignorant of the underlying schema by locking the front-end into a specific SQL implementation for queries.

Seems like at least some SQL-like flexibility could be added by using custom keywords in the Accept header. There's an example of using `depth` parameter for specifying how far server should dig into related resources[1].

Surely other keywords could be added as well (e.g., specifying a set of resource fields, ordering, etc). A RESTful solution, although probably not quite elegant.

[1] http://www.stereoplex.com/blog/mobile-api-design-thinking-be...

If a "BaaS" for a web app sounds appealing, try http://github.com/adelevie/parse_resource.

It lets you use Parse in a Ruby/Rails app with a very ActiveRecord-y API. By using it, your web app gets the following for free: documented REST API, documented iOS and Android SDKs, and the guy who helped scale Scribd is your DBA.

I've been using parse_resource to make some throwaway apps, and I'm currently "dog-fooding" it for a production app. I've found that the schema-lessness and zero-db-admin makes prototyping a snap.

I'm fairly active in developing it, and would really like some feedback, feature requests, and contributors.

Parse co-founder here.

We do believe the BaaS model can extend far beyond our current mobile focus. We have a full REST API upon which many people build rich web apps, desktop apps, and much more. Third party libraries like Alan's excellent ParseResource have been great enablers.

I'd encourage everyone to give Parse a try and keep an eye out. We plan to expand our selection of officially supported SDKs soon.

Hmm that sounds intriguing. With Decal we're basically avoiding building any "features" or "modules" and instead using a totally SOA means of providing extensibility so the presence of services that provide a "plug n play" capability is something we're banking on.

The MongoHQ API is making great strides moving in this direction and as was mentioned in another post, the team is dog-fooding it so much that they are building the next version of the MongoHQ web app on top of it.

The added benefit that we see is that we are experts in managing, optimizing and scaling all of the backside data structure for you, while offering solid analytics, performance metrics and visual tools that allow you significant control over your data as well as critical insight into what is happening.

Really excited about this type of development going forward and good to see other offerings in the works!

(full disclosure: I am one of the founders of MongoHQ.)

Hello Jason, this is unrelated to the thread but since I have one of the MongoHQ founders here let me abuse a bit and ask:

I'm considering using mongo in the cloud but the problem I see is that there's no way to create a secure (encrypted) connection. Mongo doesn't support SSL and I can't create an SSH tunnel like against a server I control. I don't want my db data in the clear on Internet and I may not want to use AWS so that we're on the same 'network'.

I cannot be the first with this issue, is someone looking into adding encrypted connection to mongo? (I just see in mongo's site an open ticket about this) thanks!

Hey I'm Ben, the other founder of MongoHQ.

Encryption is a common request for mongo and you can actually compile it with ssl support (scons --ssl), but it just isn't in the build by default.

Since most of the drivers support ssl already we leverage that and do offer encrypted mongo in a beta. If you would like to use an encrypted version of mongo in the cloud then shoot us an email to support@mongohq.com

thanks for the answer, that's great news!

I've written a small ajax wrapper for Parse.com's API.

https://github.com/srhyne/jQuery-Parse

Just $.parse.get/post/put/delete is needed; served from file:// or a Chrome extension and you have a completely client side prototype or internal business tool.

I'm really excited to see StackMob, Parse, and MongoHQ take off. Talking to one of the MongoHQ guys at MongoDB Seattle and they mentioned they were working on some kind of client side authentication system. jsOAuth is cool as well with StackMob.

See CouchDB.

What is CouchDB?

A document database server, accessible via a RESTful JSON API.

Ad-hoc and schema-free with a flat address space.

Distributed, featuring robust, incremental replication with bi-directional conflict detection and management.

Query-able and index-able, featuring a table oriented reporting engine that uses JavaScript as a query language.

http://couchdb.apache.org/docs/intro.html

More specifically CouchApps [1] which can handle not only the DB part but also side things like serving ressources, authentication, validation, replication,... No silver bullet but interesting nevertheless.

1: http://couchapp.org/page/index

Schema-free != handling data model changes transparently.

I don't understand the distinction.

In CouchDB, if you want to change your schema you just save the new data.

Your code will have to handle the old and the new form, but that is always true. In CouchDB the data is just a JSON document, so handling the changed model is about as easy as it gets.

What am I missing?

This is what you are missing: > but that is always true

You may have an old form you can't change.

For example:

- 10 apps connected to the same database. Changing the model breaks 10 apps. And you may not have access to change the code to any of those apps.

- In healthcare, when either the database or app go down people literally die.

This isn't as easy as it gets: no database vendor has a solution for this to date.

(Edit: formatting)

Okay.....

I'm not sure what your point is. As you say, this is hard, and no one else has a solution either.

But the nature of CouchDB can help here - views can be used to create backward compatible versions of new data, and reading a JSON document with extra fields won't break existing code.

It's true it's not magic, though, if that is what you are after.

Edit: I see now you are working on http://chronicdb.com/ which looks like it tackles this problem.

10 apps connected to the same database. Changing the model breaks 10 apps. And you may not have access to change the code to any of those apps.

"Database as an integration layer" is an antipattern that should always be avoided (except possibly in the case of reporting applications). CouchDB isn't unique there.

In healthcare, when either the database or app go down people literally die.

Yes, and? CouchDB is reliable, distributable, etc etc. If your point is that you should be careful when you upgrade, then yes, I agree.

Adding extra JSON fields can break code if you have matching entity types, say in Java, for example, and its trying to "cast" those JSON objects to that data type I am fairly certain it will throw an exception if the JSON object has a field thats not in the Java type

Views are the reflection of the light at the end of the tunnel, but solve less than half the problem to date. Views work on SELECT, but not on UPDATE/INSERT/DELETE.

Also, what makes you say that database as an integration layer should always be avoided?

Also, what makes you say that database as an integration layer should always be avoided?

15 years of writing software. Every time I've had to deal with a shared database it has caused problems, and every time I've built a system avoiding that antipattern it has worked much better.

But it's not just me, here's some references:

Enterprise Integration Anti-Patterns # - The Shared Database: http://ianfnelson.com/archives/2010/11/08/shared-database/

Database as an IPC Antipattern (which is a subset of the broader shared database antipattern): http://en.wikipedia.org/wiki/Database-as-IPC

There is a good reason why SOA was created, and why exposing applications' data via defined interfaces is very popular.

Database-as-IPC is about the transport aspect, not the logical integration (which is the problem of back-compatibility).

Ian F Nelson's blog post is about a DB that is tightly-coupled to the app: “my” application is using NHibernate as an ORM layer, so until this invasion we have been able to perform database schema refactorings with relative impunity.

Conventionally, RDB schemas are designed in terms of the data itself, not a particular app. In the Enterprise (where RDB are most used), data typically outlives different applications for the same task over decades, often written in different languages. Typically, the data needs also to be accessed by several different applications.

But here I'm talking about logical back-compatibility (surviving version changes) - the blog makes good points about "caching and application-level security". Where those vary with the application, it makes sense to separate them from the DB. But like Database-as-IPC, they are closer to the transport layer than to logical structure.

Views work on UPDATE/INSERT/DELETE with some restrictions, in MySQL 5.0 anyway: http://dev.mysql.com/doc/refman/5.0/en/view-updatability.htm... More expensive RDB's might too.

(I should have included this link in my other reply on problems with updating aggregated queries).

That's what versioning is for, isn't it? You still can submit old data under the previous version, and old apps will works just fine as they won't see the new data.

New apps will have to deal with old versions somehow, or some back-end job will produce new data out of the old.

If you change the schema then the old version of the app can break. For example, if two tables are merged together.

If one suggests old data and new data be stored separately then the door to data inconsistency opens.

Sick of seeing projects that use twitter's bootstrap I'm building a simple api only cms: https://github.com/gvnn/flatwhite. It's a simple project that I'm using to teach myself node.js... in the future would be great if users can define their data structure and execute CRUD operations via REST calls

The wheel of reincarnation spins.

We distributed the intelligence from the mainframe to minis/workstations and then later PCs.

Then we re-centralized it to the datacenter/cluster with the web browser 1.0.

Now we're redistributing it to the browser (thin clients finally win).

Sadly, it appears that the democratizing effects of the PC revolution won't apply when the code is delivered from a centralized server. Doing the computation at the edge doesn't subvert control if the code comes from the center. That (absent possibly impossible in practice cryptographic measures) jailbreaking edge devices is possible isn't terribly relevant if 99.999% of folks can't even understand the things they could do with a jailbroken device.

I was looking at kinvey.com mentioned in the comment of the article while researching an alternative for Adobe Publish at work. Their 1-2-3 explanation on the home page is compelling, except for the 1 part. Yes, one should have the option of creating or extending a custom model, but why not have a library of pre-built solutions? "Here's our e-commerce starter kit, it's got a cart, products, customers, tax tables, and inventory management. Also we built an awesome back-end and some integrations with these payment processors." If you're building a BAAS, why not pick three or four common web-app types and make the models for them.

In fact, there are several such backend-as-a-service offerings in existence already, including StackMob -- a ycombinator company if I'm not mistaken. And note that any such BAAS that can be used for mobile development can just as easily be used for desktop web development. More and more desktop web apps are doing most or all of their logic locally using Javascript and then accessing their servers RESTfully for data persistence. StackMob is well-positioned to take advantage of both the surge in mobile apps and these new trends in desktop web development.

The problem with webapps using a BAAS intended for mobile apps is the same origin policy. I've yet to find any providers that support Cross-Origin Resource Sharing. This restriction doesn't apply to mobile apps (or Chrome extensions)

OAuth 2 + JSONP might be a good solution to the cross-origin issue.

The primary client for an API I'm currently working on is an iOS app, so for the web version of the client, it just made sense to build on top of the same OAuth-authenticated API. Instead of traditional sessions, we store the access token in a cookie and sign each authenticated request just as you would any third-party API.

If it were to operate as simply a REST API, then the service could simply return everything via JSONP to avoid the CORS trap.

JSONP only works for GET requests. It's not 'proper' AJAX. You wouldn't be able to POST, PUT or DELETE.

JSONP should only be used for public web services. otherwise, any third party website could make privileged calls.

The BaaS architecture is actually a perfect situation for making privileged calls with CORS, because the server is wholly responsible for the user's identity and permissions.

We use CORS at spire.io.

True, but don't those services require schema definitions and similar setup? It seems like a lot of prototyping could be sped up by simply inferring relationships based on RESTful URLs and relying on schema-less storage.

Seems like that would only work in the simplest of cases. And then why would I pay someone for the backend of a trivial app?

I'm not sure. It certainly couldn't handle every use case, but maybe with some convention-style configurations (e.g. a record created via a POST to /users would hash any "password" field) you could handle a bunch of typical web app scenarios.

And if that doesn't work, perhaps the schema-less prototyping is a free tier with a more robust backend service offering for the paid tier?

"The password field is handled differently than the others; it is encrypted on the server and never returned to any client request."

https://www.parse.com/docs/rest#users

Isn't this what frameworks like rails, grails and spring roo are for? They make creating a basic CRUD app trivial. If you just need a trivial app then use one of these frameworks, anything more is obviously going to require customization on the backend code. I suppose there could be a niche for managed RAD (rapid application development) packages, or something like that, where they help people build an app using rails, or something, and then host it, but its not nearly as groundbreaking as seems to be suggested, these frameworks are fairly trivial to use as is to get basic CRUD functionality and if you need more customization then you're already moving out of the territory of having the backend as a service without having to pay custom development rates.

I've thought for a while about a service that would allow you to define a set of basic models/domains, including basic validation rules and relationships on them, then offer to generate the CRUD code, but in a variety of frameworks - grails, zendframework, rails, asp.net, django, etc. Beyond being a 'neat idea' though, I couldn't find any real use case for it.

Yeah... the people who can make use of this sort of thing already would probably be able to do the steps of generating the code in the RAD framework so I just don't think there's much room to actually make money on something like this but maybe if integrated with something like Heroku it could have potential.

On the REST part it is not ideal for complex object based system where there are multiple resources which have n number of relationships and different http verbs on them can be mean different operations when exposed in a different way. ( I know its confusing.. :)). I just look at it as a design pattern which makes client side devs consume your services easily. It definitely helps in creating simple webservices but REST as a whole I would say doesn't satisfy everything a backend developer would want to expose or do in an efficient way. Think of batching... in REST.

Many developers are already using our StorageRoom CMS (http://storageroomapp.com) for this.

An administrator configures content types, editors manage the content and developers fetch the content through the RESTful JSON API (including CORS & JSONP support).

How is it different to Parse/StackMob/Kinvey? It is more about delivering editorial content to mobile or web apps and not about a read/write backend.

This highly depends on the kind of apps which needs to be done. Its highly difficult to generalize a backend for a n number of apps. I am not comparing the term "backend" here with Parse or YQL or aggregating datasources which is great. You write a backend to custom suite your app so that its highly performant both on the client and server side and you have paas hosting services for the language of choice. I would personally not like to have a generalized backend.

Disclosure, I work for Zipline Games on Moai Cloud.

Moai Cloud (http://getmoai.com), while focused on enabling backends for mobile games provides some of what you describe. We use mongo on the backend to provide schemaless storage and are building out a set of RESTful apis. Additionally if you need to you can write and run your own custom code on the server.

There cloudmine.me also. They are providing similar services. I think mobile BAAS is a limited market. But, domain based BAAS is a huge market. You can never build a backend model to suffice the needs of different mobile apps. It will be difficult to build a backend model for EMR system but providing a backend model to convert dicom data for consumption on a mobile platform will be a good service.

Is there a more strict definition? Would this work:

  SAAS - HTML = BAAS

Too broad or too narrow?

My startup is building a BaaS that focuses on flexibility and any client not just mobile. We are working on Javascript bindings to improve ease of use but it has REST endpoint that uses JSON. You can read more about it at http://iknode.com

RunMyProcess a startup from Paris, France does this. http://www.runmyprocess.com . I teach university students how to use this. Impressive product. Sort of like an enterprise-level configurable IFTTT.

That's what these guys are doing: http://www.ibcom.com.au/ despite a bit of a naff web presence I've had a peek under the hood and they're doing some really interesting stuff.

Rest apis on mongolab and mongohq look extremely promising.

We are also doing the workflow and job scheduling bit at http://www.thecloudblocks.com

Reply on news.ycombinator.com