Open Sourcing Our SDKs
blog.parse.com
blog.parse.com
The real problems show up when you've all of a sudden got these old apps out in the field with no way to upgrade them to a newer version of the DB. If you aren't being diligent and making all your Parse DB calls through their cloud functions, you are guaranteed to end up in a situation where your older mobile clients are holding back schema and feature evolution. You can try to hack around it using triggers, but these are too buggy to be reliable.
Other examples of where Parse is painful:
- beforeSave/afterSave triggers are completely unusable! They are unbelievably limited and each one you add slows your app down by as much time as it takes to run. If you want to do something like add users to Mailchimp, you have now tied the length of time it takes to save a user to the length of time it takes to talk to mailchimp!
- There is absolutely no way to test your parse code other than uploading it to a parse app and hitting it. You can hack together some node code that sort-of tests it, but it's nowhere near representative of what happens on the server.
- You can't manage Parse.Config programatically. There's a web interface and that's that. None of your config can live in your git repo because there's no way to read or write it from the command-line.
- The schema tools are pretty bad on the website. IIRC they introduced REST APIs to make changes to schema recently, but unless you plan on writing tooling around those, you'll be manually syncing schema between prod/staging at some point in your life and you'll probably also miss a critical column that got added.
- Parse has a bunch of weird datatypes that act strangely in their cloud code. Want to make a pointer and store it in a parse field? Construct its internal representation and set it! I wish I was kidding. If you construct a pointer incorrectly, it can actually create a phantom object in another table. Also wish I was kidding about this.
- createdAt and updatedAt are special and Parse won't let you touch them or set them. Heaven forbid you want to actually set these to a meaningful time other than the actual last time the record was written. Nope. So you end up making your own updatedAt columns and just using those.
For your own sanity: stay away from Parse. The range of use cases where it is actually useful is so narrow that you'll leave that very quickly unless you are building a toy app.
I'm Héctor, a Parse developer advocate for the past 3.5 years. I've probably seen every question related to Parse, spend most days supporting apps with millions of users, and hopefully I can address some of these.
You bring up some valid points, some specific to Parse, others related to backend systems in general.
- "The real problems show up when you've all of a sudden got these old apps out in the field with no way to upgrade them to a newer version of the DB. If you aren't being diligent and making all your Parse DB calls through their cloud functions, you are guaranteed to end up in a situation where your older mobile clients are holding back schema and feature evolution."
Maintaining compatibility with a long tail of old, stale versions of your clients in the field is something all of us need to deal with as app developers. Our examples do place more emphasis on client-originated object updates and queries due to our approach of making it very easy for developers to get started. Some of our more sophisticated developers end up using the approach you've used as an example of moving as much of the operations over to Cloud Code. This is, in fact, one of the reasons we decided to work on our Cloud Code server-side compute environment.
- "beforeSave/afterSave triggers are completely unusable! They are unbelievably limited and each one you add slows your app down by as much time as it takes to run. If you want to do something like add users to Mailchimp, you have now tied the length of time it takes to save a user to the length of time it takes to talk to mailchimp!"
A beforeSave, by definition, will block a save until the operation has finished. Save operations on the client side are performed asynchronously, so this should generally not block your app. An afterSave can be used for things such as sending an email through Mailchimp, which in general you would not block a user operation on, but rather these would be queued up. There are some sensible default timeouts in place to discourage lengthy operations that could lead to a unsatisfying user experience (e.g. creating a new row should not take more than 3 seconds). If you are OK with having the user wait more than 3 seconds, you can use Webhooks to redirect the beforeSave/afterSave/cloud-function computation to a third party server of your choosing. In that case, you would have up to 30 seconds to return a response to the client.
- "There is absolutely no way to test your parse code other than uploading it to a parse app and hitting it. You can hack together some node code that sort-of tests it, but it's nowhere near representative of what happens on the server."
This is a big pain point many of our users have highlighted. Cloud Code is not node, so the best way to test code is by running it on Cloud Code itself. We do not have a good solution for satisfying this need at this time, so I would suggest using node.js on a third party server and taking advantage of Webhooks. This effectively allows you to Bring Your Own Server and provides you with limitless flexibility on what can be done as part of a save hook or cloud function call.
- "You can't manage Parse.Config programatically. There's a web interface and that's that. None of your config can live in your git repo because there's no way to read or write it from the command-line."
I think that everything you can do on Parse should have a REST API equivalent. We've recently launched a Hooks API as well as a Schema API. Perhaps a Config API can be considered. I personally use Config to store credentials that I would not want to be checked in to version control, and I use Parse Objects for anything that changes often enough to require programmatic updates.
- "The schema tools are pretty bad on the website. IIRC they introduced REST APIs to make changes to schema recently, but unless you plan on writing tooling around those, you'll be manually syncing schema between prod/staging at some point in your life and you'll probably also miss a critical column that got added."
You can clone an app and its entire schema from your Settings. This is the best way to ensure there is a 1 to 1 mapping between prod and staging. One thing that is missing from this is cloning data from prod to staging. Bigger apps, with gigabytes of data, could take a very long time to clone. In this case, I would suggest having some sort of seed script that populates your staging database with placeholder data. I understand this does not solve the problem of having an exact copy of your production database, but this is not a simple problem to solve (would you want to keep both apps in sync as the prod app gets updated? would you be willing to wait hours/days for your an initial sync from prod to staging?). With that said, this is something we're aware of and hope to be able to address soon. Stay tuned.
- "Parse has a bunch of weird datatypes that act strangely in their cloud code. Want to make a pointer and store it in a parse field? Construct its internal representation and set it! I wish I was kidding. If you construct a pointer incorrectly, it can actually create a phantom object in another table. Also wish I was kidding about this."
Mucking with JSON to successfully create a valid pointer sounds like an easy way to introduce bugs in your code. You shouldn't need to to do this, use YourClass.createWithoutData(objectId) instead.[1]
- "createdAt and updatedAt are special and Parse won't let you touch them or set them. Heaven forbid you want to actually set these to a meaningful time other than the actual last time the record was written. Nope. So you end up making your own updatedAt columns and just using those."
createdAt and updatedAt are automatic fields set by the underlying mongo database, and you can depend on these being accurate as no user or admin operation can force them to use arbitrary values. You're definitely welcome to create your own Date fields if your use case requires arbitrary values. This shouldn't be too much of a burden, IMHO.
Hopefully this helps! As always, if you have any questions, feel free to join us on our developer community[2].
[1]: http://parse.com/docs/js/api/symbols/Parse.Object.html#.crea...
[2]: https://groups.google.com/forum/#!forum/parse-developers
- "Maintaining compatibility with a long tail of old, stale versions of your clients in the field is something all of us need to deal with as app developers. Our examples do place more emphasis on client-originated object updates and queries due to our approach of making it very easy for developers to get started. Some of our more sophisticated developers end up using the approach you've used as an example of moving as much of the operations over to Cloud Code. This is, in fact, one of the reasons we decided to work on our Cloud Code server-side compute environment."
The problem is that Parse does not offer an "afterRead" hook, so if you offer a field and a client consumes it, you must continue to offer that field until that client no longer exists. When writing traditional server APIs, I can synthesize a field for older clients during read options. With Parse, not so. Parse clients old and new read the same, raw table no matter what -- this is where I feel Parse fundamentally fails.
- "A beforeSave, by definition, will block a save until the operation has finished. Save operations on the client side are performed asynchronously, so this should generally not block your app. An afterSave can be used for things such as sending an email through Mailchimp, which in general you would not block a user operation on, but rather these would be queued up. There are some sensible default timeouts in place to discourage lengthy operations that could lead to a unsatisfying user experience (e.g. creating a new row should not take more than 3 seconds). If you are OK with having the user wait more than 3 seconds, you can use Webhooks to redirect the beforeSave/afterSave/cloud-function computation to a third party server of your choosing. In that case, you would have up to 30 seconds to return a response to the client."
If it is the case that afterSave does not block a save, then the documentation is pretty poor around this. My understanding is that afterSave and beforeSave are both required to complete before you'll get a return value from saving an object. If this is not the case, I suggest making this clearer in the docs.
- "This is a big pain point many of our users have highlighted. Cloud Code is not node, so the best way to test code is by running it on Cloud Code itself. We do not have a good solution for satisfying this need at this time, so I would suggest using node.js on a third party server and taking advantage of Webhooks. This effectively allows you to Bring Your Own Server and provides you with limitless flexibility on what can be done as part of a save hook or cloud function call."
At the point where you are using Webhooks like this, the advantages of Parse disappear entirely. You can get far more flexibility from setting up a node.js server that hits MongoLab (or some other hosted Mongo). Honestly I feel that it is absolutely _wrong_ for your DB to block on a backend server dealing with app logic. This is an inverted version of what your design should look like.
- "I think that everything you can do on Parse should have a REST API equivalent. We've recently launched a Hooks API as well as a Schema API. Perhaps a Config API can be considered. I personally use Config to store credentials that I would not want to be checked in to version control, and I use Parse Objects for anything that changes often enough to require programmatic updates."
If there's a way to set Parse.Config via REST (maybe there's an undocumented PUT?), it is not documented anywhere on the site. Since Parse.Config is public data to anyone using my app, I would certainly avoid putting credentials in there.
- "You can clone an app and its entire schema from your Settings. This is the best way to ensure there is a 1 to 1 mapping between prod and staging. One thing that is missing from this is cloning data from prod to staging. Bigger apps, with gigabytes of data, could take a very long time to clone. In this case, I would suggest having some sort of seed script that populates your staging database with placeholder data. I understand this does not solve the problem of having an exact copy of your production database, but this is not a simple problem to solve (would you want to keep both apps in sync as the prod app gets updated? would you be willing to wait hours/days for your an initial sync from prod to staging?). With that said, this is something we're aware of and hope to be able to address soon. Stay tuned."
It's not cloning a DB that's the problem -- it's programatically updating schema so that I can deploy schema changes to my staging environment and then deploy the matching set of schema changes to prod environment, automatically. This is not possible without a bunch of scripting. And, obviously, there is no easy way to mass-migrate parse objects to a new schema either.
- "Mucking with JSON to successfully create a valid pointer sounds like an easy way to introduce bugs in your code. You shouldn't need to to do this, use YourClass.createWithoutData(objectId) instead."
This needs to be better documented. Google searches bring up results where people are constructing pointers via JSON and this was indicated to be "the right way to do things". I don't have a reference off-hand to the post where this was mentioned, but it is definitely there.
- "createdAt and updatedAt are automatic fields set by the underlying mongo database, and you can depend on these being accurate as no user or admin operation can force them to use arbitrary values. You're definitely welcome to create your own Date fields if your use case requires arbitrary values. This shouldn't be too much of a burden, IMHO."
But when you combine the fact that I've got apps out in the field looking at updatedAt instead of my new field this becomes a very large burden. If I want to mass-import data into my system, I'm at the mercy of system timestamps. There are no options to fix this. These fields absolutely _should_ be under control of the admin, at least. If I have the master key, why not let me change it?
- "If it is the case that afterSave does not block a save, then the documentation is pretty poor around this. My understanding is that afterSave and beforeSave are both required to complete before you'll get a return value from saving an object. If this is not the case, I suggest making this clearer in the docs."
I was wrong earlier. A client will not get a response until an afterSave completes execution. What I should have pointed out is that ongoing http requests to external endpoints will not block the afterSave from returning. As such, you can still use Mailchimp in this case without worrying that a client will be blocked. Of course, you will want to avoid blocking the afterSave on the third party network request (e.g. adding on to the promise chain), which should not be a problem if this is truly a fire-and-forget operation.
- "At the point where you are using Webhooks like this, the advantages of Parse disappear entirely."
I think you're not giving the client-side SDKs enough credit here. Server side business logic is only one part of the equation. With Parse, you also get a data browser, analytics, push notifications, and probably most important of all, a database.
- "It's not cloning a DB that's the problem -- it's programatically updating schema so that I can deploy schema changes to my staging environment and then deploy the matching set of schema changes to prod environment, automatically. This is not possible without a bunch of scripting. And, obviously, there is no easy way to mass-migrate parse objects to a new schema either."
This feedback seems contrary to your earlier ask for a way to script Config variable updates. :) There's the Data Browser for UI based manipulation, there's a Schema API for programmatic manipulation, and there's a dedicated schema migration setting. You can use any of these approaches as needed, and depending on how much scripting you wish to do.
- "This needs to be better documented. Google searches bring up results where people are constructing pointers via JSON and this was indicated to be "the right way to do things""
Adding Parse objects to pointer fields is documented in all of the guides. createWithoutData is an alternate constructor that can be useful when creating pointers explicitly, but it's not the only way of creating a pointer. The link I posted is straight from the API Reference docs.
- "If I want to mass-import data into my system, I'm at the mercy of system timestamps. There are no options to fix this."
These two fields, as well as the object id, can be set to arbitrary values as part of a JSON import. Otherwise, you would not be able to import data exported from another Parse app.
Honestly, this is part of the problem with Parse. It's a big, black box where the documentation doesn't _actually_ dig into the cases that are really important. For a big, black box the documentation really doesn't step up to where it needs to be.
So some stuff blocks afterSave, and some stuff doesn't. Does saving another object from afterSave block a save? What about the nested triggers on that other object? Why isn't that documented anywhere?
- "Adding Parse objects to pointer fields is documented in all of the guides. createWithoutData is an alternate constructor that can be useful when creating pointers explicitly, but it's not the only way of creating a pointer. The link I posted is straight from the API Reference docs."
I've never seen a reference to this, despite repeated Googling.
Open the JS developer's guide. The doc for"createWithData" says that it creates a "reference" to to an object. OK, that's a synonym for a pointer, well... sort of. Let's look at the actual method example:
Creates a reference to a subclass of Parse.Object with the given id. This does not exist on Parse.Object, only on subclasses.
A shortcut for:
var Foo = Parse.Object.extend("Foo");
var pointerToFoo = new Foo();
pointerToFoo.id = "myObjectId";
Instead we Google for "parse set pointer" and this page comes up:
https://www.parse.com/questions/javascript-api-set-and-creat...
"That format, {"_type":"Pointer","className":"User","objectId":"asdadsasd"} is what is considered a Pointer object. That is the recommended method"
This has long been documented here: https://parse.com/docs/js/guide#cloud-code-resource-limits
- "I've never seen a reference to this, despite repeated Googling." ... "Instead we Google for "parse set pointer" and this page comes up:"
It sounds like you're leaning a lot on Google searches bringing up two year old questions. There's a massive archive of questions from the last 3 years, around 10,000 on my last count. Going back through every single comment and answer to make sure they're still accurate would be a huge undertaking, and a year ago we stopped using those archived questions.
I highly recommend reading through the entire documentation for the platform you're working on, as well as the API Reference for your SDK. You can find both at https://parse.com/docs. I think now is a good time to point out we're also accepting pull requests against our Docs[1]. We welcome diffs that make the docs clearer for everyone.
I appreciate your responses, but I'm done here.
> It works great as a fast way to get your app up and running
It's a really neat and useful tool for prototyping. Obviously as you start to get more and more business logic that should be in your backend, you may want to migrate to your own backend, but if all you need is a simple key:value store, it's great. Heck, it even does files, auth, email, etc. You can take a prototype pretty far before reaching the edge of its usefulness.
1) Don't even bother with their hosted Cloud Code, spin up your own backend and use the webhooks. As of the JS SDK 1.5 you can finally run a node backend securely.
2) Push complex logic off the client and into Cloud Functions.
3) Put plenty lot of thought into your schemas before creating them.
That said, there are a few big areas of need that I see that I hope the team will address:
1) Development/Staging environments are a pain, particularly around schemas. Tooling around migrations would help a lot here.
2) Testing tools leave a lot to be desired (I had to roll my own backend mock library for node, https://github.com/HustleInc/parse-mockdb )
3) Limited exposure of MongoDB feature set (e.g. aggregation framework)
4) Downtime (and notification speed / transparency thereof)
5) Security/ACLs leave a lot to be desired, particularly for complex schemas. For instance, it would be great to specify object-level access controls in terms of Parse queries (e.g. Users that satisfy a query built around the object in question).
Basically I want a more powerful version of this: http://blog.parse.com/learn/secure-your-app-from-the-data-br...
To me, it's just as easy/fast and more powerful to use RoR/Sinatra or Django/Flask from the get-go and for limiting longer-term tech-debt (migrating from one to the other).
I'd really like to see all infrastructure SDKs open sourced (obviously a tough sell, but maybe Parse paves the way). In building our core stack, we don't tolerate closed source. But we end up allowing it for things like Crashlytics, MixPanel, Segment, etc - and it's not a free lunch.
Also, we're doing a series of blog posts that will cover each part of the SDK stack. Lots of learnings to share from our kick ass SDK team.
Here's the first: http://blog.parse.com/learn/the-parse-sdk-whats-inside/
When I was working at MoPub, our open source SDK was one of our major selling points.
It seems like that would be a promising direction to give Parse apps flexibility to swap backends down the road, without having to change the entirety of their client code base...and preserving the out-of-the-box speed and convenience of using Parse for brand new and prototype apps.
Essentially Disk I/O was using the same synchronized lock as the main thread. On some devices this caused a reproducible ANR (App Not Responding). I ended up writing a small REST Api client using Retrofit to replace the SDK. It worked well but as mmastrac mentioned there are just too many other critical issues with Parse to make it suitable for anything other than quick prototypes.
Besides, it's super expensive if you need have any sort of scale. You could almost hire a part-time engineer to build and maintain a backend for those prices.
If you can find a part-time engineer that can build and maintain a backend with all of Parse's features, you should absolutely do that.
> If you can find a part-time engineer that can build and maintain a backend with all of Parse's features, you should absolutely do that.
Most likely, you're not going to be using all of Parse's features; probably just a small subset of them. So I think it's easily doable. In fact, if you were intending to use Cloud Code, you could probably do it yourself with the time you'd save by not having to deploy to production to test your code!
We've actually fixed a few ANR deadlocks in this last release [1], as well as a few others in the releases since 1.8.0 [2].
If you can supply reproduction steps, we'd love to be able to fix this issue as I'm sure you're not the only one experiencing it. In addition, we're more than happy to accept fixes as contributions :D
[1]: https://github.com/ParsePlatform/Parse-SDK-Android/releases/...
I think the guy on the ticket didn't really understand what I was saying. I'm happy to provide any extra info necessary (what I can remember anyway).
From looking through the code, the issue still appears to exist. [1] is called on the Main Thread. It eventually calls to a synchronized block [2]. [3] is called from a background thread and uses the same synchronised lock.
--> Main Thread + Background Thread using same lock == Trouble
One other annoying issue I reported (#937744189590638 - private) was not being able to update the GCM token. The field is readonly, even though the GCM docs explicitly state that the key changes over time. Meaning you have to delete and recreate the Installation - and what happens if you have other objects linked to that installation? Super annoying. Could be easily solved by making that field writable.
[1]: https://github.com/ParsePlatform/Parse-SDK-Android/blob/mast...
[2]: https://github.com/ParsePlatform/Parse-SDK-Android/blob/mast...
[3]: https://github.com/ParsePlatform/Parse-SDK-Android/blob/mast...
WRT the ParseCommandCache links you sent, you're right that there is some trouble using the same lock on a background thread and UI thread. It would possibly cause some UI stuttering, but it wouldn't necessarily cause a deadlock on it's own. There's probably a bit more to it and I'd love to dig deeper and find out why.
Posted more info for the ParseCommandCache ANR in the ticket.
Cheers for taking feedback seriously! :)