Datastore API (beta) | Hacker News Reader

Datastore API (beta)

dropbox.com

207 pointsgluegadget84 comments

The biggest problem right now with online file storage services (sorry, I meant "The Cloud") is that while they all do basically the same stuff, there's no standard protocol for accessing them. It's a bit like if every website used a custom protocol instead of HTTP, and web browser vendors had to explicitly collaborate with each web site (or use the website's SDK) in order to be able to retrieve content.

Imagine this: All the major players in the space (Dropbox, Box, Google Drive, Skydrive, Sugarsync etc.) as well as app developers got together and came up with a standard way of doing this that makes it possible to use a single protocol & API for all online storage providers. I see this being a good thing for everyone - users can choose whatever service they want, app developers can easily provide their customers with a ton of options, and storage providers don't have to explicitly produce their own SDK or court developers to do integration with their service. We did have WebDAV, but that doesn't seem to be widely adopted by any of the major players.

I've been facing this problem myself with my own app; it's a lot of work to support all the major services and this is really just plumbing stuff that everyone would rather be taken care of so they can focus on the more important stuff. I had a meeting with Box recently where I proposed the idea of an open source framework for iOS which would provide a common native API that abstracts over all the different REST APIs used by different storage providers. The people I spoke to seemed quite open to the idea and agreed that it would be useful. I don't think a library like this would necessary be considered a bad thing by any of the major players, as it evens out the playing field and makes it easy to support different services.

If you're interested in taking part and/or using such an API in your iOS app, I'd love to hear your thoughts on the matter. I have some code already (which I haven't released yet) and am looking to clean up the API a bit, but I think it would be useful to a lot of developers.

Big players aren't interested in anything that reduces lock-in. Seamless integration is one of the keypoints they have to get you to use their services.

Obligatory presentation by Bruce Schneier on the online feudalism: http://www.infoq.com/presentations/cloud-security

I found Box to be the opposite of this. While I obviously can't speak on their behalf, and I was discussing the issue with their developer relations people (not senior executives), they said they didn't see a problem with this idea.

exactly...it is unlikely they would be interested in data storage being treated as the commodity it is.

That's exactly the idea of Remote Storage [0]. The problem is here that the lowest common denominator of functionality isn't actually very much.

Anything but fairly trivial apps quickly run into problems with application data (how relevant), file patching, file intents (Google have this),

Google have, IMO, the most advanced solution in the space, so you could argue have the most the lose from standardising. But in fairness, some of their stuff (I'm thinking the real-time collaboration part of the Drive API) is so complex, there's no way you'd ever see it in a common standard.

That said, it'd be really nice to see a common API over the lower stuff.

[0] http://remotestorage.io/

Thanks for the link, this does look very interesting.

I believe filepicker.io have something along these lines as well, but haven't had a chance yet to look into the details of it.

Filepicker is a API across remote storage as a service and everything goes through the FilePicker servers. There's no library to use directly.

RemoteStorage is specification primarily, but it's very much about their JavaScript library that you use directly to communicate to the remote storage.

By Jove, you're right! Perhaps what we need is some kind of universal protocol for transferring files. We could call it the 'File Transfer Protocol' or something.

LOL yes ;)

What happened to FTP anyway? I know there were a number of issues with it (security, working through NAT, lack of a standard directory listing format etc); but these were all problems that people managed to solve in different ways.

It just goes to show how far how our industry is willing to embrace change for the sake of change (with the results sometimes being inferior) every time a new buzzord appears. I reckon a modernised version of FTP would do the job pretty well.

What I'm mainly concerned with at the moment is building something that recognises a widely-adopted protocol is going to be very difficult politically to achieve (and get implemented), but a library which abstracts over the different protocols would relieve most of the pain.

> What happened to FTP anyway?

It's too hard to set-up for casual file sharing (backup, docs, photos etc.), and it's less efficient and scalable than BitTorrent for wide file distribution (piracy).

Perhaps with HTTP 2.0 and IPv6 looming though, we will see some progress on an FTP 2.0 that resolves the issues you mentioned.

People didn't "solve" those problems so much as hack their way around them in a wide variety of ways with no standardization, that is the problem. Having to occasionally support customers that need FTP servers, I can tell you that it is a never-ending nightmare of incompatible clients, server settings that must be set differently depending on customer requirements just because of issues with FTP, and firewall problems (on the customers side, though it also creates hassles on our end).

Pretty much anything would be better than FTP.

As it turns out, the protocol for actually transferring the files is utterly irrelevant compared to the protocol for getting updates, keeping various local stores synchronized, retaining versions, managing conflicts, handling client-side bugs and surprises, etc.

It's a good point I suppose. I've never had a need or urge to use a service like Dropbox to keep files in sync though. I prefer just to carry my working copies around, or use Google Docs. Maybe I'm missing out on seamless splendour, I don't know.

> "Maybe I'm missing out on seamless splendour, I don't know."

I'd say that yes, you are (no snark intended). Dropbox is one of those things that you didn't really know you needed until you've used it for a while. It crept up on me and became indispensable for pretty much every collaborative project I've worked on.

Yes, file synchronization is really the hard part from an architecture POV.

From everything I've seen so far, Git appears to provide the best solution. It is rather complex though, and it's a big challenge to build an intuitive user experience on top of this that shields non-technical users from a lot of the complexities.

Check out https://www.inkfilepicker.com/

Hey, this is exactly what NimbusBase (http://nimbusbase.com) wants to do. It's a work in progress to integrate all the services, but we got the major 2 Dropbox and Google Drive.

I'd like to coin a modern update of Zawinski's Law[1]:

    Every software company attempts to expand until it
    offers a cloud-based platform API. Those companies
    which cannot do so are replaced by ones which do.

[1]: http://en.wikipedia.org/wiki/Jamie_Zawinski#Zawinski.27s_law...

Something I don't get about Dropbox API: Why can't I just throw URLs at it?

If a user wants to save <thing at URL>, I have to pull it to my server and push it into Dropbox. Dropbox could just have grabbed it directly and in many cases, won't even have to thanks, to system-wide caching.

(This is based on the last time I looked at the API; maybe this is supported now, but I don't think so.)

You're describing the Saver Drop-In: https://www.dropbox.com/developers/dropins/saver

Can the Saver Drop-In be used with arbitrary third-party URLs? To me this is what was being described, so if not, then no, it's not describing the Saver Drop-In.

Yes, you can provide arbitrary URLs to the Saver and our (Dropbox's) servers will download them, assuming they are publicly available.

(Note: I am an engineer at Dropbox and worked on this product.)

Thanks. Off topic but slightly related if I'm going to rely on your service as a developer: as an engineer at Dropbox, can you tell me why I am seeing random photos that are not mine in my dropbox Photos folder? And does this mean that other people get to see my random photos? Edit: only on iOS. And maybe it's not a "folder" per se, it's the second tab from the left at the bottom of the screen.

That photos tab shows you photos from anywhere in your Dropbox account, so these photos may be from shared folders you're a member of. (Likewise, other people can only see your photos if you shared them with them.) Another thing that may happen is someone "camera uploaded" their photos to your account (e.g., by plugging into your computer and choosing to import photos when Dropbox asked.) If you're still not sure where they're coming from, you should open a ticket: https://www.dropbox.com/support/s/92/3201098/c/2

Thanks, very informative answer. I think they must be from a certain shared folder I have access to, which would explain it. Quite a relief, actually.

Great to know they got around to this and did so in a way that leverages the existing HTTP caching protocols. Thanks!

How would system-wide caching work unless the HTTP server dropbox requested the file from sent a checksum of the file being requested? Just because something has the same URL doesn't mean that it's the same file at the other end (http://wordpress.org/latest.tar.gz for instance)

It obey's HTTP caching controls. Checkout the end of the docs: https://www.dropbox.com/developers/dropins/saver

I see - but the typical dropbox deduplication methods wouldn't work here (at least not in terms of bandwidth usage), would they?

Such an awesome idea. However, in light of the recent events the first thing to come up in my mind when I see "datastore" and a US corporation is "PRISM integration". I'd like to see proper encryption of Dropbox as the next feature, so that we - innocent users- are not uploading our data to the US government.

I combine dropbox with TrueCrypt for sensitive stuff. For me, that's just digital copies of important documents.

So are you just syncing entire truecrypt .tc files to Dropbox? Does that work incrementally or does the entire .tc file have to get resent every time you modify a file within?

The entire file is resent.

Citation? Other comments are offering conflicting reports.

No, only changed blocks are resent. The scattering effect of Truecrypt encryption plus the relatively large blocksize does amplify the number of changed blocks somewhat.

Previous investigation and results:

http://news.ycombinator.com/item?id=2546626

I always assumed it wouldn't work incrementally, but it seems to sync my 200MB tc file too quickly to be uploading the entire thing when I make a small change. Perhaps someone who actually knows something about encryption can speak to this. Does every byte in an encrypted volume change when a piece of data changes? I guess that would be too inefficient to be the case, no?

Yes, only parts of the underlying file are changed. TrueCrypt uses XTS mode of operation (https://en.wikipedia.org/wiki/Disk_encryption_theory, http://www.truecrypt.org/docs/modes-of-operation).

The entire file may be resent, but practically it works just fine for me. The total volume is only 250mb and it is updated infrequently.

Yes - in my dropbox public folder, I keep a truecrypt bucket with scans of everything in my wallet, should I lose it.

Can't you just encrypt the data before storing it in your app? Why does Dropbox need to do this for you? Would you even trust Dropbox if they did add it themselves?

The right approach is to encrypt before pushing down the wire - it's useless if Dropbox just encrypts on their end.

Well, the nice thing about PRISM is that it's API is so simple that you don't even need to muck around with SDKs, REST, JSON encoding etc. Integration is completely straightforward.

Expect to see the NSA as a back-up-of-last-resort cloud solution provider any time soon?

Sadly, you're going to need to encrypt before uploading. Just as the webserver should never trust the client, the user should never trust the server with sensitive data.

Try boxcryptor - works with more than just Dropbox.

Proper encryption meaning metadata is encrypted as well? Isn't that the whole appeal of http://mega.co.nz/?

After quickly clicking through the api documentation I don't understand why I would use this over a hosted NoSql database like mongohq. I still like having additional options though.

Off-line access.

Perhaps you don't want to host a db just for user data.

"List – A special value that can contain other values, though not other lists."

https://www.dropbox.com/developers/datastore/tutorial/ios#re...

Seems like a very significant limitation. No mechanism for storing hierarchical data structures. For example, this is certainly not capable of storing arbitrary JSON data as a Record.

There's also limit on Postgres hstore and arrays. No nested sturcture.

As someone who has been playing around with Parse and Firebase SDKs (on Android), this seems to be the first one that offers full 2-way sync. I haven't actually used it yet but it seems that way due to the presence of conflict resolution [1]. I'm personally really excited about that for mobile side-projects and MVPs.

Parse has an offline cache, that only caches query results and not actual objects. So any local saves/updates are not reflected until you get network access again, unless you manually implement your own cache.

Firebase is better at handling realtime changes, but is lacking in any realtime capabilities at the moment. Also Firebase's mobile authentication SDK still isn't released yet so no way to handle users yet either.

[1]: https://www.dropbox.com/developers/datastore/tutorial/androi...

Yeah, the Datastores API does two-way synchronization. Also, offline caches are done for you and offline changes are propagated (with conflict resolution) when the internet comes back up.

Correction: "lacking in any realtime capabilities" should read "offline capabilities". And by that I mean offline without manually managing cache.

Also by 2-way sync I mean 2-way cache sync. Firebase is actually really good with 2-way sync, but without cache/offline (yet) it's kind of a dealbreaker for me.

What app.net wanted to accomplish.

At first glance it also looks like WinFS for the Cloud.

this is basically what iCloud was supposed to be.

So now app developers have a choice: they can use iCloud or Dropbox to manage distributed app state. Any iOS app developers (I'm not one) care to compare & contrast the two options?

It looks like Dropbox may be using Operational Transformation [0] to merge changes to lists:

Lists are special field values. They contain an ordered list of other values, though not other lists. Lists can be manipulated via four list operations: put (i.e. replace), move, insert, and delete. These list operations allow Dropbox to handle merging changes to the structure of the list automatically. [1]

iCloud, on the other hand, uses the coarser, state-centric approach of whole document versioning. [2] (Edit: okay, key-value property versioning is also available.)

But iCloud lets the app control conflict resolution, while the Dropbox Datastore API only lets you choose from a predefined set of conflict resolution strategies. [3]

Which approach provides greater value to the app developer? Has Dropbox realized here that developers are just not comfortable implementing a custom merge strategy for entire documents, and not having to worry about this is worth some loss of control?

[0] http://en.wikipedia.org/wiki/Operational_transformation

[1] https://www.dropbox.com/developers/datastore/tutorial/ios#re...

[2] http://developer.apple.com/library/ios/#documentation/DataMa...

[3] https://www.dropbox.com/developers/datastore/tutorial/ios#co...

Not answering your question, but developers can also use http://parse.com or similar.

For the iOS library specifically, I wonder if it would be possible to build a library that would (nearly) automatically map a core data store in to this Datastore API. If so, including such functionality would make the library a no-brainer for many.

I think with NSIncrementalStore it would be totally doable, and a great alternative to Core Data Sync.

are there any kind souls that could help please explain the difference between dropbox datastore, firebase and parse? thank you.

In comparison to firebase, this solves the offline sync problem, if your user is offline, you can still save to dropbox datastore locally, when you get a connection, it will automatically sync and take care of conflict resolution.

Firebase dev here. :-) Firebase does handle offline syncing / conflict resolution.

If you have a Firebase app open and you lose network connectivity, the app still continues to work fine and any modifications you make will be synced back to Firebase when you regain connectivity.

What we don't (quite yet) do is handle the case where you make changes, kill the app (without regaining network connectivity), restart the app, and then get network connectivity, but that's coming soon.

As for other differences, Dropbox Datastore seems to be tightly tied to Dropbox, so the end user for your app must have a Dropbox account (and if they want to collaborate with other users, those users must also have Dropbox accounts, etc.). So it's really for building apps on top of Dropbox.

Firebase and Parse are just generic backends, with no ties to other services, so your end users don't need anything to use your app.

As for differentiating Parse and Firebase, Firebase deals with data in real-time, pushing updates to apps as soon as data changes. Parse is a traditional request/response model, where your app has to explicitly "refresh" to get new data.

good stuff.. thanks mike!

Thanks mike, I didn't realize firebase did offline syncing now!

So good!

Does anyone know of an OSS alternative to this similar to http://helios.io/ that supports Android/iOS?

Couchbase Lite is an embedded JSON database with sync capability. It's Apache licensed and available for iOS and Android:

https://github.com/couchbase/couchbase-lite-ios

https://github.com/couchbase/couchbase-lite-android

Thanks for the link to this library. It seems like I would still need to implement a lot of features like login, but it's a great start.

We ship with support for login with Mozilla Persona. Hope to have baked-in support for Facebook Connect soon. Of course you can use it with any auth you want, but you'll end up writing some code...

What is the long term vision here? Is the idea that consumers will have little apps here and there that will procedurally interact with the owner's data on Dropbox? Or, is Dropbox making a product along the lines of S3 so that backends can consider using it too? I guess given how expensive space is at this point, that's not really an option.

dhouston has said more than once that he wants dropbox to be the filesystem of the internet.

I wonder how Apple will react to @dropbox's new Datastore sync in the long run. Hopefully it won't pull the rug out a la IAP, UDID, OF, ...

I would rather they react positively by opening up their iCloud API for web application for third parties. Apple already has such in place as can be seen from their cloud apps at icloud.com.

Apple is a hardware company.

I highly doubt they would do anything to stifle Dropbox's syncing functionality in iOS. Apple is still focused on selling iOS devices and ensuring people keep buying iOS devices, they will do anything in their power to keep you happy.

Limiting Dropbox functionality on iOS would be met my swift user opposition, something that doesn't guarantee recurring sales.

So Dropbox is becoming a more-direct competitor with iCloud. Meh.

I'd rather they work on an end-to-end encryption system with a focus on ease of use.

Hacker News is a tiny audience. I don't see any business sense in focusing on features that 0.1% of users want.

Not to sound self-righteous, but the HN audience (or more plausibly, some subset within) has a high influence on what the rest of the world sees and wants.

Any Dropbox employees read HN?

If they did want to serve this audience, they would actually want to make it easy for someone (possibly the employees themselves) to build a Free encryption layer on top of them.

I think this is exactly the direction they should take.

Grammar check: "These days, your app need to store and sync more than just files."

Why would I use this over something like Parse?

I haven't looked into it but it may be cheaper. That being said, both Parse and Azure support more platforms than this offering. I think that even Firebase has an service in this space as well.

Reply on news.ycombinator.com