Amazon Cognito
aws.amazon.com
aws.amazon.com
A different (and more disruptive?) approach to this would be to put more control of these things into the hands of end-users such that they provide the 'backend' into which you (the developer) load your application. i.e something like the app-store model but it connects to the user's 'personal cloud' (or the desktops of old - if you prefer). Such a system needs to be FOSS at its core but with a way for developers to get paid for providing value to end-users (who themselves get control of their data/networks).
Of course, this won't happen overnight, but the alternative is that everyone ends up using proprietary silos, with huge lock-in and innovation suffers as the tech giants get distracted with lawsuits.
I'm working with others on the distributed systems infrastructure we need to make this possible [1], so that we can get to a place where everyone can have their own piece of the cloud. In fact, one of the major components of this is Mirage, which has been discussed quite a bit on HN recently [2].
[1] http://nymote.org/blog/2013/introducing-nymote/
[2] https://news.ycombinator.com/item?id=8020125 and https://news.ycombinator.com/item?id=8005130 and https://news.ycombinator.com/item?id=7726748
It also adds an extra step in the data storage process within the app, which increases friction and seems like it would be hard to get developers to adopt. All that said, I'd love to be proven wrong and see an implementation that is killer enough to solve these problems.
We don't need most users in the beginning. Just those that are technically savvy enough to understand the landscape and the way the world is headed. From there we can develop more stuff and eventually get it into a state ready for the masses. For example, I can run my personal website as a unikernel [1], which is not suitable for mom-and-pop but great for anyone currently using any static site generators. As more libraries get released I can build on that an eventually run more of my infrastructure this way (my prototypical examples are mail, contacts and calendars).
In terms of the friction for developers, we simply have to make the libraries and tooling fun and easy to use. With well-designed systems it's not necessarily any more friction that what developers have to do now (i.e build their own backend).
> "All that said, I'd love to be proven wrong and see an implementation that is killer enough to solve these problems."
We're working on it.
[1] http://amirchaudhry.com/from-jekyll-to-unikernel-in-fifty-li...
Futurists have a tendency to imagine a world of changed human behavior and it's compelling to do so. The reality is that the future rarely arrives as sweeping change, but rather as metaphor and specialization.
Whereas you can imagine others adopting new patterns of behavior because you understand the underlying reasons why such behavior is reasonable, the metaphor through which you explain this change is not readily understood. Why, as a User do I want this? If the answer is control and privacy, you might be barking up the wrong tree (time and again we've shown that those are not things consumers want or are willing to pay for).
If you want to drive dynamic change in the world, you have to change the underlying structure of complicated systems while steadfastly avoiding changes in user behavior. It turns out this is quite hard.
I applaud your efforts but encourage you to avoid the rabbit hole of endless specialization and to improve the marketing metaphor/rhetoric.
"Bitcoin, billions of phones, and NSA eavesdropping are among trends that may eventually result in cloud successor: data-local computing"
CITATION please. I think we have never seen such things. We've not had studies that control for all other factors and then conclude that users don't want control and privacy. I think your claim is flat out wrong. You're taking the fact that most users won't sacrifice by using a lower-quality, obscure thing for privacy and control, and leaping to the baseless conclusion that nobody cares about these things. I think you're completely wrong.
I would however, anecdotally highlight the progression of social services from places of privacy to places of publicity. One could argue that sites like "The Well" were the original social networks but they did not grow (whether by choice is of course a matter for debate). Over time, each successive social startup defaulted to a more open stance and scaled an order of magnitude in users with each step. The trend suggests that all things being equal, users will say they want privacy but want the features publishing and syndication provides (voting purely with their registrations and advertising value).
I agree that privacy has value, but I would argue that the anecdotal evidence suggests that users want less private services (and yes I'm aware that correlation is not causation, but I think this observation is apt). This is not law or dogma but rather a thought and I'm sorry if it was communicated poorly previously.
I don't think that's a clear-cut anecdote. Facebook does actually feel very private, because it's isolated from Google and outside users. Look at the number of female profiles where everything but the name is hidden from non-friends. I don't remember MySpace or blog users ever being this paranoid. And then, anecdotally, many people around me have started to live in WhatsApp/LINE gossip groups, which is something that could easily be decentralised.
Twitter was unambiguously non-private from the start so there were no broken expectations.
I'd be wary of assuming from this data point that people are willing to be even slightly inconvenienced for it. Just take a look at Facebook: the entire history of the company has been one of disrespecting and subsequently pissing off users, and yet out of the huge amount of people I know who have repeatedly expressed anger at how Facebook treats their users, I know exactly one person who stopped using it or even reduced their usage.
I'm not sure whether your assumption that people's concern will translate into even the slightest bit of action is sound.
For example, think of the number of people who maintain their own static websites, either on GitHub or EC2 or elsewhere. All those people could just use Tumblr or Wordpress.com, yet they chose not to.
I think the "personal cloud" that makes sense both architecturally and in the minds of consumers looks very different from what we as developers understand it to be. When we took our software and began prototyping it on hardware devices (like NASes) and showed it to people their eyes lit up. They understood it. Finally.
[1] https://www.kickstarter.com/projects/jmathai/openphoto-a-pho...
[2] http://blog.trovebox.com/post/36911180540/all-cloud-storage-...
This has a simple value proposition: Apple charges $100 for incremental 16/32/64 GB mobile storage. Users can buy 2TB worth of wifi storage for $200 and replacement firmware+source (http://www.hackseagatesatellite.com/) for $35.
Most of these NAS devices run some version of Linux which makes it trivial to set up our software on them. Some are easier than others though.
Beyond that I can't say much regarding what we've been tinkering with internally but it's pretty damn cool :).
This is the first time I've heard of this particular Seagate drive but the notion of having a drive you can connect to over wifi and which also fits in your pocket or bag is really interesting.
The reasoning that users do not want to maintain that infrastructure is a very valid one. However, it is like having a car vs. using public transportation. Most car owners do not maintain their car themselves, but at least you can remove your personal items from the trunk before you drive it to the mechanic.
Now the question is does this analogy extends to the advantages of using your own car vs public transportation. My guess is that need for privacy could be one big driver, but the company that can make home computing infrastructure an item of conspicuous consumption will be a big winner. Think Apple or Sonos, but in terms of what comes next (Augmented Reality, Voice HCI etc).
Without funding, they don't have the resources to build truly compelling products. You can prototype something in your spare time, but to polish something to be truly competitive outside enthusiast circles you need to dedicate serious time to it.
Even if users did care there is no mechanism for them to pay for it, so it doesn't happen.
How do you get grandma on her own personal cloud?
User Dependency Injection!
Sandstorm = packaging & separation of web apps from policy-managed data
Nymote & Mirage = UK systems research (OS, storage) on strong enforcement of claims similar but not limited to those being made in the above two projects
Based on a brief skim of the Mirage docs, it looks like a really big difference between Mirage and Sandstorm is that Mirage wants apps to be written in OCaml based on a custom API whereas Sandstorm builds of Linux. Any server app that runs on Linux should be easy to port into Sandstorm, but it seems like a lot of things will need to be written from scratch on Mirage.
In fact it appears Mirage intends to replace the kernel entirely. I can certainly see the appeal of dumping legacy baggage, but it's going to take quite a bit of work to reach the point of practicality.
All the code is also fully open source under a permissive license [2].
They also had some hack whereby large common files (like app executables) where compressed using their hash code or something, but that is not the important idea in my opinion.
I think having that transparent local/cloud duality is one piece of the puzzle for a service which is both fast and local and also cloud based. You shouldn't have to care where your data lives. The other pieces of the puzzle are how to grant fine grained shared access without confusing users and introducing security holes.
I can definitely see the argument that some mobile games only need to store/sync some basic state information on behalf of the user, but I just don't see it as being that simple for most other apps.
I suppose this datastore-only backend model works for apps with users who are completely isolated from each other, but any time you want to allow users to interact with each other (or provide rules for how their resources interact with the world outside their little "user bubble"), you need business logic that resides somewhere other than the app; otherwise, your business logic (which may affect more than just the authenticated user) is in an untrusted environment. Not only that, but there are huge scalability problems associated with using a logic-less backend storage model if you want to do anything that involves more than just a single user's data.
Even for games, something as simple as providing a leaderboard is impossible (or extremely impractical) with this model. If you have no server-side logic to perform aggregate operations over multiple users, the only other option you'd have is to basically grab all the data and perform those operation in your app, which is absurd.
Backend development, server management, and deployment can be hard (though many PaaS solutions are making it easier than ever), but personally I'd never recommend building an app with a logic-less backend data store like this. Even if you don't think you'll need it today, in most cases you're going to need logic on the backend eventually. If you've started with this, you're just creating a lot more work for yourself when you finally do need to implement a feature that crosses the line from isolated-users-only to users-that-can-interact (or even just obtaining aggregate user data for business intelligence purposes).
instead of having to worry about building and managing a backend solution to handle identity management, network state, storage, and sync.
From my reading of this, this wouldn't in any way obviate the need for a backend system of your own. It just does the identity management, network state, storage and sync parts for you. Your own business logic on your own backend can use this to jump start you.
> Building the backend... is a lot of work. You have to build it, deploy it, and manage the infrastructure that it runs on."
To me, it certainly sounds like they're trying to market this as an alternative to having your own backend system.
And to be perfectly honest, if you do use your own backend system for anything, authentication and simple data storage are not hard to add - especially if you're going to use a SSO service like Amazon, Facebook, or Google (which Cognito uses). There are drop-in solutions for most of those things that require little more than a couple lines of app code to authenticate users. I really just have a hard time seeing the value in what Cognito offers.
Also, assuming you do just use Cognito for the auth and storage/sync parts, how do you plan to use that data in the rest of your backend system? Are you going to pull down all that data from Amazon's servers just to do what you need with it (will they even offer that ability)? If so, how do you make sure the rest of your backend infrastructure stays in sync with Amazon's user data? It just sounds like it creates a lot more complexity than actually solving anything (except in the most basic scenarios without a need for a separate backend system).
For your example of a leaderboard, Parse has Cloud Code. It's designed to do exactly what you describe -- define a function that sits on Parse's server, handles client queries against your data, and returns the relevant data. Their example is for computing the average rating for a movie from your users' reviews: https://parse.com/docs/cloud_code_guide#functions
Amazon, Google, and Microsoft aren't there yet, but I can only imagine it's on their roadmap. And Parse's implementation only works for fairly easy stuff. But I think things will look very different in 5 years.
And, as you say, if these solutions mature, we might have some very powerful stuff in this space in a few years... maybe almost as powerful as just having your own server-side code running on some backend servers ;) But at that point, we're right back to where we are with traditional/custom backend systems being run on cloud server providers (which is not a bad thing!).
It just kind of feels like we're trying to re:Invent the wheel. You can offer these kinds of "a la carte" services all day long, and they're nice (on the surface) as long as they play nicely with each other in an open ecosystem - e.g., "let's grab this identity provider, that storage solution, this analytics engine, etc." It might look similar to Heroku's "add-ons", but with components that perform more specific functions. I just don't see that openness coming from Amazon, Google, etc. as they will all want to lock people into their services (which is understandable from a business standpoint). And unfortunately, anyone who buys into partial (and proprietary) solutions in the meantime are - in my opinion - just shooting themselves in the foot by limiting the use of their data.
Even in a perfect world with a vast marketplace of plug-and-play, interoperable cloud components, I would ask: is the "soul" of your app's code really captured in a conglomeration of components like this? If you have a need for a backend with custom logic of any kind, I think you're probably doing something unique (hopefully anyway). If you're going to need some real, unique business logic that can't be offered as an abstract cloud component, then I strongly suspect you're going to want more control over all of the backend. Even Parse's approach ends up looking more and more like a traditional custom backend system as you add more power to the "Cloud Code" approach.
As a programmer, you can often find libraries and drop-in components that accomplish 80% of what you need, so you write some custom code to "glue" those components together. That custom code usually contains business logic that really couldn't be abstracted or offered as a drop-in component. Where does that custom code live in the "perfect" ecosystem of plug-and-play cloud server components? How do you control the thing which makes your app/backend business logic unique? I don't think it's going to be in the client/mobile app as some people tend to think.
Maybe there's a good solution that combines these kinds of components with custom (and flexible) server-side code - I just haven't seen what it looks like yet. If I tried to use this stuff today, I'd be kicking myself tomorrow as I figure out that it's a pain to juggle data between components that I don't control and components that I do control.
I think we'll start seeing more "opinionated" platforms, similar to what Rails and others have done for web development. Parsing query strings, managing database connections, and wiring your own user reg code are all distant memories for most web developers most of the time today.
We might see something similar here, where groups of common functions or services are stacked together to form a unit that works well together and covers some set of use cases very well. LAMP comes to mind as a comparison, and you can definitely use SQL Server instead of MySQL, but it probably won't work as well when it comes down to drivers, libraries, recruiting, etc.
These stacks will differ as well to serve a broad range of customers. I cringe a bit when I see Amazon EC2 comparisons to Hetzner or Digital Ocean because they are serving quite a different set of customers with different needs. To think that all hosts should end up standardizing on the same units would leave many customers unserved. It's easy for innovation to be called vendor lock-in, but it's hard to compete or innovate on heavily commoditized products/services.
It's [still] an exciting time to be in tech.
The solution we've come up with, which we'll hopefully get to open-source one day, is to model state over time as a persistent "event log", which means instead of syncing documents we sync events that clients perform, replaying events instead of document-update statements. This approach is infinitely flexible I believe, but requires more work if your clients are written in different languages (we use JS throughout so that isn't an issue).
As far as I can tell, there is no guarantee anywhere that they'll maintain any service for any length of time.
If that happens, it could waste weeks or months of engineering time trying to migrate to something else, or I'll just have to shut down my own services if I decide it's not worth migrating.
Amazon doesn't just create services, throws them over the wall and see if they stick. Rather, they almost always RESPOND to customer requests. Dr. Werner Vogels, AWS's CTO makes a point about that approach at about any keynote he gives. Similarly, they try to also avoid feature creep by only providing features which customers really ask for.
I think I remember amazon people saying they are actually dogfooding this stuff so it should stay available for a long time anyway, rather than being a random Google Reader.
Retiring any of the services that have been launched would cause not inconsiderable pain to other teams :)
I think there's a notion of persistence in AWS' leadership that propagates through the teams. AWS is willing to fight through the early painful years for products to see them arrive at the stage that EC2/S3 have attained.
Sitting here in Seattle I see Amazon hoovering up all the local talent they can hire. And I see them putting up new tower after new tower to house them. I don't worry about them stretching themselves too thin as far as manpower goes.
And they don't have a history of closing down AWS products – for instance the first, (and not a runaway hit) Mechanical Turk is still with us.
I also wonder how they "manage[s] the complexity of conflict resolution" without manual intervention.
That's why I always advocate for building your own backend, if you have a long-run plan and/or expect to generate a lot of data for the app.
See you at the race to the bottom!
Besides, what makes you think you would be prevented from moving all this to another provider? From the FAQ (https://aws.amazon.com/cognito/faqs/), it seems that Cognito data is stored in a straight-forward Key-Value scheme. That's about as easy as it gets when it comes to data structure.
Moreover, the actual service itself would almost certainly be implemented differently with another provider. Why should Amazon, or anybody else who offers similar "menu of services" cloud computing, make it easy for you to do so (they make money off of you not moving)?
This is one of our core values and while we get odd looks from investors or MBA-types we know that it's the right move to make in the long run. This is the way things are moving and I hope that more companies will go this route.
Sure, "barriers to exit become barriers to entry" is applicable in some contexts, and for some part of a potential market it will be the barrier that keeps those customers away. I wouldn't dispute that. I would dispute that "barriers to exit become barriers to entry" be treated as a universal maxim applicable along all businesses and products. It's just as much of a strategy mistake to make as assuming the barrier can't exist in the first place. It ought to be considered; not treated as a key issue until or unless the data suggest it should be.
That's a pretty big assumption to make. People, even technical people, make assumptions all the time that they'll be able to get their data out of a system and then aren't able to at a future date.
Any other OSS out there for this?
Any feedback on it?
The big difference I am seeing here is Amazon Cognito doesn't yet have a story around syncing documents between multiple users/roles. Sync gateway has the notion of a sync function that providers developers with a hook to customize server-side syncing in a deeper way.
I'll be interested to see how Amazon developers the multi-user/role sync capabilities and integrates it with their native identity management solution. Could be very compelling down the road..
As far as Hacker New goes, we announced EnduroSync back in May, and got no promotion. Not one up vote?
"Parse"? OK, sure...but that's not any more functionally descriptive than "Cognito"...and Parse is a more narrowly focused company with a primary service...In the case of "Cognito" and "Zocalo"..., Amazon has to come up with names that are distinctive because they provide so many core services...so many that even me, an engineer, get them confused. Imagine the difficulty of a fly-by marketing person trying to differentiate between similar sounding names.
Furthermore, these names are distinctive industry wide...presumably, if Zocalo becomes big, you will no longer have to say "AWS Zocalo", but just, "Zocalo"...just like you do with "S3" and "EC2".
To drive home the situation that Amazon is in, let's consider another great product name: Dropbox. Unlike Parse, it is very easy to relate to the core service. But consider the phrase, "Amazon Dropbox"..."Dropbox" is unique compared to the currently named AWS services...but it has a huge collision with Amazon's core service, which is, stocking and shipping products. If Dropbox.com didn't exist, and Amazon used it instead of "Zocalo"...there is a considerable risk that people associate "dropbox" with some kind of packaging thing.
The http://en.wikipedia.org/wiki/Z%C3%B3calo is the city center (of Mexico City's historical district), a simple and untaken metaphor for a service that "...provides users with a central location for both the documents and files they are...".
It's a shame Amazon can't afford to hire a marketing genius like you, but I think they're doing the best they can with the idiots they have to work with.
Not really the API you're looking for but I thought it was worth mentioning.
http://www.kinvey.com/ https://backendless.com/ https://www.parse.com/ http://www.baasbox.com/
imagine a game with 100k DAU (big, but not huge)
2 sessions per user per day
say 10 "saves" per session
100,000 * 2 * 10 = 2 million synchs / 10k * $0.15 = $30/day
for $900/month you could do FAR better just rolling your own
these numbers are all reasonable for a mobile game... so it seems to me that the main use pattern would be for apps that don't need to synch very often. and if an app doesn't synch very often, why even bother throwing an auth wall at your users?
I find the aws pricing (and "cloud pricing" in general) fairly confusing, and the things is that you have to be so careful that the wrong checkbox isn't checked or something that opens the spigot to the credit card# you had to enter to try out the service on the free tier in the first place. Note that I saw that you can get an alert when it first charges anything via billing alerts (http://docs.aws.amazon.com/gettingstarted/latest/awsgsg-intr...), but still...
There are many cost/efficiency arguments against outsourcing infrastructure. I dont think this is one of them.
What would be really impressive is if there was an option to client-side encrypt the data before sending it to AWS. Of course, that would mean you'd have to move your syncing logic to the client-side, too, but having a good client-side encryption option would be a real differentiator.
So if it works like that, you absolutely can encrypt your payload client-side, and use Cognito as a mere transport.
Exactly. I don't know why people are so attracted to locked in solutions like Parse or now this when Couch has already been doing it for a long time, and there are mature third-party providers (Cloudant, Iriscouch, etc) that can help you get off the ground just as easily without ever being locked in with them.
I watched the video and the focus of this service seems to be on storing user data in a way that can be accessed by any device. I'm an API developer, isn't this just what every mobile app that uses an API to store data does, right? I personally don't write a different backend datastore for every device type that could connect to it, nor do I 'permanently' store data on the device! user data is always synced 'for me' over an API. I don't get this, can someone explain why we need this service? Do people develop apps different from me? I'm not being sarcastic, I'm genuinely interested.
The data pulled from the backed is handed off to the front end. The front end determines what's displayed and how it's displayed based off of the device. What does data storage have to do anything.
Instead being the vanguard of innovation, they're letting other companies validate the need for a service before swooping in.
Who's next? Probably Stripe.
For Amazon Cognito, the answer appears to be "yes, until all of your users together store more than 10GB". ie. until your app gets popular.
So for free open-source apps, I think I'd prefer to stick with Dropbox or Google Drive, where the billing is associated with an account the user already has (which most users can operate entirely within the free tier). That way you don't run into a situation where you are a victim of your own success, by exceeding a free tier that applies to the sum of all user data in your app.
So this service seems to be geared toward developers who don't want to store or see user data at all. But for a developer that wants to gather user data (for crowdsourced insights) or for a developer that wants to offer a web app connected to their own DB, then this isn't going to work.
Ancillary factors:
1. The title is clickworthy. While link-baity titles do work well on HN (compared to very vague titles)...this title sticks out because it is two words: One of them is "Amazon", which has very immediate connotations and brand recognition. The second is a proper noun. Those of us who haven't heard of "Cognito" are likely to think, immediately, "Whoa, new Amazon product, must click-through to see how it will impact my work". That it is two words also makes it visually distinctive regardless of what those words even are.
2. It's 11AM, which means almost every developer on the East Coast is awake, and West Coast developers, particularly the ones responsible enough to reserve HN viewing for before-work hours, are getting their coffee and checking HN. So, lots of potential upvoters.