Shelter Protocol: End-to-end encrypted, federated, user-friendly web apps
shelterprotocol.net
shelterprotocol.net
Like the SPMessage format here too, uses JSON.stringify to indicate structure encoding, which necessarily implies that any implementation of the protocol is bringing along needing to handle JSON semantics. JSON does not seem to be well-suited here as it doesn't composing cleanly in the way a data structure defined in terms of bytes would.
Indeed, this has already created considerable problems for the request and event signing and getting different Matrix implementations written in different languages to interoperate correctly. I now firmly believe that cryptographically-signed JSON is always a disaster waiting to happen, no matter how hard you try to canonicalise it.
Which is strange because JavaScript has had the Uint8Array that makes manipulating arbitrary byte structures easy for a while, it doesn't feel any less natural than other dynamic languages in that regard. Defining a "signed message with a header" structure in terms of bytes like I described above seems like a no-brainer. If you want the payload data of your protocol to use JSON then that's fine, but that shouldn't dictate the cryptographic layers underneath.
We've done our best to make sure that the way we're serializing objects can either be done in a consistent manner across implementations or, if done inconsistently, it doesn't matter.
The consistency is really an important part here, are you spending less time doing the testing and bugfixing to ensure that JSON ambiguity is not possible / is harmless than you would writing serialization for cryptographic types in unambiguous byte-oriented structures?
Nailing all that down so you can e.g. have a truly deterministic signing process often requires custom encoders/decoders and is rather fraught with problems even then, at which point why even bother with JSON. Use something designed to be consistent.
But a healthy ecosystem of clients never emerged. When I looked into why, I discovered it relied on node's particular method of JSON serialization[0]. Are you confident that the shelter protocol has avoided this pitfall?
However, thinking on it more, there is one thing Groxx mentioned that has given me pause, and that is "Numbers don't have a specified precision". This could be an issue in the future if, for example, computers suddenly use 128-bit numbers and JSON implementations start to actually use such numbers. It's an edge case that might result in incompatibility with clients that use 64-bit numbers. So we'll consider the possibility of changing the serialization format after researching this more.
I asked Brendan Eich if JavaScript will ever change the way it represents floating point numbers (from 64-bit IEEE 754 values), and he said no ("don't break the Web"). And to represent larger bit values in JSON you're supposed to use a JSON string and schema to specify the type. I'm not sure exactly what he means by that, but I assume it means something like using JSON keys like "<BigInt>mykey" and string values instead of number values. Then manually parsing the string into whatever native 128-bit type or whatever representation you have.
So - JSON still seems to be the favorite here, but I totally get the anxiety around it because of how under-specified it is. If you avoid re-serialization nothing should break.
https://bnfc.digitalgrammars.com/
I agree, having/using a well-defined standard that can have automated tooling to general language-specific implementations would be ideal.
https://shelterprotocol.net/en/federation/
"Federation: Coming Soon!"
And I'm seeing a lot of "under construction" and "coming soon" in other parts of the protocol which seem critical to functionality, to the extent that I can't see how this protocol is even meant to fit together.
However, the core of the protocol is stabilizing and I can answer generally as far as missing sections go. Regarding federation, instead of publishing events to the server you're connected to, you publish events to a different server. Everything else works pretty much the same.
Would it be accurate to say you're trying to get broader adoption of this idea, but currently the only implementation is in your group-income repo?
I ask that with an especial emphasis on the zero knowledge password system which is currently undocumented but, if I'm understanding correctly, also unimplemented? https://github.com/okTurtles/group-income/blob/3e50642ab8866...
Maybe you want to build a federated application yourself, but want certain features that some of these protocols don't have. Well, now you know a little more about what's possible in this design space. Perhaps you've already settled on a protocol (e.g. have started building on ActivityPub), but want end-to-end encryption. Well, now you're aware that you can in fact add it in to your existing ActivityPub-based app. For example, there's no reason why Mastodon couldn't use Shelter Protocol to encrypt DMs between users. Shelter Protocol is very lightweight, and can be an add-on to existing apps.
Regarding the ZKPP, it is implemented, but like the rest of the code is not yet separated out from that repo. That is one of the next steps we'll be working on, a standalone developer SDK.
We are also welcoming early feedback and contributions from those who would like to participate.
I'm not sure how that squares with the
// TODO: Insert cryptography here
That one’s easy, it’s just (likely augmented) PAKE. There are a couple options in this space. The zero-knowledge stuff likely comes from the blind salt (OPRF, Oblivious Pseudo-Random Function). There are different variants to chose from, with slightly different security & performance trade-offs. This stuff indeed allows password authentication without ever revealing the password to the server, but the password database itself is still vulnerable to dictionary attacks once it’s stolen.
What worries me though is how they failed to mention PAKE. I’ve seen much of their talk, and to be honest it smelled like snake oil. Very shallow explanation of the protocol, and many missing steps between that and an actual use case. It’s supposed so solve many Big™ problems, but how it achieves it is very unclear to me.
The way that it works, at a high level, is similar to how SRP works. Two random salts are generated (let's say, A and B), where A is used for authentication (and hence public) and B is used for deriving the other cryptographic keys.
When you authenticate, you retrieve A and then you prove to the server that you know what scrypt(A, password) is. At this point, the server provides you with B, and you can use this information to derive scrypt(B, password), which in turn you use to derive other cryptographic keys.
It being an oblivious password store, there are other steps taken to make the protocol stateless (from the perspective of the server) and to make parties commit to random values used so that runs of the protocol cannot be replayed.
> but the password database itself is still vulnerable to dictionary attacks once it’s stolen
This is correct, and I'm not sure there are good ways to prevent this or equivalent scenarios from occurring. Therefore, you should see this as an additional layer on top of your already secure password and not as a substitute for a secure password.
The reason for having this mechanism rather than not having it is to protect your password from brute-forcing by the public at large, in a scenario where the server operator is semi-trusted. Without this implementation, you have three alternatives: (1) Forego passwords entirely; (2) make the salted password 'public', along with the salt, which is all the information you need to brute-force it or (3) make more "normal" authentication flows without PAKE, in which case you need to trust the server even more. If you insist on using passwords, this is a compromise solution between (2) and (3), i.e., between anyone can break it and trust the server entirely.
[1] <https://github.com/okTurtles/group-income/blob/e2e-protocol/...>
[2] <https://github.com/okTurtles/group-income/blob/e2e-protocol/...>
I believe there isn’t indeed, sorry this part came out as a criticism.
I actually have deployed a PAKE at work for a corporate CRUD app once, and the entire security hinged on login/password. Clients authenticate to the server with the password, server authenticates to the clients with its database entry. Sure the password could be brute forced if the databased leaked, and sure anyone could impersonate the server with a database entry, but this reduced security allowed simpler and more convenient administration: no need to bother with a PKI, just take good care of the password database and reset everyone’s passwords when we suspect a leak. (Now if this was for the wider internet I would have added a PKI layer on top to prevent server impersonation.)
> The reason for having this mechanism […]
Yeah, PAKE is real nice. Ideally every password based login system would use augmented PAKE under the hood. Not only does it protect the passwords better, the protocol itself doesn’t need to happen in a secure channel (this can help reduce round trips), and the bulk of the computation (slow password hashing) happens on the client side. This reduces both network load and server load, what’s not to like?
For the blockchain skeptics, good news, it's not a blockchain: it's a distributed virtual machine without waste-heat-enforced-global-consensus. Of course it's much faster to see "smart contract" and hit the back button than investigate a new way of doing things so Shelter has their work cut out for them.
We'll look into fixing this!
EDIT: should be fixed!
I think I'm not following how the checksums are produced (are you going to serialize and replicate the code too? seems odd if you're going to include Vue in that serialized data...), nor how that does much at all different than an append-only log would achieve (maybe you can compress kv-sets and discard the history after some time?)... but docs are somewhat incomplete so maybe that's just not covered sufficiently yet. Or I skimmed too quickly and missed something.
Regarding storing Vue (or any other frontend framework), the answer is gonna get a bit technical (sorry, we haven't documented this on the website yet). This is a curiosity for how we happen to be using Shelter Protocol in Group Income (which uses Vue). Vue.js both is and isn't saved to the contract. Each contract manifest has two versions of the contract: one "complete" contract containing all of the code necessary for reproducing the state, and one "slim" version that has any large dependencies passed in to the contract at runtime. Group Income uses the slim version so that the contracts load quickly, but it's also possible for anyone to reproduce the state of a Group Income app independently, without using Group Income, by loading the "full/complete" version of the contract because all necessary dependencies are bundled in.
Yeah, that's definitely not clicking in my brain at the moment. I guess I need to actually watch/listen to that video rather than skimming the slides.
For storing code: yeah, I guess a more "storage polite" version of this would be to serialize the "engine" of whatever you're doing, rather than its presentation. That can often be quite small, certainly in comparison. Would it be possible to simply store e.g. a git repo name + sha, and then clients download as needed? Or do storage-providers need to be able to execute any of this (beyond enforcing key permissions)?
The server stores all of the events and all of the contracts in a content-addressable way, in whatever key-value database it is using. It does not execute any of the contract code though, no, that's done locally by clients.
I suspected that was going to be in there
With the exception of the inefficiency of proof of work, nearly all the problems with cryptocurrency are human problems related to the toxicity of the ecosystem rather than intrinsic issues with the tech. Code doesn't scam people. People scam people.
And honestly, the vast majority of people on the "inside", i.e. those actually working with it, were opportunists as well. Most of them saw the tech narrowly as an unregulated financial instrument.
its like theres no amount of Adam Curtis documentaries that will shake silicon valley folks from the myth that the computer will lead to a better, more equal society.
you cant compute yourself out of a broken world.
Most certainly not, but surely you can build better tools with the aspiration of facilitating certain goals, can't you? It's not the tools in or by themselves that will improve (or worsen) the world, rather something at your disposal to pursue your goals.
> the myth that the computer will lead to a better, more equal society
Agreed that it won't. But, IMO, the strength of Shelter is that it covers a niche that many other systems (blockchain-y or otherwise) don't, which is data autonomy and confidentiality. Most popular web apps today are centralised silos that don't give you privacy from the operator, and those that aim for federation often also don't give you much privacy either.
Now, it can be that those factors are not important for the specific thing you're developing, and that's fine. But, if they are, having an existing framework to build on top of can give you a head start (even indirectly, by showing you what works or doesn't).
Disclaimer: I'm involved in the development of Shelter. All opinions are my own.
As an Adam Curtis fan (for all his faults), I don't believe technology is neutral nor that progress is teleologic. I do believe that people could be better served that software that works in their interests instead of against them.
And funny you mention a broken world, as if we're doomed to be excluded from the paradise of eden, the very first walled garden. Those of us working on distributed applications are trying to make walled gardens obsolete, no forgiveness required :)
thats it. thats all it is. all the madness about cryptography replacing trust just makes that power more concentrated.
But on a broader scale, I don't see what in cryptography makes power inherently more concentrated. Crypto is just a way for enforcing certain trust relations that have already been established or agreed upon. Just like you can use crypto to help centralise power (e.g., allowing you to only run signed applications that can only show signed content), you can use crypto to help decentralise power with tools for confidentially presenting content and allowing you to vet your applications haven't been tampered with.
In both cases the underlying technology has many common components, and what changes is the use you make of it.
blockchains have little to do with that though, they're more of an easily-validated data replication technique than anything that has social implications. and they've been around for much, MUCH longer than bitcoin: https://en.wikipedia.org/wiki/Merkle_tree
Is it? From where I sit it's an environmental catastrophe as we burn squillions of CPU/GPU cycles for the modern day tulip craze (bitcoin).
For any purported use case of blockchains, with possible exception of buying illegal things online, existing technologies are better.
An immutable ledger != bitcoin.
Responding to the nonsense introduction:
> By design, traditional web applications enable server administrators to monitor all user activities.
That's a choice. What is the compelling reason for a company that wants to monitor the use of their apps to use a system that ostensibly says you can't?
> Although these web apps offer “privacy settings” to users, they fail to provide any real privacy protection.
Yes, because the options are either the data is inherently insecure, or the data is fully encrypted. Governments and users both have difficulty with this concept: you cannot have data security and also backdoors, you can't have data security and also "I have lost every component of my account identity: devices, passwords, and passcodes, but want you to recover my data".
This is ignoring companies for whom "privacy settings" are an intentional lie (Facebook, Google, ...), and again, why would such a company adopt a platform that ostensibly forces lack of spying?
> Shelter Protocol introduces new ways to handle logins and data storage on the server while preserving the conventional username/password experience that users are familiar with.
The username/password system people are familiar with is widely understood, and clearly demonstrated, as being bad for security.
> Instead of storing data in a database in clear text on the server, data can now be end-to-end encrypted and synced across multiple devices, and even across servers operated by different individuals.
Already completely doable, and the companies that don't do so have chosen not to, for a variety of reasons - some good, some bad, but those reasons are not because encrypting content securely is hard.
> The Shelter Protocol (SP) defines operations for a high-level, lightweight, federated, end-to-end encrypted virtual machine.
Or you can use JS, which is already available, runs on every machine that exists at this point (is this good?), is already federated: any device can run any JS you send it.
> [remainder of front page]
Largely nonsense.
* Key concepts *
> Since every action in SP is signed using a user’s private key, which in turn is derived from their password
So it's bad crypto. Huzzah!
After this I got bored reading this nonsense.
There's no actual justification for why this magical VM is necessary or good, nor any explanation of how they're going to make it "federated" (because despite advertising federation, it does not appear to be), what they consider federation to be, or why that is good.
Their one example app does nothing that requires any of their advertised features - literally every part of this could be done with existing web tech, and largely be done better.
User friendly.
Pick one.
Federation is mistake. IRC is dead, Usenet is dead, email SHOULD be dead.
Federation sucks and causes no end of issues.