326 karma · joined May 27, 2014
Absolutely, few people really get this. Even those that do get it generally don't know what it looks like in practice because it's so rare.
In case you're curious about what it looks like in practice (at least one way), we presented direct user code compilation and verification[1] in Jan for our smart contract language Pact[2].
The idea that you can write a doc test that triggers a formal verification about some aspect of your code is, for lack of a better term, strange and yet the power to weight ratio is just off the charts. For the first few days that the system started working (I built the FV portion, Stuart the language + H-M type system) I thought it was broken... turned out we just had a bug in our demo code that we never noticed.
Usually, tests check the things you know to check so when they find something it's not a surprise what they find. Sometimes you can fuzz and randomly find something, which is cool but doesn't guarantee that there isn't a problem somewhere else. However, FV just feels uncanny (as a dev) because the test either finds a problem in the code -- and tells you how to replicate the bug -- OR proves that a problem cannot exist.
[1]: https://youtu.be/Nw1glriQYP8?t=1071
[2]: kadena.io/pact
Right now, the risk isn't that high as the worst case (added uncertainty breaks bitcoin completely) is very unlikely. Instead, (a) one side wins you still have the same amount of coins or (b) both sides win and you probably have +2x the value long term.
No one has hard-control over bitcoin, but the leading dev group has some manner of soft/non-coercive control (by momentum alone).
In a strict sense, you're right. All a smart contract language needs to run is a linear set of inputs. However, they have a lot less utility without the other parts of a blockchain (distributed, crypto-data structure for assurance, etc.). If you don't care about being distributed, just use an API or run the external code in a container/vm.
> they refuse to abide by the decision by retroactively modifying the interpreter to interpret the contract in their favor. And why wouldn't they?
While I don't agree with the hard-fork decision, I understand (from a biz POV) why they did it. They didn't have too many options. Either hard fork or let the "hacker" collect the prize. IMO this lack of options was a failure of the EVM and solidity language itself.
While the DAO debacle had a bunch of causes but the main problems, from a language point of view, were: (a) centralized governance (upgrading/migrating) of a live contract isn't a first class citizen in solidity, (b) there's no distributed governance mechanism at all, (c) formal verification of solidity contracts is still years away.
Pact has (a) already and we'll be upgrading it to handle (b) as well next month (via a weighted vote mechanism). Either of these would have at least been able to mitigate the impact of DAO's issues -- just upgrade the contract and migrate the table the contract is guarding. Pact also has (c): https://youtu.be/Nw1glriQYP8?t=1072
Blockchains, both public and private, are really just distributed DB's. You wouldn't fold a protein or price an option in your DB, you'd use some external system and save the results to the DB. The same line of reasoning goes for blockchains.
This is why we made our own language from scratch -- Pact is a bit like regular SQL + procedural SQL + programmable auth. It's not Turing complete and can compile to Z3 for formal verification. Given that every execution is a transaction, and it's interfacing with a normal DB, it's performance bottleneck is are the underlying DB itself.
I've built one of these types of systems (kadena.io). The key differences between a replicated DB, though let's use consul instead because it's closer to a private/permissioned blockchain (distributed) and the differences are the same, are two-fold:
#1 They are designed to run in a multi-administrative context (i.e. where you don't own all the servers). Imagine running consul where you only had direct access to 20% of the nodes. You could do it, but you'll eventually have problems as it's possible for a few misconfigured servers to slow/halt/collapse the cluster. The problem is that in multi-admin you can't directly fix the troublesome servers. The system itself needs to be robust against them -- you need BFT consensus (for this and other reasons).
#2 The killer feature of private blockchains is having a smart contract language. It allows you to write and run code that runs on my servers without me having to trust/audit your code. This is only possible because of the frankly ridiculous level of security that the system provides. It is also a fundamentally new feature. For a new language that's actually appropriate for serious work (which EVM-based langs are not) take a look at Pact on the aforementioned site.
Simply, if you want to be able to host logic and data, with the infrastructure distributed among multiple administrative units (businesses/firms) that don't completely trust each other, then you end up building a "private blockchain". You need a decentralized way to replicate new information (consensus), that is robust against bad nodes (BFT), that linearizes the new information (blockchain-like), that can execute logic on your machine written by individuals that you don't trust (smart contracts), that tracks/enforces auth at every level (PPK-Sigs everywhere!).
> "Problem: There is the PostgREST written in Haskell, not in Go"
Moreover, if you want to use something like PostgREST but need to extend it, and don't know Haskell, this is a completely legitimate problem. While I'm all for competing implementations, the pREST author's claim for why it's needed is inaccurate.
The "characters to clever/unmaintainable" ratios you achieve with k (which most kdb+ platforms end up reaching for at some point) are almost unparalleled. A famous example of how awesome/powerful/ridiculous k can be is the "4 lines of K" text editor: http://www.kparc.com/$/edit.k
I guess my point is that letting your guard down, for even a single line, can be orders of magnitude more trouble than it would be in most languages and for that reason I wouldn't base a new stack on it. Also, the developers for it are rare-ish and expensive + it has a serious learning curve for those unfamiliar with FP or Lisp.
If you like writing dense, clever regexs (which I do) then you'll love k & q. The amount that you can get done with just a few characters is unparalleled.
Which leads to, IMHO, their main drawback: k/q (like clever regexes) are often write-only code. Picking up another's codebase or even your own after some time has passed can be very hard/impossible because of how mindbendingly dense with logic the code it. Even if they were the best choice for a given domain, I'd try to steer clear of using them for anything other then exploratory work that doesn't need to be maintained.
import Data.ByteString (ByteString)
import qualified Data.ByteString as BS
import qualified Data.ByesString.Lazy as BSL
import qualified Data.ByteString.Char8 as BSCI suspect that may have something to do with it. Maybe Vermont has more initially out-of-state college students, or they represent a higher relative proportion? Also, keep in mind that NH is 2x the population of VT and is close to a major metro area (Boston, ~1hr away). I have friends that live in Southern NH and commute to northern regions of Boston for work.
I'd like to see a 20-year later study that looks into the number of people that went to VT for school (from out of state), went to another state for a career (not a huge number of options in VT) and returned late. Everyone I know that went to school there wants to do this; they love the place but couldn't find a non-service/tourism job post-graduation.
As for "a horseless carriage with a horse" perhaps, we'll see. These systems do give you something that you can't get otherwise which is a distributed DB that you can share with competitors without worrying about them stealing from you/breaking it. Blockchain is to overloaded right now, but it's the name that has stuck likely because "a Byzantine-Fault-Tolerant Consensus database" (BFTCDB) is just too impenetrable. If one of these BFTCDB take off, if may not actually be a blockchain (though I suspect it will be because cryptographic data structures are just too useful for them) but I bet it will be called one.
But that the end of the day, that's what a permissioned blockchain is -- a DB you can share, which is a seriously new capability.
The gist of this is that TLS doesn't give one a durable/persistable form of security vs ppk-sig's ability to verify that x signed y at any time. TLS is much faster than verifying the sig with every transaction on every node but after, say, a cluster restart the question of "who sent x" is answered via consensus and not cryptography.
There are a couple other issues too. Using ppk-sigs means that the mail clerk and the CEO use different keys to sign transactions vs TLS where you'd either have a logical "from:" field or a different connection for different users. Both of the TLS solutions have rather obvious flaws. Finally, a TLS based approach puts an attacker 1 server away from impersonating, potentially, an entire organization because the system using TLS cares only that it came via the correct pipe.
Overall I'd describe systems using TLS as not being paranoid enough about who authored a transaction, as PBFT/SmartBFT are by default very trusting of a client's transactions vs Bitcoin's model where everything is to be questioned/require independently verifiable cryptographic proof. Hence the tradeoff.
This way, if a bug is found, the admin can update the smart contract module guarding the DB table directly vs having to track down the on-chain contracts for migration (if it's even possible). From a normal user's pov, they don't see a difference (unless the admin decides to change the namespace or something).
It's a subtle but important difference. When someone using Pact sends the transaction `(payments.transfer "me" "you" <amt>)` the transaction (after doing a bunch of auth with ppk-sigs) looks up the `payments.transfer` command from the on-chain smart contract and executes it with the provided args. As such, the admin can update buggy contracts without issue (and roll-back, fix tables that have bad data).
For intra-organization, one way to view private blockchains is to look back to the days of mainframes and the shift to databases. While you technically never need a DB and can get things done with code, DB's are incredibly useful as a general utility because of the baseline features you get by using one. Private blockchains are similar, in that they give you some new features (e.g. sharable DBs) and are much easier to build with as they raise the baseline on how bad a solution can be that is built with them.
While you can have a pretty messed up DB schema, bespoke mainframe solutions can be much worse. Private blockchains are like that (especially when they have smart contracts) -- sure you can really mess up the systems setup but issues that DB's can have like high availability and disaster recovery are 100% commoditized. Also, they demand a higher bar for authorization making them generally as more robust the most hardened DB by default.
That isn't to say that a private blockchain will replace the DB as blockchains can be wild overkill for many domains. A small company that needs a DB doesn't need a blockchain's feature set, but a large company that needs a globally distributed DB for holding a ledger (of anything) would be well served by one. When you're working with a system that by default is BFT, incredibly paranoid about authorization, perfectly replicated and highly available then build a solution gets much easier.
As for the specific problem that they solve, it's the settlement problem: organizations currently hold internal ledgers of truth about what they own but they can't inherently trust other organizations' ledgers of truth about what they own, so we need 3rd parties to settle cross-ledger. Currently, institutions handle this role but private blockchains represent an algorithmic solution.
As for Tendermint BFT, it's one of the few that somehow I never got to dig into at JPM. Do you know if Tendermint BFT's performance & performance-vs-scale numbers are published? These are notoriously hard to come by and I'm always curious for more data-points on the matter.
I spent a year investigating blockchain applications for JPM so I've seen most of the permissioned blockchain solutions that are out there. Unfortunately, none of the solutions I saw were real blockchains. The more technical groups in banks (which GS has) surely realize this as well. The solutions I've seen consistently lack at least one of the following necessary features:
- Fast, Deterministic BFT Consensus (mining doesn't work as intended in private contexts[1])
- Smart Contracts (you need a deterministic language)
- Signed Transactions (TLS based authentication trades security & audibility for speed)
- Cryptographic data structure for transaction storage
There are other features you need, but these are the big distinguishers I noticed. While technically you don't even need smart contracts or fast BFT consensus, I believe the tech isn't useful enough to justify the migration costs without them.
Disclaimer: I'm a founder in the private blockchain space[2] and founded specifically to make an infrastructure that addresses these issues.
[1]: http://kadena.io/blog/MiningInPrivate.html [2]: http://kadena.io
Also, I would note that "with a consensus protocol" is very, very tricky... well if you want BFT anyways. Rolling your own consensus is on the order of difficulty of rolling your own crytpo -- this is experts only, don't do it. It's not quite as difficult as crypto but you still need to formally prove it which is very hard. We didn't make our own consensus, instead we used Raft for consensus and figured out how to make it BFT which is hard but nowhere near as hard as making consensus from scratch.
Here's my off-the-cuff guide to figuring out if you want a blockchain and what properties it needs:
Do you want a distributed DB?
* No -> postgres.
* Yes -> you need consensus of some fashion.
Will some nodes for the DB be hosted by competitors in your space?
* No -> You don't need BFT, use hashicorp's stuff.
* Yes -> You need BFT, welcome to "I need a blockchain" territory
Will competitors be able to directly write logic that will run on the system?
* No -> regular language is fine
* Yes -> you need a smart contract language to write in
What type of scaling and performance do you need?
* 3-100 nodes + high performance -> PBFT/SmartBFT for consensus is fine, but keep in mind that if you ever need to scale beyond this you're screwed (maybe you can get to 200 but I doubt it).
* 100-10k nodes + high performance -> email us at info@kadena.io, we're literally the only ones able to do this currently
* +100 nodes + low performance -> Mining based consensus could work, but you need to lock it down and centralize pending transactions a bit to avoid incentivization issues.
* >10k nodes + high performance -> wait 5 years, maybe someone will figure it out.
What do you need to do with the chain?
* track allocation of things/conserve mass -> no need for a smart contract language, adapt a bitcoin approach
* conserve mass + run logic -> you need some type of language, though maybe not a smart contract class one
Assuming you need to run logic...
* and don't need BFT -> glue python/whatever onto consul, be careful to only write deterministic code
* need BFT, and want to be able to fold proteins -> use EVM (though I'd recommend rethinking your approach... why should you replicate massive computations like this?)
* need BFT, and want the system to work a lot like a very paranoid DB -> email us
However, when last I checked, there are several operations that fall outside of this guarantee, e.g. new account creation, that are not rolled back on error. This would be as easy fix but to my knowledge is still unaddressed. Hopefully, they've changed this and if they have please let me know.
Edit: I stand corrected, thanks!
Pact should be O/S Sunday or Monday, we're putting the finishing touches on the web editor now and the Atom plugin is already out there[1] though without a binary to run alongside it there's not much to see.
The other deterministic BFT systems like PBFT and SmartBFT are a little bit faster for small cluster sizes but slow down for every added node, encountering full system locks before 256 nodes. It's worth noting that these systems generally do not do pubkey sig verification on every command, instead relying on consensus and direct replication via TLS for authorship guarantees. The drawback, of course, is that an attacker is one server (the client's TLS server) away from inputting fake commands that the system deems real + smart contract authority is handled logically and not cryptographically... we see the CEO and the Mail Clerk having different keys to be a huge benefit from a Bitcoin-like approach.
This makes the bottleneck for us crypto-verification. Verifying 8k-12k signatures per seconds is pretty tricky if you don't skimp out on the crypto, which a lot of vendors are doing in the name of speed.
For one, there's no Mining so what's the meaning of the initial coins? Also, it'd be much closer to an AWS Lambda + Stripe/Ripple than anything else which makes fee structures tricky. The big reason is that we would run all of the nodes ourselves because our core IP is ScalableBFT and we think that's the most valuable bit.
It's the first/only deterministic BFT Consensus that can scale into the 1000's of nodes + maintain constant performance as it scales. Being a bit of a breakthrough, we are keeping the protocol and its implementation proprietary.
Permissioned/Private Blockchains don't need to look like Bitcoin, they just need to offer many of the same features plus a few more. What we're really talking about is a database that competitors can share, which is largely inspired by Bitcoin's system. Some things it needs to actually work:
- Vastly increased performance: 5-15 transactions per second is useless for industrial use.
- Smart Contract languages designed for safety over generality: more of an empowered SQL than a C/Go. Tokenization doesn't provide enough of a benefit over traditional approaches and generalized computation is too dangerous.
- Deterministic BFT Consensus: this is to replace mining + remove anonymous participation, it needs to be fast and scale.
- Confidential transaction support: the trickiest of the lot, as we cannot currently hit performance + double-spend protection at once.
In case you're curious what our approach is: BFT Raft-like for consensus (8k-12k trans/sec), soon to be open sourced smart contract language, signal protocol over the chain for confidential stuff.
[1]: http://kadena.io
When I was doing vendor vetting the talk usually revolved around the "blockchain revolution" but glossed over over the consensus issues. In the permissioned blockchain space, mining is really not an option (how do the incentives for mining work when there are no coins involved?). Luckily, we have a lot more flexibility in the consensus system we can pick.
Moreover, the term "blockchain" itself is ill-defined/overloaded. If all blockchains use mining then the 51% issue is ever present (though if I remember correctly problems arise at 30-ish%). If the broader "BFT consensus" definition is used then no, you can use PBFT/SmartBFT and a host of others to come to consensus without ever needing to touch crypto let alone mining. The issue, of course, is that Mining-based consensus is the only system that works for truly public blockchains, as it allows for anonymous participation at all levels and can also scale flat.
If anyone else is a consensus nerd, I wrote a technical-biz-persone level white paper on permissioned blockchains that goes into more depth and covers the issues associated with using TLS vs PPK sigs [1].
[0]: http://github.com/kadena-io/juno
[1]: http://kadena.io/docs/Kadena-ConsensusWhitePaper-Aug2016.pdf
Anyway, to your question (carter focuses on the smart contract language, not the consensus layer): This is actually a semi-normal/expected thing in the case of a network partition. Say one node gets cut off, transitions to a candidate and achieves some high term (but potentially has an out of date log index). When it comes back online/partition ends, it will start spamming the rest of the cluster requesting votes.
What protects the rest from this or other faults is a combo of signed RPC's and extra logic (i.e. lazy votes). With sigs, chatter from someone who you don't have the key for (or invalid msgs generally) just get dropped + authorship is verified (Follower can't pretend to be some other node). With lazy votes the portion of the cluster in a good state basically chills (stays in same state, doesn't vote) until the Followers hit an election timeout (AE clears the lazy vote).
Of course you can still DoS the NIC or Haskell server, but the important thing to note is that this isn't meant for a public-facing deployment. It's designed for an internal network -or- inter-corp via leased lines/secured pipes setup. That's the whole point of Juno really... for enterprise applications bitcoin-class tech gives you more features than you need (at the cost of performance).
Tangaroa was a great first step, but we have internal fixes for a number of cases it didn't cover, issues caused by incomplete implementations and (I think) some novel ideas to give back... but I'm still finishing the paper on the enhanced/new (not sure what the line for that is) protocol and its implementation so it didn't make the cut for this release.
> Tokutek claim to improve MongoDB performance 20x. Unclear if this also mean lose 20x as many documents.
On a biweekly basis I'm shown something new causing my brain to melt, leak out of my ear & pool onto the floor near my desk. It's awesome[0]!
Today's was using impredicative types + `$` has a special case for dealing with it[1].
[0]: ...some number of hours later when I've collected my thoughts and think "OHHH... duh" [1]: https://ghc.haskell.org/trac/ghc/wiki/ImpredicativePolymorph...