Immudb 1.0 – open-source, immutable database with SQL and verified timetravel
codenotary.com
codenotary.com
> Companies use immudb to protect credit card transactions and to secure processes by storing digital certificates and checksums.
This explanation is available on their github repo [1]. It has been a common refrain on Hacker News that you don't need a blockchain and instead can just use a database, but this product may actually fill the gap where tamper resistance is desired.
I think in the future, all enterprise storage solutions will be append-only by default. To protect against cryptolocker malware. But also with isolated functionality for actually deleting data, for example because of GDPR requests or because of malware that tries to fill all writable storage with garbage. So that data can still be deleted, but not from any of the regular servers that are reading and appending data to the system. Instead from separate servers that are isolated and for data storage management only.
Compared to blockchain which is running on X number of nodes that you'd have to have access to in order to modify something, immudb doesn't actually seem to replace the use case when you need something actually tamper-proof.
I would say that in practice what differentiates a blockchain from other applications of hash trees is a mechanism for consensus, not whether the blocks being formed into a trees conceptually represent time or not.
In a block chain it is easy to find history because the link to the history is included. Merkle trees require n-1 additional hashes.
A blockchain is a growing list of records,
called blocks, that are linked together using
cryptography. Each block contains a cryptographic
hash of the previous block, a timestamp, and
transaction data (generally represented as a
Merkle tree).
Merkle trees are used in various ways within cryptography in general and cryptocurrencies specifically, but blockchains and Merkle trees are distinct data structures with different uses. The colloquial use of "blockchain" has perhaps made the word somewhat ambiguous in some contexts but not in the context of cryptographic data structures, and Merkle trees are in fact formally defined.
Cool, so let's say we're using this to track financial transactions.
Server A has an Immudb at some state x and server B has an Immudb at some state y. Which one is correct, how do I decide?
Keeping track of the "HEAD" hash on the clients is what consensus protocols achieve. You can also achieve it with trusted counters like the one SGX provides (depends on Intel ME so not exactly recommended, also most probably switched off in cloud environments). Alternative is an implementation of something like https://dl.acm.org/doi/10.5555/3241189.3241289.
You can of course say that it's the clients' responsibility to do this, but in practice they won't and they'll implicitly trust the server state.
Having said this, the project does look promising, we may end up using it in a confidential compute setting where clients can verify the server code running, and we'll add rollback protection on top
This is an impossible guarantee. Suppose the state that is sent to you from the server needs some time to get to you. meanwhile the state on the server could have changed. You don't even need a remote server to have this issue. Your thread (where you see the latest state) is put to sleep for a while (sheduler, os, ...) It wakes up. Is the state it observes still the latest? That's impossible to know. The only thing you can do is to refuse future updates if the state they were built upon is not the current state of the database.
With crypto protocols in general "guarantees" are always prefixed with "if the protocol completed successfully, then ...". For example authenticated DH + e2e encryption guarantees that you will send data to the intended participant only. But an attacker can still disrupt the network packets, so the true guarantee is "if the protocol completed successfully, then you have sent the data to the intended participant only".
Same thing here, you cannot of course guarantee the "latest state", if we want to go into the extreme, one could even argue that actually time doesn't work like that because of relativity/speed of light limitations:D. What you can guarantee is that if your commit protocol succeeded, then it updated the latest state at the time of consensus/monotonic counter update.
> That's what I meant.
> time doesn't work like that because of relativity/speed of light.
Now add a mirror: git upstream, FS snapshot, immudb replica, etc…or even just an outside log of the merkle proofs themselves. Then, if your database ever fails a check against that proof, you know the data has been modified, not just appended to.
To use a familiar Git workflow example: you can do whatever local writes you want, but if you disallow force pushes no one can erase history on the upstream repo.
Put another way: if you have immutable backups you don’t need a blockchain to ensure data integrity. OTOH, if you can’t trust your own infrastructure even as far as a secure remote backup you have other problems that a blockchain won’t solve either.
> The majority decision is represented by the longest chain, which has the greatest proof-of-work effort invested in it. If a majority of CPU power is controlled by honest nodes, the honest chain will grow the fastest and outpace any competing chains.
If the majority of the hashpower (not nodes!) is dishonest, you can rewrite history. It's the reason why the current difficulty is part of a block.
This has been done numerous times in the past for small but highly traded altcoins.
1. You can't rewrite everything. Given enough hash power to create a longer chain, you can create a block that a) removes any transactions from a block in the original chain and b) contains new valid transactions (must be signed by you so you must be the owner of the Bitcoins used in the tx), allowing you to double spend your coins, but you can't change other people's transactions.
2. With each new block changing a past one becomes harder, while you make it sound as if you could arbitrarily rewrite history. Merchants usually wait several blocks before accepting your on-chain payment. Exchanges wait 6 blocks as that's seen as infeasible to change a block that's buried under 5 other blocks for non-nation state actors.
TL;DR: 1) Other people's transactions can at most be removed but not be changed and 2) data on the Bitcoin blockchain is tamper proof after x blocks.
Sure, someone, somewhere will know the truth but who cares - it's all about information control, and you control 51% of the network. Now your opponents have to scramble with social countermeasures to try and discredit your chain. A politically savvy attacker will simply buy off the main social nodes favorably.
EDIT: The problem with "oh but people will know" argument is that it assumes credibility outside the network, the thing which cryptocurrencies like to claim is unnecessary. In reality, if the network is 51% attacked, your entire recovery strategy is assuming enough people believe you about what's real or should be real. This isn't even theoretical: this is literally what happened to Ethereum.
Are you talking about some kind of node sybil attack? Because the only way I can make sense of what you've said is if a system had client implementations maintaining zero state - relying completely on unsecured network consensus. But in that scenario hashing power wouldn't enter the equation. I don't know of any cryptocurrency that operates that way, and doubt that one ever has outside of a home lab.
Or are you talking about someone trying to hijack a cryptocurrency by creating brand confusion and convincing people to run modified code, like the failed bcash campaign against bitcoin?
A 51% attack on a blockchain has no rewrite ability past the point that where an attacker establishes a longer chain. That is why it is a "chain"... where do you think the hashing function inputs come from that are being fed into the adversarial mining hardware? An earlier block - the one where the attack started, you can't go back further than that without having to throw out and rehash your entire attack fork.
If I had >50% hashing power of BTC(for exactly 1 transaction), I can make the current chain say anything I want, like give me all the BTC, and it would become valid and "permanent". To re-write actual history takes a lot more work, and wouldn't be possible with 51% for 1 transaction.. If I was able to maintain 51% control for a long time, then I can do anything I want for as long as I have 51% control. Though I imagine after that very 1st transaction, the world of BTC would blow up and everyone would stop hashing BTC as there would be zero point as the current chain is now effectively useless.
This is the real issue that I see, 1 transaction of 51% power is enough to permanently wipe out all of BTC's worth. As far as I'm aware every crypto currency basically has this same problem.
The only way what you propose would work would be if they could cobble together enough resources to break my 51% control.
So BTC would go 100% bust if someone managed a 51% attack. The question is, can someone with 1 transaction of 51% get enough converted to USD/etc before enough people noticed. Otherwise the financial incentive isn't there to try. I'd guess no matter what, it would be a huge fricking mess and if you did it in a country that didn't like you, it probably wouldn't end well for you years later when whatever govt you live in gets around to ruining your life, even if you managed to extract a few billion.
Because you know the exchanges like Coinbase as soon as they noticed would do their best to stop you(as it's in their best interest).
> If I had >50% hashing power of BTC(for exactly 1 transaction), I can make the current chain say anything I want, like give me all the BTC, and it would become valid and "permanent".
You clearly don't understand it, and it is kind of amazing that you think such a design would survive for as long as bitcoin has (going on 12 years) - by relying on altruistic miners not taking advantage of such a silly flaw. The Satoshi white paper is 9 pages long and written in very plain language... do like everyone use to do back in the day: read the paper and take a peek at the source, it goes a long way in inoculating you from misinformation that you subsequently repeat, accidentally (?) misinforming others.
To help, here is my understanding.
In a < 51%(i.e. normal transaction), if I win the BTC mining lottery and get to write the next transaction, I can make it say whatever I want, but then 51% of the mining network has to agree that what I said was sane(as defined by the current mining software).
If I control 51%, then I can make the transaction say whatever I want AND I can make everyone else accept it as sane, because a majority of the network agrees with me.
This is how BTC changes over time, > 51% of the network agree that they will accept X as the new reality, and it then becomes the new reality.
> ..make the current chain say anything I want, like give me all the BTC..
You know that transactions are cryptographically secured by public and private keys, right? That would be like saying "I can h4x0r all the hotmails and rewrite every PGP armored message to say whatever I want!" Do you think that miners, upon building the next block, have the opportunity to ignore all the rules with regard to PKI?
> ..and it would become valid..
There is a very straightforward block validation sequence, your attack would impotently collapse against it for any number of reasons: https://en.bitcoin.it/wiki/Protocol_rules
> I have read the white paper.
No you didn't. Sorry to have to put it so bluntly, but you couldn't have read it and still be so comically wrong - like I said, it is written in very plain language.
I.e. These rules CHANGE, and since they change(and have changed in the past) if you can convince a majority, you can change the rules to be whatever you want.
Also see: https://en.bitcoin.it/wiki/Economic_majority
Which is exactly what I said. If I have 51% of mining I can make BTC do whatever I want, but that doesn't mean the majority(or any) of the exchanges will accept it, which is basically what the above is saying.
Also see: https://en.bitcoin.it/wiki/Bitcoin_is_not_ruled_by_miners
Again, what they are saying is what I've said, they just put flowery language around it saying, see, miners can't do EVEYRTHING, which isn't technically true, but practically true. Miners can technically do whatever the hell they want, assuming they have the majority, but that doesn't mean exchanges like Coinbase will accept it and exchange the BTC for USD.
so what I said above is generally and technically true, see my other comment in this thread as well, where I said it would be a giant mess and likely ruin BTC forever if someone ever did execute a 51% attack. So there is little incentive(financial or otherwise) to do so.
The closest real-life example we have(that I'm aware of) is the BTC cash stuff, where it hard-forked and became it's own crypto currency because they couldn't get a majority to agree, but enough agreed to fork themselves.
Sorry, can't hear you above the noise of your furious backpedaling. If somebody rewrites the protocol rules in their software to allow anything approaching what you've described (lol, "yes cryptography is involved"), then why are they worried about btc hashing power? I mean, you specifically said:
> If I had >50% hashing power of BTC(for exactly 1 transaction), I can make the current chain say anything I want, like give me all the BTC, and it would become valid and "permanent".
I'll tell you what it would look like to the rest of the network if anybody enacted your diabolical plot: suddenly somebody starts submitting invalid blocks to the network at regular intervals and only those who are monitoring for weird traffic like that even notice, then the block solve rate slightly sags until the difficulty automatically adjusts. Congratulations, you've turned your massive hashing advantage into an incredibly expensive joke.
False, everyone running a node is enforcing whatever rules their node is written to enforce.
You could have 99% of the hashpower, if you generated a block saying that Coinbase is handing over all their Bitcoin to you (with an invalid signature because you don't have their private key), their node and the nodes of most/all exchanges and businesses would just go "lol, wtf is this shit, not a valid block because it has a transaction with an invalid signature, ignored".
A majority of hashpower is not the same as a majority of economic participants.
No, this is a big misunderstanding of how it works. Having more than 50% doesn't give you superpowers, it just means that you're able to create VALID blocks faster than the rest of the network combined (creating invalid blocks is useless), which allows you to censor transactions (by not including them in your blocks, which will be the chain with more work because you have 50%+) and you can TRY to do double-spends.
A double-spend gets exponentially harder and more expensive to pull off the longer your target waits before considering the original transaction final, because you'll have to rebuild all the blocks generated since that transaction was included in the chain, so that your parallel chain that doesn't include that transaction becomes the chain with the most work that everyone follows.
Not exactly, for every block you rewind there is extra work you have to do, to the point where it may take you close to 2 years to rewind 1 year's worth of blocks (because while you're doing that the rest of the network is still creating new blocks).
It gets exponentially harder/more expensive the further back you want to go.
Presumably you can create a config where have your "main" beefy server where all the activity is -- which is backed up, redundant, etc... And a bunch of "client" servers, which just pull and verify the data all the time. And the client servers notify if there are any errors using some out-of-band channel, probably using the same system you use for general server health monitoring.
So you are getting same security guarantees as "private blockchain", but with drastically higher performance, and only needing one beefy server. And the downside is that you won't auto-stop all operations on tampering, you'll only get an alert for it.
I think that's an unrealistic requirement. There's tamper-evident and tamper-resistant but AFAIK nothing is tamper proof. Best you can do is an HSM with a tamper resistant HMAC with keys and a running checksum in unrecoverable ROM coupled to the packaging.
I beg the differ.
If I place a signed message in the Bitcoin chain, can you then modify that message?
If you can prove that you can somehow modify the message, I'll give you $1,000,000 USD tomorrow.
It makes no sense to say that the blockchain is tamper proof because the blockchain is just information. Tamper "proofness"/resistance is first a property of the devices storing the information - once you get into custody chains, provenance documents, etc. that's when a system becomes tamper resistant. At best the blockchain as a system is "tamper evident" in the colloquial sense because the network of all the other nodes decides which bits of information form the "real" blockchain. However, without verifying the (physical) identity and data integrity of the devices that run (at least?) 50%+1 of the nodes, you have no idea whether the system has been tampered with.
But that's just a question of scale - if you have a rando-blockchain you use for immutability internally then how trivial would it be for me to spin up five servers to outhash you and rewrite history?
Finally but super important, immudb can be deployed anywhere and it's open source!
SELECT * FROM history('foo', `2021-05-18T20:00:00`, `2021-05-18T20:00:00`) as t WHERE t.metadata.id = 'bar';
In the examples shown in the AWS docs, the results of a historical query are not changes made to the document, but the fully resolved state of a document at the requested timestamp (or within the timestamp range). Like other threads on this page mention, this is an unusual but not uncommon DB feature these days.
What I really want is a way to get a hash of a root node / snapshot.
I'm not saying that this isn't a good project but it's a bit strange to frame it as if it was a major technical breakthrough.
If anything what catches my eye in this announcement is the "time travel" feature as well as the wire protocol compatibility with Postgres, that's pretty cool.
I agree that’s sufficient.
One non technical addition I see in Bitcoin is the incentive to verify the checksums.
I don't think it is
e.g. Datomic already had this for a long time, no?
Though this does not do what immudb claims:
> immudb is the first database to provide tamper-evident data management, immutable history and client-cryptographic proof.
And:
> Clients do not need to trust the server and every new client adds trust to the deployment
Maybe they are claiming to be the first with it built-in as a core part of the engine that it is specifically optimised for, but even that might not be true.
It's not. For example, see SAP HANA's "Timeline Index" https://websci.informatik.uni-freiburg.de/publications/sigmo...
For example, here is a link to a live query on our Data Delivery Network (DDN) that runs a JOIN on two daily snapshots (20200809 and 20200810). [1] In this case, these images are the result of a daily script that builds and pushes a new image each day. The storage costs are minimal, as each new image only needs to store the changed rows, rather than a duplicative snapshot.
Each immutable image is comprised of a set of small content-addressable cstore fragments uploaded to object storage, which we only load into the database when they become necessary to satisfy a query. When a query arrives at the DDN, we intercept it at the network level by scripting PgBouncer with embedded Python to orchestrate the infrastructure required to answer the query. The embedded code parses the AST of the query for table references, which it uses to "mount" a temporary schema for serving the query. The temporary schema includes an FDW that implements a "layered querying" protocol (think AUFS) to lazily download only the fragments required to satisfy the query.
(Also, we support live data. But that's for another time!)
[0] https://www.splitgraph.com/docs/concepts/images
[1] https://www.splitgraph.com/workspace/ddn?layout=hsplit&query...
> Each cell in a Bigtable can contain multiple versions of the same data; these versions are indexed by timestamp. Bigtable timestamps are 64-bit integers. They can be assigned by Bigtable, in which case they represent realtime in microseconds...
What sort of use cases are most common? GitHub repository says:
> Companies use immudb to protect credit card transactions and to secure processes by storing digital certificates and checksums.
But I am not sure how people are building that into their architecture to be honest.
The basic idea was that every operation (create, update, delete) are actually normal SQL inserts and all reads are against views defined such that the most recent tuples are returned unless they are flagged as "deleted."
I have typically used these types of designs in mostly simple applications with tables where the row counts are in the low millions of tuples. Dealing with this design in the billions of tuples (probably sharded somehow) might have motivated us out of normal RDBMs and into one of the specialized immutable DBs mentioned.
TL;DR My thinking it allows you to capture more the intent of that event. Although I'm not sure you need an immutable database to do this from scratch - I've seen this in schema designs in the past?
Or are there downsides that would relegate it to specific use cases? A what would those use cases be?
The "right to be forgotten" is not compatible with immutable data. You can't simply need to mark data as deleted, you need to 'purge' it from your system (and possible backups, depending on how long you keep historic backups) - that isn't possible in a system with immutable data.
- store the data encrypted using a secondary protocol, lose the key
- rewrite the whole db
If either of these is not feasible then you should have thought longer about what tech is suitable for which application. Operating your company in a legal manner is a pretty strong factor when making such choices.
You are allowed to keep aggregations and hashes of data. These shouldn't allow to identify a subject. E.g. you can keep list of banned emails as MD5s to verify on sign up etc.
Assuming the clients and server are operated by different entities (otherwise the immutability and verifiability are not that interesting), if someone comes to the server operator with a court order and ask that data be removed, it seems like there is nothing they can do.
You have to not give away the key in the first place, at least not to any clients that you don’t own.
E.g. following the rule “any problem can be solved with a level of indirection”, external clients get some Auth key A, which they feed to internal client, who internally maps it to some data key B, and decrypts the data and hands it back to the external client.
When the data is removed, you delete the mapping from your internal client.
Thing is that you have to do this upfront. I think it's very possible to get into a situation where the data you have to delete is in plaintext. Dropping the whole DB and recreate it from scratch is a bit hefty.
serverless := you don't have to operate the infrastructure yourself
The above approach also allows me to choose any database because I can model this data however I want.
Then, if anything changes you know it's been mutated because the computed signature has changed.
It's a little harder than you might think to make a database with tamper resistance.
In my mind the only way to guarantee is to maintain a copy yourself and check against the "original", but if you're going to do that, then what I described is sufficient, no?
I only mention this because the project mentions that the history is protected by clients, which I imagine is similar to what I'm describing, e.g. copying and checking against the original.
The attacker in that case could update your copy. But you have somewhat started to fix the issue.
To cover the case where a bad admin has access to the DB and any copies, you need to send a hash every so often to an outside source. In this case they use clients (I'm not sure exactly how they do this).
In fact you need a list of hashes one for every 100 rows for example. Re-generated the hashes and checking against an external source should detect a tamper.
In the case of Bitcoin (which is extremely tamper resistant) every node operator is a validator. The hashes are stored in a merkle tree.
"You can [...] never change or delete records."
https://docs.microsoft.com/en-au/azure/azure-sql/database/le...
At a first glance, I don't see anything like that in there.
If you have questions about immudb, you are welcome to join us!
https://www.codenotary.com/blog/immudb-release-1-0-release-p...
This is useful for example in banking applications that keep an audit trail for example.
A sysadmin would not be able to update or delete items in the audit table and so can't cover up a crime.
If the database is tampered with at the file level, they have a way to detect that. (Probably some kind of merkle tree.)
Sounds like they are recording all changes (like SQL2011's system versioned tables, as implemented more-or-less by several common DB engines) but with some sort of hash-chain ledger so that history can be verified and therefore any tampering detected.
> If you can add to the table, that means mutation, right?
It isn't keeping the current view of the data immutable, but is keeping an immutable history of the data. It is immutable in the sense that nothing written to it is ever lost, and you can use the "time-travel" query functions (like SELECT stuff FROM atable FOR SYSTEM_TIME AS OF '2021-03-05') to retrieve it even if it looks to have been completely mangled or deleted if you use a non-time-travelling query.
So we can actually treat our databases like immutable infrastructure and actually roll back changes now without the hulking cludge that is snapshots/restores and database migrations? That's game-changing.
Therefore, you must carefully check that no personal data is stored in immutable databases.
Hard no.
https://www.codenotary.com/blog/immudb-release-1-0-release-p...
Snark aside, it‘s still not 100% clear what‘s the upside of using a completely different database, just for that use case.