Vanadium – Create distributed applications that can run anywhere | Hacker News Reader

Vanadium – Create distributed applications that can run anywhere

v.io

124 pointschuhnk50 comments

The cool thing I really like about Vanadium is a unified worldview it provides with the ability to access every object in the Vanadium ecosystem through its powerful naming/mounting mechanism. In Vanadium, you can "mount" things on your namespace (think Unix VFS) on a hierarchical namespace and invoke RPC methods on the objects. The method invocation is mutually authenticated with ed25519 keys and end-to-end encrypted on the wire. I sometimes jokingly called it "Plan 10". It is certainly not just another "synchronizing database" though that is a subset of it.

Disclosure: I very briefly worked on it at Google and I can attest the team behind it is brilliant beyond imagination.

After 10 minutes, here's what I found out:

- The authors are either Google and/or heavily affiliated with Google

- On their website, the two highlighted projects/products are Syncbase and Vanadium Core.

- Syncbase is a distributed, works-offline datastore, with well-defined ways of doing reconciliation. Think, git merges, or collaborative docs, or y'know anything with vector clocks but approached from the side of the UX they want to target.

- Vanadium Core is "the discovery, RPC, and security layer underlying Syncbase", including its custom binary data serialization format 'VOM' and its IDL called 'VDL'.

> Syncbase: "Offline-first and peer-to-peer, Syncbase provides local storage that opportunistically syncs data".

I have an application for that _right now_. Syncbase doesn't look ready though, does anyone know of an alternative data store that ticks all those boxes? e.g. I was looking at Cognito, but it lacks peer-to-peer.

You should take a look at Noms:

https://github.com/attic-labs/noms

Happy to answer questions here or in our Slack channel.

Nothing else does what vanadium does, especially at its sophistication level. I don't know if I'd use it in production yet unless you are willing to put in some engineering resources, but you're unlikely to find something else as well made elsewhere. This is the product of a very elite team... which means it's also very complicated to use.

This is the product of a very elite team... which means it's also very complicated to use.

That's an unfortunate assumption. Looking back at historic examples, the Windows API was complicated to use while the NeXTSTEP API wasn't. Is there any question which was created by an "elite team"?

Syncbase is the attempt to simplify it and abstract a lot of it. It's API is much more straightforward and is fairly easy to work with.

Part of the complexity is that it's solving like 15 major problems at the same time in an interconnected way. It's also solving them for people like google which means there are a lot of requirements for all the things it needs to do.

Because it does new things, in addition to old things in new ways, there's a steep learning curve for all of its concepts. I totally agree that doesn't mean it couldn't be more elegant and "simple," but the win32/nextstep analogy isn't fair as they were far more similar than vanadium is to anything else. Vanadium is like a hybrid of (to use existing concepts) dns/http/ssl/json/bluetooth/nosql/ntp/kerberos and more. When you put it all together there isn't a lot of existing references or cognitive frames to help you "get it" righ away.

The outcome I'm looking for is basically CRDT (or something very like it) using only BYO smartphone & laptop devices (acting as both the forwarding nodes and the application client), robust against nontechnical user mishap and patchy/absent Internet connectivity.

I was looking at http://www.opengarden.com/meshkit.html but they don't seem interested in my scale.

I'd go with something that's more production ready / supported. Don't know about meshkit but sounds cool.

Hod @ Couchbase here. Depending on what you mean, I have to disagree. Are you talking about Vanadium Core, or Vanadium and Syncbase? From what I'm reading, Couchbase does significantly more than Syncbase. It supports many of the security features of Vanadium, although it doesn't have ACLs with things like proximity built-in.

Couchbase Lite does all this, and is a mature, shipping product with many real-world deployments (including peer-to-peer as a primary use). It's a JSON document database, with full map/reduce indexing/query capabilities. It uses MVCC (similar to git) to handle offline first + conflict resolution.

Couchbase Lite can be used stand-alone, or as part of the Couchbase Mobile stack, which includes Sync Gateway and Couchbase Server. Everything is open source, too.

I'm happy to answer questions here, you can ping me on twitter, or through our forums, etc. Hod @ Couchbase

ours https://github.com/amark/gun .

Pouch https://pouchdb.com/ .

You could even do http://webtorrent.io !

Also something about Syncbase on Youtube.

https://www.youtube.com/watch?v=2cHzd8pBYmU

Hilariously awkward video.

But the key points:

-) All data is encrypted both on the wire and "at rest" (presumably on disk)

-) All options are versioned to allow merging

-) Provides builtin policies for resolving conflicts (eg last one wins)

-) Handles structured data (nosql api) and binary data (blob api)

-) Fine grained ACLs for permissions

-) Apparently offers p2p syncing without a central server

It's written to be security and offline first. Most systems have this bolted on afterwards which is notoriously difficult to do and never quite right.

You're also missing:

- a very sophisticated ACL/role model all built off cryptography that allows me to do things like grant Bob permission to watch my TV if Bob is within 100ft of a vanadium-powered Bluetooth device that can verify his proximity. And that's just the tip of the iceberg.

- solves RPC in a cross-platform way, handles bi-directional steaming, and data marshaling

- solves decentralized DNS using something called mounttables that can work just as well for servers in a cloud environment as routing to your IoT devices

- basically reinvents how devices talk to each other in many other ways that are too much for a HN comment.

The core repository has an AUTHORS file, and the only entry is "Google, Inc."

Disclaimer: I work for Google, but I have no idea what this is.

You keep using that word ("disclaimer"), I do not think it means what you think it means.

You're right :). But I've also seen "disclaimer" commonly used in specifically this context (disclosing your affiliation with a company you're mentioning), which is why I use it that way. "Disclosure" would be more accurate, but that's not what people actually say.

HN users have lots to disclaim.

http://mbrock.github.io/disclaimer.html

How did you find the association with Google?

But you are probably right: page is in material design, the repos are mostly Dart or Go.

I'm so glad you asked!

- I read the Vanadium ToS [1]

- In the video embedded from the page on 'Syncbase' [2] everyone shown works at Google.

- At this point, the association with Google looks very likely. I searched for 'syncbase google' and found a ton of intriguing stuff, including the prospect that it's being used in production settings by Google.

- I searched for 'vanadium google' and found this Twitter thread [3]

[1] https://vanadium.github.io/tos.html

[2] https://vanadium.github.io/syncbase/

[3] https://twitter.com/chuhnk/status/685455321465237504

I wonder if Vanadium is the project Jeff Nichols http://www.jeffreynichols.com is talking about.

  Quote:
  I am a Staff Research Scientist at Google managing a team of roughly seven people at the Mountain   View campus.
  I work on a project that you haven't heard of and that I can't talk about, but, needless to say, we're doing really cool stuff that you'll be excited about when it's finally released.

Yep

Their intro video is done by 2 Google engineers.

And they say 'working on here at Google' in the video

Let me provide a brief history/overview and answer the questions that have been asked as best I can. I started vanadium about three years ago to build a secure rpc and naming system that would make it easier to build distributed applications, including those on mobile and embedded devices/IoT. Security was a central goal. Distributed, synchronzsed storage has long been a pain point, especially when devices are often disconnected and we built syncbase to address that need.

I left Google in May of this year and joined Grail, where we're developing an NGS-based blood test to detect cancer early, when it can still be cured. We're using Vanadium for the infrastructure we need to support our internal development and analysis pipelines. Since I left Google Vanadium was split into a 'core' component (security, rpc, naming) and syncbase separately. The core component is very stable, syncbase is likely stable but not quite complete. We're currently using only Vanadium core.

To answer some point questions: - yes, this is the project Jeff Nichol's is referring to - and here, https://arxiv.org/pdf/1604.06959v2.pdf

I ported vanadium and syncbase to iOS. Happy to answer questions. Yes indeed this is a google project but is now being taken on by its original lead, and former head of search infrastructure, now at Grail bio.

The project looks interesting but something about it rubs me the wrong way. Which organization is behind vanadium? Is it google? Then how is grail bio involved? Or is it a side project of google employees who now work at grail bio? I'm confused.

Also, why are syncbase sources not on github?

It's all there. Just look under Go/Java/Swift https://github.com/vanadium

Thanks, I see it now! The first part of my question is still unanswered though..

I'd defer to cosmosnicolaou there, as I've not been involved in some months. Check out his answer below.

Also might be worth asking on our mailing list https://groups.google.com/a/v.io/forum/#!forum/vanadium-disc...

Everything is open source and on GitHub. This is a rare project that was 100% open source from the beginning.

Quickly looked over this and the thing that caught my eye is that it says you can set up distributed systems that are available offline through device to device communication. At least, I think that is what it does. I could not quickly find more information on that. A diagram or something would be helpful to visualize this, with an explanation of the protocols that are used. Maybe I missed the examples or case study section.

I didnt know io sold single letter domains, sexy.

So does .org, see http://www.project94.org/.

I wonder how much they cost.

I believe the first round auctions (a couple of years ago) started at $50,000 for single-letter domains. As an additional barrier, it may also be that PIR require some kind of public-interest usage commitment.

It seems like it is no longer under active development. I hope I am wrong, but does anyone happen to know?

It looks like a lot of the Googlers previously working on the project has now moved to Grail bio.

I was on the team working on Vanadium @ Google and have since moved on to Firebase.

I can't speak to what level of commitment to the project those that have moved on to Grail have; you should direct questions to https://news.ycombinator.com/user?id=cosmosnicolaou or refer to his comments.

I actually recently gave a presentation / live demo of Vanadium at Drinks on Tap SF. It's not production ready, but a cool glimpse into a possible new paradigm ;)

"a lot"? who else do you see besides Cosmos?

Wonderful to hear about more P2P database sync systems!

I would love to hear how you compare to Deepstream and us ( http://gun.js.org/ ), and http://Horizon.io ?

Edit: Also how on earth did you snag `v.io`? That is amazing and I thought single-letter domains weren't allowed?

I'm assuming Horizon is in limbo along with RethinkDB?

Can anyone comment on the disadvantages of last-write-wins? Either that or how tricky it is to always have to define your own merge policies?

Is this similar to Irmin (https://github.com/mirage/irmin) ?

In the documentation I see

  Note
  Requiring a cloud Syncbase is temporary. We are planning to allow the API to be used without a cloud Syncbase soon.

Does it mean that the following use case will be supported:

I have two devices in the same LAN with no internet access, and they will be able to synchronize ?

Correct.

besides wifi, what other type of p2p communication channels are supported in Vanadium? bluetooth? USB? radio?

Prototypes also use BLE / Bluetooth Classic but the vision being that it's transport agnostic.

this vs. firebase for sync ?

Firebase doesn't have offline p2p at all, let alone sync.

That said, right now I'd build using firebase unless you have very specific needs.

Reply on news.ycombinator.com