Show HN: AvionDB: A Decentralised Database with MongoDB-like Developer Interface | Hacker News Reader

Show HN: AvionDB: A Decentralised Database with MongoDB-like Developer Interface

github.com

104 pointsAlex-Potsides20 comments

One thing I don't like about the MongoDB dashboard is that if I'm already logged in and go to Tools, I still have to use commands to export data. Why can't I just press a button in their UI and export the data however I want? I'm already logged in. This might seem trivial but it would be faster. Especially if you're working with someone semi-technical and who has only a basic understand of looking through documents, you don't want to have to walk them through different commands and using the CLI.

I never tried exporting data using the Dashboard (always used mongo dump).

> One thing I don't like about the MongoDB dashboard is that if I'm already logged in and go to Tools, I still have to use commands to export data.

Man, that sucks. Why would they do that? Anyways, we will definitely have an export button for AvionDB.

Yeah, I would say UI + analytics/filtering for documents built into the dashboard are things that I wish existed. I know there's are lots of third party tools but a lot of times in the early stages of projects I feel those aren't yet a priority.

We would love to build a Dashboard soon. It's actually in our Roadmap. https://github.com/dappkit/aviondb/issues/8

It would be great if you could join our Discord channel: https://discord.gg/88YpNuQ

I would love to have your feedback on our Dashboard+what features that you may want. I can get that up in a few days.

Can you clarify what you mean by the Dashboard? Are you talking about a specific GUI util?

He is referring to Mongo Compass[1]

1: https://www.mongodb.com/products/compass

Some useful links:

How to Install: https://github.com/dappkit/aviondb#install

How to Use: https://github.com/dappkit/aviondb#usage

API Documentation: https://github.com/dappkit/aviondb/blob/master/API.md

Roadmap: https://github.com/dappkit/aviondb/issues/10

FAQs: https://github.com/dappkit/aviondb#frequently-asked-question...

Hey hello everyone. One of the Aviondb developers here. Feel free to ask me any questions.

Neat to see this made it to HN! Congrats.

So, my major concern with OrbitDB is the use of in-memory indexes to hashes. It's a very naive approach for multiple reasons. It just doesn't scale well. Node.js handles these particularly poorly, and develops memory leaks in my own experiments with AOILs (append-only immutable logs) inspired by Secure Scuttlebutt.

What I mean is, take a look at this code: https://github.com/orbitdb/orbit-db-kvstore/blob/master/src/...

And this: https://github.com/orbitdb/orbit-db-docstore/blob/master/src...

I'd suggest perhaps building that in SnapDB:

https://github.com/ClickSimply/snap-db

Though, there are some really hard limits to file locks / mutexes that are needed for a decent persistent K/V store. Since, pretty much any database is going to need an index, and an index is just a K/V store. So, in essence, if your platform can't be relied on to implement a good embedded K/V store, it can't be relied on to create anything other than toy databases.

I considered adding SnapDB to OrbitDB, but it would've required a lot of refactoring to support asynchronous lookup. I eventually realized I could do a lot of what I wanted to accomplish without the elaborate CRDTs like OrbitDB employs to keep an ordered log of information.

I would also add that, although it should be obvious, just in case it isn't, AvionDB now builds on OrbitDB which itself builds on IPFS. So, that's a big consideration for those looking to work with this.

(The below is mostly from personal experience and not terribly relevant to AvionDB and OrbitDB)

I've been meaning to do a more formal post-mortem on the project, but haven't gotten anything fully organized yet. I discovered all this after creating a project a while back for what was essentially a P2P image gallery built with IPFS and Electron. I wrote it in TypeScript, React (with Hooks), Redux Toolkit (which makes Redux a lot less insufferable than it used to be, my god), and SnapDB and OrbitDB (but not together). I also had go-ipfs running in a process managed by the Electron client, and included it with the build.

As a result, the app image binary would generally be in excess of 500MB. And this was just our 0.1.0, we hadn't gotten to the really good features yet. The performance was decent in the UI, but the app itself couldn't scale very well past 1000 images. go-ipfs was the real limiting factor at the time; it would just fail in really bizarre ways, and I'd have to not only write code for restarting the go-ipfs process in case it died, but also retry requests that didn't go through. I suspected it might've been due to me having pubsub enabled, but my ability to debug that was hamstrung by the fact that OrbitDB hard fails without pubsub enabled. I haven't tried it with go-ipfs 0.5.0, I think pubsub is enabled by default with that now, and I've already moved on from that implementation.

My biggest concern in terms of performance aside from go-ipfs (which was disappointing and unexpected), was the fact that due to the design, there would be multiple copies of the same data in-memory everywhere: the in-memory indexes used by OrbitDB, the data in Redux, the data in the DOM, etc. I'm able to effectively integrate all of those using a Rust desktop binary. I'm using Iced for the UI, and I've moved on from bundling IPFS entirely. It allows me to build desktop and WASM binaries from the same codebase.

It's possible I could probably make the Electron app work better at this point, but there are still the fundamental limitations I pointed out earlier.

For me, content-addressable hashes and IPNS signatures are perfectly fine. I value performance (CPU, RAM, and disk) over anything else, primarily because I'm developing P2P apps, and without a server, the nodes have to do all the work itself, including the UI. That's why I moved off of TypeScript and into Rust. PubSub would be nice, and OrbitDB has a decent implementation of Lamport Clocks and CRDTs for decentralized ordering of events, but it's just often overkill for most things that could just be content-addressable.

---

Just to be clear, I've made a lot of claims here, and I'm hardly an expert on any one of these things, so if I've gotten any of this wrong, I'd really like to know more. Also, the code I wrote earlier isn't available to the public. If I have any say in the matter, it will be locked away within an enormous warehouse, Indiana Jones style, hopefully never to see the light of day again.

Hey cryptoquick, I remember you from a while back. I'm https://twitter.com/aphelionz, one of the maintainers. Nice to see you :)

You're not wrong about the memory usage. We decided early on that the tradeoff would be more RAM than to take the performance hit with the I/O (either to and from IPFS or to and from the filesystem in the case of SnapDB).

That being said, I have been experimenting with removing indexes like the ones you pointed out in favor of generators that emit the values as they are traversed. There are two main efforts here, one is feasible and the other is more difficult.

1. The `entryIndex` inside of ipfs-log can probably go, and the ipfs-log#traverse function can be made into an async generator function that passes the oplog values up to the store 2. The indexes you linked to that hold the calculated STATE are harder to get rid of - they could be persisted to IPFS or the file system as well

Open to ideas on #2. SnapDB might work for all I know, as I haven't attempted it yet.

I can't speak to Electron, React, Redux, Iced, etc, but my guess is there are optimizations one can do there as well.

Also, speaking of Rust - we'd love more contributors over at https://github.com/ipfs-rust/rust-ipfs/.

One more thing: The code in AvionDB is really more representative of how OrbitDB could be used to create a decentralized document store, but OrbitDB offers a lot more, and IPFS offers a lot more, too.

I think this project could be considered less of a library dependency, and more of an example OrbitDB project.

I totally agree with this. A lot of our work on AvionDB involves improving OrbitDB as AvionDB is basically a project built on top of OrbitDB. Exactly speaking we have 2 new store implementations:

Store.js [1]: This store manages all the "Databases" & the "Collections" within the respective "Databases".

Collection.js [2]: This is the store that implements the MongoDB-like interface so that you can run powerful queries, and manage "Documents" within the respective "Collections".

[1]: https://github.com/dappkit/aviondb/blob/master/src/Store.js

[2]: https://github.com/dappkit/aviondb/blob/master/src/Collectio...

Just in case anyone is interested in discussing this more, you can hop on OrbitDB Gitter[1] where we are continuing this discussion.

[1]: https://gitter.im/orbitdb/Lobby

I don't get it. How is this decentralized?

Hi @uberfru

AvionDB is built on top of OrbitDB[1] which implements the core decentralized database logic/protocol, including CRDTs, Lamport clocks, immutable logs, etc.

If you want to see for your self and dive a bit into how it works under the hood, you can check OrbitDB's welcome page[2] or field manual[3]

1: https://github.com/orbitdb/orbit-db

2: https://github.com/orbitdb/welcome

3: https://github.com/orbitdb/field-manual

Another database? I feel like there are too many database solutions nowadays.

It's not another database actually. It's built on top of pre-existing OrbitDB[1]. I found that there that the developer interface was not as powerful as I wanted, so a few folks from the OrbitDB community started this project.

[1]: https://github.com/orbitdb/orbit-db

The problem is with quality not quantity.

And the quality is severely lacking in most databases.

Just out of curiosity, what are the parameters that decide the quality of a Distributed Database Project?

Reply on news.ycombinator.com