Show HN: voidDB – A transactional key-value DB written in Go for 64-bit Linux | Hacker News Reader

Show HN: voidDB – A transactional key-value DB written in Go for 64-bit Linux

github.com

67 pointsjoel-ling31 comments

  // voidDB is a cherished toy, a journey into the Unknown, a heroic struggle,
  // and a work of love. It is the “Twee!” of a bird; a tree falling in the
  // forest; yet another programmer pouring their drop into the proverbial [bit]
  // bucket. Above all, it is a shrine unto simple, readable, and functional
  // code; an assertion that the dichotomy between such aesthetics and practical
  // performance is mere illusion.

[1] https://github.com/voidDB/voidDB/blob/ac104867749e2c698b3d61...

I admire the author who took up the challenge of writing a KV DB and cherishing self's efforts. I hope someday I could do something similar and feel accomplished.

Five years ago, I was new to KV DBs and Golang. You can!

Love the poetry in the comment prose! Endeavour so much needed in the industry. It can only come out of the joy of coding, how else could it be born...

It always made me happy to hack nearby the Neo4j page cache "Muninn", because it meant I'd pass by Chris Vest's poem and giant ascii Raven: https://github.com/neo4j/neo4j/blob/release/5.26.0/community...

Beautiful reminder of the often forgotten aspect of technical work.

Huginn: thought, perception, comprehension, ...

Muninn: care, urge, wonder, curiosity, interest, ...

https://en.wikipedia.org/wiki/Huginn_and_Muninn#Etymology

Thank you for highlighting. These contain references to some philosophies I relate to and find comforting, including those most prominently articulated by Ernest Becker and Alan Watts.

I can’t help but think this would be more valuable as a C (or even Rust or Zig!) library with a Go API. Then it could be used elsewhere too.

My thoughts as well!

Seemingly faster and more compact than lmdb-go, bbolt, Badger, and goleveldb at <2K LOC.

  goos: linux
  goarch: arm64
  pkg: github.com/voidDB/voidDB/test
  BenchmarkVoidPut-2            131072      14933 ns/op
  BenchmarkVoidGet-2            131072       1060 ns/op
  BenchmarkVoidGetNext-2        131072        245.8 ns/op
  BenchmarkLMDBPut-2            131072      22414 ns/op
  BenchmarkLMDBGet-2            131072       1826 ns/op
  BenchmarkLMDBGetNext-2        131072        602.2 ns/op
  BenchmarkBoltPut-2            131072      66984 ns/op
  BenchmarkBoltGet-2            131072       2552 ns/op
  BenchmarkBoltGetNext-2        131072        254.6 ns/op
  BenchmarkLevelPut-2           131072      44182 ns/op
  BenchmarkLevelGet-2           131072      30949 ns/op
  BenchmarkLevelGetNext-2       131072       3441 ns/op
  BenchmarkBadgerPut-2          131072      15182 ns/op
  BenchmarkBadgerGet-2          131072      33114 ns/op
  BenchmarkBadgerGetNext-2      131072      12895 ns/op
  BenchmarkNothing-2            131072          0.3239 ns/op

Check out the docs at https://pkg.go.dev/github.com/voidDB/voidDB

The benchmark doesn't accurately represent the real-world database performance because the dataset is too small (roughly half a gigabyte based on [1]?), meaning it fits into the page cache bypassing disk I/O.

[1]: https://github.com/voidDB/voidDB/blob/master/test/bench_test...

This is a cool effort but their claims are wildly misleading as their benchmarks aren't fair across the board:

For example:

- BenchmarkVoidPut runs a single O(n) sized transaction

- BenchmarkLMDBPut runs a single O(n) sized transaction

- BenchmarkBadgetPut runs O(n) O(n) sized transactions (!!!)

I made a local change so that all Put benchmarks ran O(n) O(1) sized transactions and the results were quite different: Void was the slowest, followed by LMDB, Bold, LevelDB, then Badger.

I'd also wager the LMDB author would also (lovingly!) tell us we're holding it wrong

LMDB is the only one here in C, so the interop is probably what makes LMDB so slow in here

Curious, what overhead is there calling C code from other languages?

Go doesn't use the C calling convention, but has its own growable stack system and goroutine scheduler that maps to goroutines to threads. So a goroutine can't just call a C function directly.

In order to interface with C code safely, Go's runtime has to jump to the system stack and do some additional setup, make the call, and then switch back. (Adding to that, if the call takes too long, this prevents other goroutines on the same OS thread from running, so the scheduler must jump in and move those goroutines to a different thread.)

All of this is expensive, though we are talking about nanoseconds, not milliseconds. Performance is mostly a problem when doing lots of very quick calls (e.g. you're writing a game engine interacting with something like OpenGL) or lots of slow calls (causing scheduler trashing).

Thanks! Is this also the case even Rust consumes a C library?

No, my understanding is that Rust uses normal stacks, and it uses a classic threading model, so aside from async, calling C doesn't need to any runtime stuff.

That is correct, and was a major motivation for dropping green threads way back in 2014.

Since there are no strict OS dependency I can see, has anyone tried to compile this and run it on Windows?

seems pretty bloated

https://github.com/voidDB/voidDB/blob/master/go.sum

A lot of it is indirect dependencies, and most of the direct dependencies in the go.mod are just for benchmarks, as far as I can tell.

That is correct. Remove directories "test/" and "cursor/test/", then run "go mod tidy"; you should see only one dependency in go.mod:

  require golang.org/x/sys v0.29.0

Could have used "syscall" in the standard library instead of "golang.org/x/sys/unix" if not for these: https://pkg.go.dev/golang.org/x/sys/unix#F_OFD_GETLK

... and this: https://go.googlesource.com/proposal/+/refs/heads/master/des...

Only "x/sys/unix" outside of tests/benchmarks

      ~/go/src/github.com/voidDB/voidDB  git:(master)
      $ go run github.com/tailscale/depaware@latest
      github.com/voidDB/voidDB dependencies: (generated by github.com/tailscale/depaware)

            github.com/voidDB/voidDB/common                              from github.com/voidDB/voidDB+
            github.com/voidDB/voidDB/cursor                              from github.com/voidDB/voidDB
            github.com/voidDB/voidDB/free                                from github.com/voidDB/voidDB
            github.com/voidDB/voidDB/node                                from github.com/voidDB/voidDB+
            github.com/voidDB/voidDB/reader                              from github.com/voidDB/voidDB
            golang.org/x/sys/unix                                        from github.com/voidDB/voidDB+
            bytes                                                        from github.com/voidDB/voidDB+
            cmp                                                          from internal/fmtsort+
            encoding/binary                                              from github.com/voidDB/voidDB/common+
            errors                                                       from bytes+
      D    fmt                                                          from golang.org/x/sys/unix
            hash                                                         from github.com/voidDB/voidDB+
            hash/fnv                                                     from github.com/voidDB/voidDB
            io                                                           from bytes+
            io/fs                                                        from internal/filepathlite+
            iter                                                         from reflect+
            math                                                         from encoding/binary+
            math/bits                                                    from golang.org/x/sys/unix+
            os                                                           from fmt+
            path                                                         from io/fs
            reflect                                                      from encoding/binary+
            slices                                                       from encoding/binary+
      LD    sort                                                         from golang.org/x/sys/unix
            strconv                                                      from fmt+
      LD    strings                                                      from golang.org/x/sys/unix
            sync                                                         from encoding/binary+
            sync/atomic                                                  from internal/bisect+
            syscall                                                      from github.com/voidDB/voidDB/cursor+
            time                                                         from github.com/voidDB/voidDB+
            unicode                                                      from bytes+
      W    unicode/utf16                                                from internal/poll+
            unicode/utf8                                                 from bytes+

Thank you for showing substantive evidence in defence of voidDB; it is very vindicating.

It doesn't take much to split your go.mod file require blocks into needed dependencies, and test dependencies, with a few comments explaining the why of each dependency.

Go tooling will preserve your go.mod comments and structure.

You can also consider moving integration tests and benchmarks to another module in the same repo.

Great ideas. I have broken out the tests into their own modules within the same repository. Thanks!

Look ma, no more dependencies! https://github.com/voidDB/voidDB/blob/v0.1.4/go.sum

Very cool! keep it up :)

Faster than rocksdb?

probably not if it's LMDB-based (at least for writes)

it's screaming for a REPL...

Reply on news.ycombinator.com