Bolt: A low-level key/value database for Go | Hacker News Reader

Bolt: A low-level key/value database for Go

github.com

155 pointssciurus46 comments

Bolt was built at Shopify as a back-end to our Merchant Analytics reporting stack.

If you're interested in working on projects like this, we are hiring - contact me tom at shopify.com.

Cool stuff but why didn't you just use an existing LMDB wrapper or write a new one? What's this offer that an LMDB wrapper doesn't?

We originally used szferi/gomdb which worked well but it didn't give us performance visibility with "go pprof" and it was difficult to debug at times. Bolt also takes the approach of simplifying the LMDB API significantly. There are a lot of advanced, unsafe features that aren't available in Bolt (e.g. WRITEMAP, NOMETASYNC). There's more detail in the README:

https://github.com/boltdb/bolt#lmdb

Bolt was originally just a side project but we found that it fit our needs for our analytics database so we eventually swapped out LMDB for Bolt.

This is a fairly minor nit, but a persistent one nonetheless: you keep claiming to have removed unsafe features from LMDB, yet you left the most dangerous one in there: NOSYNC provides no consistency guarantees whatsoever, whereas e.g. with NOMETASYNC you merely lose durability. In particular NOMETASYNC retained crash safety, which you don't get at all with NOSYNC.

The way this is worded, it sounds like an improvement, but in reality bolt is no safer to use than LMDB was, it just lacks some configurable tradeoffs that were present in the original engine, and were safer to use than those that remain in bolt.

That said, I'm happy to see APPEND go ;)

That's a fair point. Although I'd argue that WRITEMAP and APPEND are more scary. :)

I try to point out in the docs that DB.NoSync should only be used for offline bulk loading where corruption is typically less of an issue (since you can simply rebuild).

https://godoc.org/github.com/boltdb/bolt#DB

The wording was intended to make it sound like an improvement for people wanting smaller, more concise code bases. Bolt is 2KLOC vs LMDB's 8KLOC. Smaller surface area makes it easier to test thoroughly. It's definitely a tradeoff for people who want that really low level control though.

LMDB author here - what have you got against APPEND? Fast bulk-loading is pretty important when you're initializing a system.

Note that APPEND mode won't allow you to corrupt the DB, despite what the doc says - you'll get a KEYEXIST error if you try to load out of order. We just left the doc unchanged to emphasize the importance of using sorted input.

Thanks, Howard. That's good to know. The docs always scared me away from using it at all.

Perhaps it's time for us to update the doc to reflect reality...

Hey Ben, what (refs/books/lectures) would you recommend for someone who is interested to learn theory behind these kind of DBs and implement it from scratch?

I originally started Bolt to learn about the implementation details of low-level data stores. It started as a pet project but became more serious as I filled it out and added additional layers of testing.

I'd honestly suggest reading through the source code of some database projects. That's the best way to learn the structures and understand the engineering tradeoffs. Some are more approachable than others though. Bolt and LMDB are 2KLOC and 8KLOC, respectively. LevelDB is 20KLOC and RocksDB is 100KLOC, IIRC.

If you want to go further then I'd suggest porting one of those over to a language you know well. It doesn't have to be code you release to the world or use in production but just porting the code really helps to cement the concepts in my head.

To get the theory, you could just start on Wikipedia. http://en.wikipedia.org/wiki/B+_tree

To understand why the LMDB implementation does what it does will require much more practical knowledge, not theory. The references in my MDB papers (and the paper itself of course) will give you more of that. http://symas.com/mdb/#pubs

B+trees are trivially simple, in theory. Writing one that is actually efficient, safe, and supports high concurrency requires more than theory. It requires understanding of computer system architecture: CPU architecture/cache hierarchies, filesystem and I/O architecture, memory, etc.

It does not require 100KLOC though. RocksDB is just a pile of gratuitous complexity.

Thanks for the good points! Bolt and LMDB seem approachable, and I'm especially interested how ACID compliance is implemented and how to check/prove the correctness of it...

Can you please compare the locking granularity differences between Bolt and its competitors? File-level locking is pretty coarse-grained and less concurrent than other approaches. Compare with mdbm (recently discussed here) which uses page-level locking.

I've looked briefly at the Bolt documentation a few times and I can't say that I like the idea of giving a function as a parameter to the Update and View functions. It result in some weird looking code with functions within functions, that can't really be refactored out or reused.

Pretty much Javascript callback spaghetti in Go. There might be a way around it, but I haven't found any examples. Perhaps there could be a way where transactions a manually managed.

Nested closures is a different problem than "callback hell". Callback hell is constantly dropping context on the floor with the stack frames, making every event handler like the "goto" that Dijkstra considered harmful. Nested closures, by comparison, are merely messy, but do not cause context loss. They can be refactored, it's just one more level of abstraction to it than you may be used to, and is often probably not worth it. (For example, do you really need a function that you can pass a closure into and get another one back? Maybe, but that's a level of abstraction than can be very cognitively costly; you better be getting some serious benefit for that cost.)

I have observed before that this sort of DSL-like usage can sometimes make people forget that they are dealing with plain old-fashioned language constructs that can be manipulated with other constructs. Languages like Ruby that allow this fact to disappear out of the source code completely seem to make this more likely. Go won't let you forget you're just creating a bunch of closures, but if you've been trained in an enviroment where a "block" is something special or in such a Ruby "DSL" you may have a way of thinking about these issues that inhibits proper refactoring. This is the general case of what okatsu observes before me, that nothing stops you from making real functions or methods to use as the methods.

Also, Go's relatively recent "bound methods" are really useful for this sort of thing.

Bolt author here. The DB.View() and DB.Update() functions are simply wrappers around DB.Begin(). The View/Update functions were not part of the original API but they're incredibly useful in practice and make the application code more idiomatic.

There are several benefits to using the wrapper functions:

1. It lets you return an error to rollback a read-write transaction. This is typically what you want. That error will get passed through as a return from DB.Update().

2. Because of #1, you can have one error check for application errors and system errors (e.g. disk full) after a transaction commits instead of an error check for DB.Begin() and DB.Commit().

3. If you forget to close a read transaction then it will keep old pages around to maintain consistency but this will cause your database to grow without bound. If you use the helper functions then you're protected from this even in the event of a panic().

The API is modeled around database/sql where applicable. Let me know if you find anything confusing and I can update the documentation to make it more clear.

http://godoc.org/github.com/boltdb/bolt#DB.Begin

It would be nice if you mentioned the DB.Begin() etc. in the Readme, not only the Update() & friends, so that a casual reader would know he has a choice here. I couldn't find it mentioned there, so how could I know I have to search for it among dozens functions in godoc?

Never really tried yet, but I think when using database/sql.Tx, the following snippet would be quite idiomatic, and I believe it should be perfectly safe:

    tx, err := mydb.Begin()
    if err != nil {
        return err
    }
    defer tx.Rollback()

    _, err = tx.Exec('...')
    if err != nil {
        return err
    }

    return tx.Commit()

Thanks for the suggestion. I added it to the README (33e7a07):

https://github.com/boltdb/bolt#managing-transactions-manuall...

The "defer tx.Rollback()" makes me cringe a little since it's confusing when you read the code top-down. It also throws out the error returned from Rollback(). Granted, the only possible error is if the DB is closed so it's not a big deal in most situations.

How painful does this look?

    func action(...) (err error) {
        tx, err := db.Begin()
        if err != nil {
            return fmt.Errorf("preparing transaction: %v", err)
        }
        defer func(tx *sql.Tx) {
            if err != nil {
                _ = tx.Rollback()
            } else {
                err = tx.Commit()
            }
        }(tx)
        // do something
        return
    }

I would actually prefer to have the Begin, Commit in the readme and then a recommendation to use the Update and View. It might just be a difference in learning/understanding, but I understand the reasoning behind Update, View after reading the comments regarding the caviats of manually dealing with the transactions.

I just added it: https://github.com/boltdb/bolt#managing-transactions-manuall...

You'll be damned if you have to read the godocs of a DB you want to use. =P

> Perhaps there could be a way where transactions a manually managed.

If you take a look at the code, db.Begin is available and gives you access to the underlying transaction. You can use it as you want, however you'll need to be aware of catching any error, rollbacking, committing etc... all that is taken care of for you in db.Update (or db.View for read-only stuff)

This style is also consistent with the rest of the standard library, such as filepath.Walk or sort.Search.

Ah, you're right. But it would be nice if they showed this in the Readme, along the "wrapped" examples.

Actually, going the "db.Begin" way would be more consistent with the rest of the standard library. Most of the Go APIs go to great length to be "external" (i.e. iterator-like), e.g. bufio.Scanner, and (of special note in this case!) database/sql.Tx! It appears to me the "internal"/callback APIs are introduced in those places where there are really compelling reasons, which justify breaking the consistency: e.g. for filepath.Walk, there's recurrence, so it's mostly giving up to the obviously K.I.S.S. way; in sort.Search, the benefit is that callback allows to evade need for generics/reflection.

On the other hand, the Go API for App Engine also uses a closure for a transaction:

https://cloud.google.com/appengine/docs/go/datastore/transac...

> I can't say that I like the idea of giving a function as a parameter to the Update

I remember that was a common pattern at Google when creating APIs interfacing with BigTable which also is a KV store. The premise is that you don't need to load everything into memory at once and instead do your work (for Update operations) or just pick interesting fields (for View operations) in smaller increments in your callback function.

> functions within functions, that can't really be refactored out or reused

This is functional programming but the choice of this paradigm does not impact code reuse. You can pass proper functions or methods and don't have to stick to lambdas.

Overall I think the concept is spot on.

I'm still new to Go, but the functions don't have to be anonymous, right? What would be wrong with defining all of your functions somewhere (for updates, iterations, etc.) and passing them by name to Bolt's?

You could name your functions, but you won't really be able to parse in any data. Unless you wrap then Update and View functions as well.

As others points out there's nothing wrong the Bolt code, it's fine and pretty common, I just don't like it.

> Unless you wrap then Update and View functions as well.

This is how I'd do it.

I think this is relatively standard practice for writing thread-safe shared data structures. For example, Elixir agents: http://elixir-lang.org/docs/stable/elixir/Agent.html

http://npf.io/2014/07/intro-to-boltdb-painless-performant-pe...

Namespace clashes in software are starting to become a real problem:

http://developer.amd.com/tools-and-sdks/opencl-zone/bolt-c-t...

I'm trying to understand the use case for a persistent embedded DB for Go. SQLite is embedded all over the place, often in clients.

But presuming Go is a server language, and you are running stateless web nodes, why do you need persistence?

It's awesome because it's just a go package - no external dependency... Your application still ships as a single binary.

Note that using a sqlite wrapper like https://github.com/mattn/go-sqlite3, you also don't have any external dependencies and your application still ships as a single binary. However, there are other advantages of a Go-native storage library (e.g., doesn't require cgo, better integration with tools like pprof, etc.)

Stateless nodes is just one possible type of architecture - nothing in the language itself forces you to adopt it.

Of course. But I'm asking b/c I can't think of a situation where I would want to have stateful web nodes.

Management interface for monitoring, irc/chat host, pretty much anything interactive to a persistent backend.

Go can be used for far more than just web nodes.. it can be used for any number of network or stand-alone applications.

My use case would be small local tools that need to keep some state. Essentially anywhere a SQLite would be useful. I don't think this was designed to serve as a database to back anything more 'serious'.

We use it to store and serve 100s of GB of data with a custom query interface on top.

What's the replication strategy then?

It's not the data authority. It feeds in data from Kafka and structures it in an optimized way for querying. If a node dies then it can copy from another node and restart from an offset in Kafka. Kafka essentially works as a distributed write-ahead log (WAL).

Why and for what do you use that architecture?

Site usage analytics has a lot to do with state in a way. How about a visit counter? Something lightweight that would persist page count/ip address/lastdate values with an ip address key.

I'd love to have something like that built around some serialisation protocol like protobufs. Something that would feel like GAE's Datastore API.

I haven't used GAE's Datastore API but using protobufs on top of Bolt is trivial. Bolt just uses []byte for keys & values. I use gogoprotobufs with Bolt often.

Reply on news.ycombinator.com