Bolt — an embedded key/value database for Go

If you want data persistence in your Go application, most likely you're thinking of using some database. The easiest and probably the most convenient for deployment are embedded databases. There are many wrappers for C databases, however Go developers usually prefer pure Golang solutions.

Bolt is the way to go: it's a pure Go embedded key/value database, which is easy to use for persistence in your Go projects. Bolt is similar to LMDB, which many consider the best among state-of-the-art modern key-value stores. Just like LMDB, and unlike LevelDB, BoltDB supports fully serializable ACID transactions. Unlike SQLite, it doesn't have a query language, and is much easier to use for common things.

Bolt saves data into a single memory-mapped file on disk. It doesn't have a separate journal, write-ahead log, or a thread for compaction or garbage collection: it deals with just one file, and does it safely.

History of LMDB and Bolt

In 2011, Howard Chu introduced MDB, a memory-mapped database backend for OpenLDAP, later renamed to LMDB (Lightning Memory-Mapped Database). Its super clever design made the key/value store very fast for reads and safe for writes. MVCC-like design with copy-on-write B+trees avoids locking database for reads during writes, and makes it possible to provide full ACID transactions support. (You can read more about how LMDB works here.)

Bolt initially started by Ben Johnson as a port of LMDB to Go, but then the two projects diverged. The author of Bolt decided to focus on simplicity and providing the easy-to-use Go API.

Differences between LevelDB and BoltDB

LevelDB, started by Google, while also being a key-value ordered database, is quite different from BoltDB. The biggest difference for users is that LevelDB doesn't have transactions. Internally, they are also very different: LevelDB implements a log-structured merge tree (LSM tree). It stores sorted keys and values in many files split by "levels" and periodically merges smaller files up the level into larger ones in a separate compaction thread. (You can read more about it here.) This makes it very fast for random writes (especially on spinning disks), but slower for reads. This also makes LevelDB performance unpredictable: it may be initially good when the database is small, but much worse when it grows. Separate compaction thread is known to cause problems on servers (that's why there are many LevelDB forks from companies such as Facebook and Basho, which try improve server performance of LevelDB, mostly by improving compaction). LevelDB is written in C++, but there are Go wrappers (jmhodges/levigo) and pure Go re-implementations of it (syndtr/goleveldb, leveldb-go).

Bolt uses a single memory-mapped file, implementing a copy-on-write B+tree, which makes reads very fast. Also, Bolt's load time is better, especially during recovery from crash, since it doesn't need to read the log (it doesn't have it) to find the last succeeded transaction: it just reads IDs for two B+tree roots, and uses the one with the greater ID. Bolt is simpler.

How to install Bolt

Obviously, you must have Go installed before following this guide. Run this command:

go get github.com/boltdb/bolt/...

to fetch and install the package and bolt command-line utility.

Using Bolt to store data

Opening database and initiating transactions

Bolt stores everything in a single file, making it very easy to use and deploy. I'm sure your users will be happy to discover that they don't need to configure database or — gulp! — hire a DBA to manage it. What you do with this file is just open it. If the database file you're trying to open doesn't exist, it will be created.

In this example we open database blog.db in the current directory:

package main

import (
    "log"

    "github.com/boltdb/bolt"
)

func main() {
    db, err := bolt.Open("blog.db", 0600, nil)
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()

    // ...
}

After you opened the database, you will use two types of transactions to deal with it: read-write and read-only transactions. Read-write transactions are started with db.Update function:

err := db.Update(func(tx *bolt.Tx) error {
    // ...read or write...
    return nil
})

As you can see, you pass a function to db.Update as an argument, and inside it you get or set data and handle errors (we'll get back to it in a moment). If you return nil from this function, the transaction gets committed to the database, but if you return an actual error, it gets rolled back, and everything you did inside this function won't be committed to disk. This is very convenient and natural, as you don't have to care about rolling back transactions manually: just return an error and Bolt will handle the details.

Read-only transactions are initiated with db.View function:

err := db.View(func(tx *bolt.Tx) error {
    // ...
    return nil
})

Inside it you can read and iterate through values, but cannot change them.

Storing data

Bolt is a key-value store that provides an ordered map. This means that you access values by their names, just like the ordinary Go maps, but in addition to this — since the keys are ordered — you can iterate through them.

There is another level on top: key-values are stored in buckets. You can think of buckets as collections of keys or as database tables, if you're into it. (By the way, buckets can contain other buckets, which is really useful.)

Here's how you update your database:

db.Update(func(tx *bolt.Tx) error {
    b, err := tx.CreateBucketIfNotExists([]byte("posts"))
    if err != nil {
        return err
    }
    return b.Put([]byte("2015-01-01"), []byte("My New Year post"))
})

We created a bucket called "posts" if didn't exist, and then put a value "My New Year Post" under key "2014-01-01". Note that bucket names, keys, and values are byte slices.

Reading data

After you put some values into your database, you can read them back like this:

db.View(func(tx *bolt.Tx) error {
    b := tx.Bucket([]byte("posts"))
    v := b.Get([]byte("2015-01-01"))
    fmt.Printf("%sn", v)
    return nil
})

You can read more examples of how to deal with transactions and iterate over keys in Bolt's README.

Easy persistence in Go

Bolt stores plain bytes, but what if we want to have some structure for our data? It's easy to do so using just the Go standard library: we can store JSON or Gob-encoded structured data. Or, if you don't limit yourself to the standard library, you can also use Protocol Buffers or any other serialization solutions.

For example, if we want to store a blog post described by this struct:

type Post struct {
    Created time.Time
    Title   string
    Content string
}

We can first encode it, e.g. with JSON, and then put encoded bytes into BoltDB:

post := &Post{
   Created: time.Now(),
   Title:   "My first post",
   Content: "Hello, this is my first post.",
}

db.Update(func(tx *bolt.Tx) error {
    b, err := tx.CreateBucketIfNotExists([]byte("posts"))
    if err != nil {
        return err
    }
    encoded, err := json.Marshal(post)
    if err != nil {
        return err
    }
    return b.Put([]byte(post.Created.Format(time.RFC3339)), encoded)
})

When reading data, just unmarshal the structure back from bytes.

Using `bolt` command

BoltDB ships with a bolt command-line tool, with which you perform consistency check on the database and get usage statistics.

Usage:

    bolt command [arguments]

The commands are:

    bench       run synthetic benchmark against bolt
    check       verifies integrity of bolt database
    info        print basic info
    help        print this screen
    pages       print list of pages with their types
    stats       iterate over all pages and generate usage stats

Use "bolt [command] -h" for more information about a command.

For example, to check our blog.db database for consistency, verifying that every page is accounted for, run:

bolt check blog.db

Conclusion

Bolt is a really nice simple database and I think it has a bright future in Go ecosystem. It is production-ready and is already successfully used by some notable companies, such as Shopify. It works best for read-heavy projects.

Source code and license

BoltDB on GitHub: https://github.com/boltdb/bolt Author: Ben Johnson (@benbjohnson) License: MIT.