Hacker News Reader

Comment by ThePhysicist | Hacker News Reader

If you're interested in using Bloom filters in the backend, we (at DCSO) have written efficient and interoperable open-source implementations in Python and Go, also using FNV-1 hashing:

https://github.com/DCSO/bloom (Go version)

https://github.com/DCSO/flor (Python version)

The Go version comes with a command line tool that allows you to use Bloom filters on the shell.

View on HN

One note: if you've done everything correctly under the hood, instead of reading and writing the entire filter into and out of memory, just mmap the file and operate on it directly. It turns out to be pretty fast, especially on SSDs and you can make both many filters and test them quickly this way, or make filters that are practically larger than RAM and swap partitions and make absolutely enormous sets.

Yes good point, we actually have this on the roadmap!

hey, curious why your variables in the go version are so terse? makes following along a little tough!

It is Go 'style'.

The abstractions are supposed to be self explanatory and simple and thus the variable names do not have to serve as documentation.

I think it is Go style to name things that come in as interfaces with short names. When you have an io.Writer, there's very little else you can name it other than "w" that would have any additional meaning or utility. When you have a variable or struct field of a concrete type, it can and should have a meaningful name, even in Go.

Ironically, I see several "input io.Reader" sort of things in that code, where as "number of hash functions" is k. A few minutes with go-rename would clean that right up, though.

Go to https://golang.org/pkg/net/http/#Client and use your browser to search for the string "struct {" and have a look. https://golang.org/pkg/os/ is another page with a number of struct declarations.

Reply on news.ycombinator.com