A big reason seemed to be compatibility with the RocksDB on-disk file format.

https://www.cockroachlabs.com/blog/pebble-rocksdb-kv-store/