Hacker News Reader

Can someone from FB chime in with some info how much storage is needed for the logs/data? Say, for 1 GB of raw input logs from a http server (nginx/apache), when stored in LogDevice would they take notably less space on disk (compression), or more (overhead)? This interests ne for evaluating resources/costs I'd need to prepare if I were to deploy it...

View on HN

These numbers really depend on the compressibility of the content, compression scheme and the type of batching used. The metadata overhead is fairly minimal. LogDevice allows you to configure this on either the client, sequencer or rocksdb level.

Is some form of compression enabled by default, without having to tweak options?

No, compression is disabled by default. You can enable compression either on the storage layer, or by enabling batching and compression on the sequencer, or by using the buffered write API with compression on the client.

have you had a look at https://github.com/oklog/oklog https://www.youtube.com/watch?v=gWWK2eyZ-sc

I think it's fairly simple and might be enough. Can't comment on storage requirements thou.

Dang, I was looking at oklog earlier, but looks like it is archived now...

Reply on news.ycombinator.com