Hacker News Reader

Maybe I'm not understanding this - the docs say Loki is all about storing compressed log data with metadata, such that only the metadata is indexed. Are you saying you can search the compressed, unindexed data using regex? If so, wouldn't that potentially be incredibly slow?

View on HN

The good thing about this is that the grepping can be parallelised and distributed on to several nodes. Having said that, once you select the relevant metadata right, you should be able to narrow it down enough for the queries to be snappy enough.

While this will definitely be slower than something that indexes the contents, you'll be able to store much more in Loki at much lower costs.

Yeah, I am thinking about the worst case here, but never underestimate the power of your users to perform very silly queries!

For your hosted service, will you put in place any restrictions on, for example, the size of the time range that can be queried?

Also for your hosted service, will the degree of parallelisation vary by pricing tier?

What are you using to run the regex? ripgrep could make up for some of the loss from not having it indexed.

https://github.com/grafana/loki/blob/master/pkg/iter/iterato...

Looks like the Go regex lib, which isn't super performant, so it could potentially be improved if it ends up being an issue.

Reply on news.ycombinator.com