Hacker News Reader

Comment by netingle | Hacker News Reader

It’s defiantly something that’s missing from the readme, and perhaps not that obvious in the grafana explore view either - but it is there! You can push a regexp match server side and have that distributed to each Loki node, giving you distributed grep.

Will make it more obvious. Davkals has an iteration of the UI that makes it a separate field, which will also help.

View on HN

Maybe I'm not understanding this - the docs say Loki is all about storing compressed log data with metadata, such that only the metadata is indexed. Are you saying you can search the compressed, unindexed data using regex? If so, wouldn't that potentially be incredibly slow?

The good thing about this is that the grepping can be parallelised and distributed on to several nodes. Having said that, once you select the relevant metadata right, you should be able to narrow it down enough for the queries to be snappy enough.

While this will definitely be slower than something that indexes the contents, you'll be able to store much more in Loki at much lower costs.

Yeah, I am thinking about the worst case here, but never underestimate the power of your users to perform very silly queries!

For your hosted service, will you put in place any restrictions on, for example, the size of the time range that can be queried?

Also for your hosted service, will the degree of parallelisation vary by pricing tier?

What are you using to run the regex? ripgrep could make up for some of the loss from not having it indexed.

https://github.com/grafana/loki/blob/master/pkg/iter/iterato...

Looks like the Go regex lib, which isn't super performant, so it could potentially be improved if it ends up being an issue.

There is some documentation about the Loki search syntax in the Grafana docs:

http://docs.grafana.org/features/explore/#logs-integration-l...

Please make it more obvious with exact examples of how to do this.

Reply on news.ycombinator.com