Hacker News Reader

Sampling via just enabling it for some hosts/partitions is one solution (if you're producing 100M entries a day ... probably could just grab 1/100 of those for parsing).

Another solution is pre-processing (serial dupes are not forwarded).

Another solution is heavily reduced logging (ERR or higher only on prod hosts).

These can be used together and be very helpful.

View on HN

All technical workarounds for bad pricing.

Processing that amount of data is going to be expensive, regardless.

I think it’s a situation where splunk doesn’t have a motivation to reduce cost as they can charge a lot and customers pay.

So it doesn’t need to be expensive, naturally, it just is.

No, it's orders of magnitude cheaper than Splunk.

They mean doing the processing that Splunk does is expensive so there simply needs to be less data going into the system (via the pre-processing steps I mentioned) above in order to keep costs sane.

With that said Splunk should offer such a pre-processing product (maybe it does?) which would probably increase their moat even though it reduces revenue somewhat in the near term.

Splunk has Edge Processor and Ingest actions to filter, mask and route data.

https://www.splunk.com/en_us/blog/platform/introducing-edge-...

https://www.splunk.com/en_us/blog/conf-splunklive/ingest-act...

Reply on news.ycombinator.com