I had to go through this last year with my previous company. Splunk's licensing is outrageous. They cap the data transfer on a PAID license arbitrarily. Even getting coordinated with their sales to just pay them was a chore. They do nothing to foster open development on their platform. IMO, the open offerings actually seem designed to make it impossible to improve/contribute/extend in a cost effective way. It is also ill equipped to deal with Kubernetes even though they did release a reasonable if Windows Guy bloated example of a k8s deployment at the beginning of this year. I was completely baffled when I found out it was a Django project. It feels like something from the depths of enterprise Java.

Elasticsearch (or ELK/EFK) is really eating their lunch but if you have any experience managing an Elasticsearch cluster at scale you might have some reservations about it. If you'd like to audit it I might suggest using a Helm chart like this one[0] to deploy a full stack to a Kubernetes cluster. Even if you aren't using k8s for production it's a fast way to get a handle on it. You can deploy a test cluster easy on GKE quick for nearly no money. If that seems too inaccessible, there are a few good docker-compose[1] implementations too that can get you going right away.

I don't think it's quite ready for production, but we were already using Prometheus and Grafana so I was auditing Loki[2] and was pleasantly surprised. Though at the time I determined it would not be a substitute at scale for Splunk or ELK but could be viable for many people and the project is moving fast to do cool stuff.

[0] https://github.com/helm/charts/tree/master/stable/elastic-st...

[1] https://github.com/deviantony/docker-elk

[2] https://grafana.com/oss/loki