You'll need to manage some stuff yourself, and assemble your own dashboards and stuff, so there will be some labor involved. That being said, I doubt it will be more painful than managing an ELK stack: there are just too many ways you can destabilize a cluster with it.
ClickHouse clusters from my experience are ridiculously scalable, fast, and stable. There are several other accounts to back that up, and a good case study is Cloudflare, which uses it to store and query all of their logs and metrics from all data centers (that's quite a few PB of data).
There are some projects on GitHub you can use to get inspired, but what you need is pretty much a ClickHouse cluster, Grafana, and a Log Shipper.
Depending on your volume and situation, there are hosted options or you can roll your own on-prem.
We haven't moved off of splunk for all of our logs, but have reduced the volume going there significantly.
Most of the data that was needed by business folks was sent to segment in terms of events and for genuine errors were sent to sentry with stack traces and converted most of the logs to metrics.
This helped us cut down a lot of $$ for logging and monitoring. Elastic search is a great alternative, but it takes an entire team to maintain it. I'm not sure, if its worth the time and effort.
If you'd rather not spend the time managing an ELK stack, there's a lot of logging options (disclaimer: I work at one of them, https://logdna.com). It'd be helpful to unpack what you mean by "credible" competitors however. Our product, and most others in our space frankly, can't match Splunk in terms of feature-set today, so knowing what is most important to you would be helpful. For example, if you're looking for basic log storage and search, you'll have tons of options. If you need compliance, that would narrow the field a bit.
I'm happy to chat if you're interested in giving us (LogDNA) a shot or if you have other general questions, feel free to shoot me an email, I'm just peter@[ourcompanydomain]. I'm not in sales so I don't really have any incentive to push you towards us unless it makes sense, and a lot of our competitors have great products as well so happy to try and point you in the right direction.
Hopefully that's helpful information!
It can also integrate in your existing Splunk setup if you want: https://www.graylog.org/post/graylog-splunk-integration-is-n...
(which allows you to ship data to Graylog and/or Splunk and setup analytics in both as needed)
I think that is smart.
P.D: Better to not use nosql but well marketing of nosql back in the days...
p.D2: Most logs are pure NOISE. What I wish to have in some ways to reduce them when incoming and the distilled put in a rdbms. I think this is the best for most but not see much info about this.?
On the other hand, if your primary use-case is near-term searches and reviews of data (e.g. "we just need to see the last 90 days of information for troubleshooting and stats"), you'll be pleasantly surprised with the capabilities in both Graylog and ELK without the additional overhead of Splunk. I'll say, too, that I found working with Filebeat for ingestion to be a lot easier and automation-friendly than working with Splunk forwarders—so much so that we're basically adopting an "only use Splunk for indexing" approach to ingestion and using everything else to get data into it.
Elasticsearch (or ELK/EFK) is really eating their lunch but if you have any experience managing an Elasticsearch cluster at scale you might have some reservations about it. If you'd like to audit it I might suggest using a Helm chart like this one[0] to deploy a full stack to a Kubernetes cluster. Even if you aren't using k8s for production it's a fast way to get a handle on it. You can deploy a test cluster easy on GKE quick for nearly no money. If that seems too inaccessible, there are a few good docker-compose[1] implementations too that can get you going right away.
I don't think it's quite ready for production, but we were already using Prometheus and Grafana so I was auditing Loki[2] and was pleasantly surprised. Though at the time I determined it would not be a substitute at scale for Splunk or ELK but could be viable for many people and the project is moving fast to do cool stuff.
[0] https://github.com/helm/charts/tree/master/stable/elastic-st...
For really good user/account dashboards, which is the most common first tool companies build, you check try windsor.io which does most things out of the box with no setup
If you're looking to build custom dashboard and are okay with spending time actually coding and maintaining it (just code snippets, not a full ELK stack) try retool.com
We're a relatively new, relatively small player in this space, differentiating by real-time anomaly detection and tooling to help our customers keep costs low (by filtering out logs known to be irrelevant).
We got SumoLogic. I'm very happy with it relative to ELK.
It's almost as good and much cheaper. The limitations would be that dashboards aren't as customizable and that some advance searches are harder.
I don't know about your use case specifically, but for our case (~100GB of searchable data at any given moment), searching takes a few seconds for basic queries.
Runs smoother than the ELK stack.