At my current employer, we are currently implementing Splunk. And it's taking forever and they do charge arm and a leg for their offering. I don't mind if a good product costs money but you shouldn't need a consultant on premise just to configure your logging solution.
- Make frequent visits to answers.splunk.com. It has a very active community, and I've frequently been able to type "how do I do X in Splunk" into Google and found multiple answers on Splunk Answers.
- Deployment Server. Make friends with it. In a perfect world, it should hold your configurations for all Indexers, Heavy Forwarders, and Forwarders. If you're having to populate $SPLUNK_HOME/etc/system/local/ yourself, you're doing it wrong.
- Make friends with the "splunk btool config_file_name list --debug" command. That makes it dead simple to know which configuration options a Splunk install is running. Append "| grep -v system/default" on the end of that command to filter out the defaults and you'll more easily see which of your options are being used.
- If you have the cash, attend Splunk Conf and load your schedule up with presentations. It's worth every penny.
Hope that helps.
We use Splunk for logging and I wrote custom app for fraud detection in financial security field, custom events correlation and alerting.
It's very good tool for enterprise data analytics as well as for any custom dashboarding and event processing.
Splunk consultant maybe needed if you have complex enterprise deployment scenario or wish to develop really advanced apps - but configuring logging solution?
It's a point and click exercise.
It was cool software, a bit slow (this was several years back). But with things like Elasticsearch catching up release over release (even if you don't use it directly, other platforms will build on it), Splunk is no longer the totally unique thing they used to be. I can't figure out their $8bn+ market cap with revenue under 500M and costs increasing as they grow.