I second that. I love splunk as well.

Costs can also be reduced by spending some development-effort into abbreviating logs and being smart about deciding what to log and where.