As part of my current role, I setup and managed a unified logging infrastructure as part of a SIEM install. We're talking ingesting logs in the order of TBs a day (Lots). Here are my observations:

1. Syslog type logs are great when the volume is low, but field extraction can be a pain (unless you're a Regex savant)

2. Structured logs are nice to deal with. Fairly easy to read and a dream to parse. Make sure you have a good ETL process in place to process these logs so they are searchable. 3. If you have a lot of logs, they are only as good at the usability of the search on that corpus of text.

I use Splunk to ingest the logs and once the extractions are in place, it is a dream to search. Other options like ELK are available and work pretty well too.

Show the devs the advantages of having structured logs indexed and ready to search. No more: SSH > cat | grep | tail | less | more

Real-time functionality and the ability to alert on certain conditions is an added bonus.