It would really be interesting to learn more about their domain. Builds logs you have control over and should be able to keep clean and actionable. Service or batch job logs could present a more difficult problem though. It would be useful to try and spot a key message halfway through a failed 20 minute process that is the reason you hit an error at the end. Again though using logging levels and grep should usually be enough. You can also flag excessively noisy code and improve the signal to noise of the output it produces.
In specific with a build system like Bazel you might execute 1,000,000 actions on a build producing lots of internal output but you only see the errors, and at most you have a few hundred lines to look through. That is managed in a few key ways:
- Test output is completely hidden unless that specific test fails
- Build actions only produce output when they have an error, and it’s easy to keep it that way (because they stand out, and can be quickly fixed)
- Bazel does not tell you about every action that is run only ones that fail or take or a long time