Yes, it should be the responsibility of the build system to control the environment variables set during the build.

What would happen if a build system set one of those, and then somewhere deep in the build system (or even in a shell script called from the build system), someone ran grep in a pipeline

The person adding the 'grep' would need to be aware of whether the build system is using color output or not.

This is also part of why programs like grep are removing environment variables like GREP_OPTIONS: they break scripts.

This seems unfortunate; scripts could just unset GREP_OPTIONS if they care.