You're right that a tool which runs under the same user could read your config file and thus could access to your secrets.

But there is one main difference: that tool would need to do so explicitly, with the intent of reading (and possibly exposing) your secrets. For me, that's a huge difference from having the secrets being implicitly available to the process through the processes environment.