You can also have a Shortcut that toggles the setting when you open and close specific apps.

I just did it and it’s very simple:

- in the Shortcuts app, go to Automations

- add a new one and pick “App” as the trigger

- choose the apps you want color in and pick “run immediately” and on open and close

- on the next screen pick “new blank automation”

- in the new shortcut add “Set Color Filters” and set to toggle

You can also make different automations for open and close, but toggle works as long as you toggle the effects manually. But in this case you can just toggle it back manually.