I do a similar thing with my shift keys. Single press of left/right shift places the appropriate parentheses. Holding acts like shift.
Then, I use Karabiner like so: https://cl.ly/3W2X1A2j3y3l (screenshot of my settings).
I don't go too crazy with things. But I have the caps > ctrl+esc, shifts > parens when pressed alone, and I have my backtick/underscore key swapped so a single backtick press gives me an underscore, and I do shift+hyphen to get a backtick (which I need far less frequently than I need underscores). Sometimes, when I feel like going really crazy, I swap my numbers and symbols so I get symbols as unshifted keypresses and numbers with shifted keypresses. But that sometimes gets confusing to years of muscle memory.
EDIT: You can find Karabiner here: https://pqrs.org/osx/karabiner/
EDIT2: I also have my pipe symbol & backslash swapped (mostly for Elixir, where pipes are more common than in, say, Python). You can setup your own private.xml with Karabiner to do that like so: https://gist.github.com/bobwaycott/d3a52718927519f3a11fbff8b...