Ideally, unlike in TeX, additions to nesting levels wouldn’t require a re-flow (i.e. typing a bracket wouldn’t make the code constantly shimmy around on the screen), but rather these would just be changes in the perceived size of these characters, while keeping them where they are on a monospace grid (since a lot of code assumes a pure monospace layout for indentation et al.)
In my imagination, this would work by just having a set of bracket/paren/brace/etc. graphemes in each font, that have increasingly-long ascenders/descenders, while the base size of of the “functional” part of the grapheme remains constant. As if there were five versions of the letter g, with an increasing length of the stem on which the lower hook rests.
As such, these magnified brackets would be rendered similarly to the way emoji are rendered in terminals: the grapheme would eventually “leak out” of its monospace box (in the emoji case to the sides; in this case above and below), to the point of eventually overlapping surrounding graphemes, rather than re-spacing the text to give it room.