I think it's probably about the timing. I'm not an expert, but I believe that opening the side borders requires cycle exact timing at both edges of the screen, and when you have characters enabled on the screen, the VIC-II graphics chip ends up 'stealing' memory cycles from the CPU every 8 scanlines to fetch character and font data from RAM. These lines are known as 'badlines'.

Across an entire scanline I think you normally get something like 63 6510 cpu cycles to do 'work' in, but only 23 if you hit a badline - keeping in mind that some instructions take multiple cycles to execute. This probably makes the timing difficult or impossible to manage with the characters turned on.