Further down the line you stop needing to know where you are (global view, understanding), and you only need to deal with implementation: actually doing what you need to do. Implementation can be often very complex on its own too, that's why algorithms are so scary at the beginning... and text is a better tool here. There's not much more structure to distill, and some parts are tricky, and you have edge cases and you often need to get your hands dirty. You need to keep all this complexity in mind at the same time anyway, and after you are done you can treat the implementation as a black box. So text is superior here.
That's how we manage complexity. Now we only need tools that can represent it better. But maybe it's not even visual programming languages, but rather visual system "structurers"? There are other parts of programming that could use better visual cues, and visual editors for visual stuff (colors, shapes, borders, etc.) will always be relevant, but those are all very different concerns.