I can't see any issue with representing logic abstractly with symbols. It's the same for calculus. Of course the ideas we're representing aren't actually the things we use to represent them, the same as written communication.
Non-textual programming has been explored to some degree, such as Scratch, but it's not seen as much of a useful thing.
>Instead of solving the problem, we're stuck down describing actions and formulas to the machinery. No IDE does anything to actually address that problem.
Describing actions and formulas to a machine in order to make it do something useful is pretty much the definition of programming. IDEs make it a more convenient process.
Unless you want to directly transplant the ideas out of your neural paths into the computer, maybe some AI computer in the future based on a human brain, this is how it's going to be.