Your nerves don't care how heavy something is. Fine muscle control is just as intensive as anything to the nerves. Talking uses waaaay more nerve and muscle control than lifting heavy weight.
Breaks are required so that the blood can carry away waste, and replenish glucose and oxygen. That's all.
The ability to focus without fail is not unlimited. When you're holding more than your bodyweight on your shoulders, the demands to simply maintain balance are significant. This means a lot more than just muscles holding a static position. If you are not very focused on the sensations which indicate a change in balance somewhere, you will lose form and possibly hurt yourself.
Whatever the mechanism within the brain, the breaks are also necessary there. Of course the muscles need recovery time, but the brain does also.
What I have anecdotally observed is that regularly lifting closely to your maximum (particularly at higher levels) will inevitably lead to accumulated fatigue. Whatever the root cause of that fatigue is (whether muscoskeletal, endocrine or nervous) is a factor that does lead to degraded performance over time.
The literature is lacking in longer-term studies, so I'm curious to see whether my anecdotal experience is pure confirmation bias or there is a mechanism behind it.