I can short circuit an Arduino pin for days, or (attempt to) pull 3 or 4 times the rated current from them - and they just keep on rocking. I've even accidentally hit them with 7.2V or even 12V on the GPIO and not had them let the magic smoke out.
You only have to look sideways past a 5V rail at as RasPi to kill it, and ESP aren't a heap better.
If I can get all the code I need onto an Arduino/AVR, I'll use one. If I need more complex code than I can run on an Arduino but still need GPIO (especially anything 5V tolerant) I'll use an ESP or RasPi (or some other Linux SOC board) with an Arduino (or other AVR) for GPIO. At least for the prototyping stage when I'm using breadboards and moving wires around - if I'm building something permanent (or at least semi permanent) I'll sometimes build a version 2 with level shifters to deal with 3.3-5V stuff, but mostly only once I've got to the "prototype that's got to the PCB stage" iterations. I've almost never killed an Arduino, but I've killer too many RasPis and way way too many ESPs.