Pilots generally do have some level of engineering background, in order to be able to understand possible in-flight issues, but they're not analogous to software engineers. They're analogous to software operators. Software engineers are analogous to aerospace engineers, who absolutely do understand the internals of how turbines work because they're the people who design turbines.

The problem with software development as a discipline is its all so new we don't have proper division of labor and professional standards yet. It's like if the people responsible for modeling structural integrity in the foundation of a skyscraper and the people who specialize in creating office furniture were all just called "construction engineers" and expected to have some common body of knowledge. Software systems span many layers and domains that don't all have that much in common with each other, but we all pretend we're speaking the same language to each other anyway.

I think the important point here is that even pilots dont know the full mechanics of a modern jet engine (AFAIK at least, I don't have an ATPL so not 100% on the syllabus). They may know basics like the Euler turbine equation and be able to run some basic calculations across individual rows of blades, but they most likely will not fully understand the fluid mechanics and thermodynamics involved (and especially not the trade secrets of how the entire blades are grown from single crystals).

This is absolutely fine, and one can draw parallels in software, as a mid level software engineer working in an AWS based environment wont generally need to know how to parse TCP packet headers, despite the software/infrastructure they work on requiring them.

Yes and no, for a private pilot license you are taught through intuition and diagrams. No Navier Stokes, no Lattice Boltzmann, no CFD. The FAA does not require you to be able to solve boundary condition physics problems to fly an aircraft.

Modern jet pilots certainly know much less about airplane functions than they did in the 1940s, and modern jet travel is much safer than it was even a decade ago.

Many of them know the checklists for their model of aircraft. The downside of the checklists is that they sometimes explain the "what" and not the "why". They are supposed to be taught the why in their simulator training. Newer aircraft are going even further in that direction of obfuscation to the pilots. I expect future aircraft to even perform automated incident checklist actions. To your point, not everyone follows the checklists when they are having an incident as the FDR often reports.

most pilots probably don't know how any specific plane's engine works further than what inputs give what outcomes and a few edgecases. larger aircrafts have most of their functions abstracted away with some models effectively pretending to act like older ones to ship them out faster (commercial pilots have to be certified per plane iirc, so more familiar plane = quicker recertification), which has led to a couple disasters recently as the 'emulation' isn't exact. this is still a huge net benefit as larger planes are far more complicated than a little cessna and much harder to control with all that momentum, mass, and airflow.