>>>What keeps them from changing or trying out new ways of designing/building? Is it entrenched interests that resist any changes?
There are immense barriers to entry. When I tried to boot-strap my defense startup, a Lieutenant Colonel I knew (also a VP at a defense company) told me it would take over a year just to do the paperwork to sell stuff to DoD. It's why I focused on South-East Asia. As long as I steered clear of ITAR products, I felt corruption in SEA was easier to navigate that US bureaucracy. The requirements are even more onerous if you are producing anything of a sensitive nature. My dad knew a business owner in the Gainesville area who got out of the drone-making business because the physical security upgrades that his DoD client required were such a burden. Successfully navigating these hurdles requires program managers with niche domain knowledge on the DoD's procurement, certification, and funding processes.
>>>Isn’t it in the best interest of everyone to actually deliver on projects? If they don’t how is it that they continue to be relied on by the military?
Because of the issues I mentioned above, oftentimes there are few companies who have both the infrastructure and the domain knowledge to even attempt to deliver a product. Some of the engineering problems are also REALLY difficult. But if Lockheed Martin sucks at making a fighter jet, and Boeing sucks at making a fighter jet.....who do you turn to? If you need a jet engine, you go to Pratt & Whitney or you go to Rolls Royce. No amount of VC startup incubation is gonna give you a reliable high-bypass turbofan engine in a reasonable timeframe. The Chinese have been throwing resources at that particular problem (fighter jet engines) for decades and they are STILL behind the US and Russian offerings.
If the Navy doesn't like its Little Crappy Ships....it's not like there is an abundance of efficient shipyards in the US anymore. The few existing ones are practically on life-support and reliant on Navy contracts. A good shipyard needs experienced blue-collar tradesmen (welders, electricians, etc.), which are already in short supply in the US. So again this is another field that is not quickly rectified or disrupted. As a consequence the government is stuck saying "Sigh....here's another bag of money....try to get it right this time." At this point, pushing funding to trade schools in order to get a bunch of noobs into the shipyards, and then being tolerant of them producing "good enough" ships for a decade, would be an improvement over our current situation IMO...
The government is at fault too. We've suffered from a lack of consistent strategic vision, and frequent adjustments to policies, budgets, and tactics lead to engineering changes that are massively expensive. The military is constantly moving the goalposts, so industry responds by rolling its eyes and saying "You want it to do WHAT?!?....Yeah, that's gonna cost another $100 million..." There's also an incestuous relationship between program managers who are military officers or Federal government employees, and industry. The .mil/.gov types leave the service and then, after a short cool-off period to not run afoul of the laws, get hired by the same companies they were pushing billions of dollars towards. It's the same revolving door problem that Wall Street and the SEC have.
It's a depressing mess and I don't expect it to get better until the bodies REALLY start to pile up and the American populace turns its rage towards The Powers That Be instead of each other.