"Like all proprietary AI software, I'm sure this app will generate battle plans without explaining why they make sense or what intel they were based on."

AI does what you tell it to do. As you likely know, a NN model never outputs definitive results, we clamp/truncate them to make it so. In areas of life and death you can choose to keep this information and lay it out as part of the way for an AI to establish the reasons behind its decisions and its level of certainty.

You're describing poor practices that come out of likely poor understanding of tech, poor training, maybe perverse economic incentives for Palantir to present its technology in a more assertive way than it should, but we're simply guessing.

AI is what we make it to be, literally.