It is surprising to people outside the DoD, but most applications are exceedingly mundane - I would lump a lot of them under "database frontends," like personnel, finance, and logistics systems. One problem there is that, in many cases, DoD or its contractors end up writing custom code for those things rather than using existing solutions, so projects get delayed as feature creep sets in. There is a bigger push for using commodity software but it's slow in coming.

For things where you truly need custom code - like missile guidance systems, avionics, specific process oriented tools for crunching data (intel or otherwise), open sourcing the core application is probably not going to help anyway. One problem though is that, increasingly, people want to use open source libraries for things. Take the data crunching - people want to use R, Python, Hadoop, whatever. This is where people are running into issues. And good luck getting those tools into close environments (e.g. classified networks) - many places do not have the resources in manpower or expertise to custom build the environments they need, so they couldn't use the newest shiny stuff even if they wanted to, even if their IA shop allowed it.

As to your last question, not a lot of examples to mind (maybe Accumulo like someone mentioned elsewhere), but another factor is that there are few programmers that are actually DoD civil servants - most stuff is written by contractors and DoD folks don't usually have the experience or knowledge necessary to even understand what they're getting at a technical level in order to recognize that what they have is something worth open sourcing (which might take some work). I'm not saying it's bad everywhere - I have met some pretty awesome technical folks that were GS's - but it's very uneven.

(disclaimer: USAF vet and still involved with the DoD)