Building any appreciable amount of compute capacity (on the scale of, let's say, a single large datacenter in northern VA) inside a mountain also becomes a huge heat rejection problem, since with the exception of the 0.00000000001% that leaves the facility as photons down a SM fiber cable, 100% of the energy consumed is turned into heat.

So you build your stuff inside a mountain and run a bunch of big ass liquid cooling loops to heat exchangers in the atmosphere outside the mountain. The requirement of the cooling system being out in the open air means it can also be hit by air strikes, an adversary doesn't need to penetrate the mountain, just take out its cooling.

You'd have to build a fully underground cooling system that goes to a nearby river (or huge lake) and use the cold water from that, which would get costly quick.