Yes, it is the mountains. Tunnels are very expensive. The French TGV lines only reach the Spanish border at Hendaye on the west coast and Perpignan on the east. The fast line to Barcelona is quite recent.
In northern Spain, there is a slow train line that connects Barcelona with Galicia called the "tren estrella," but it stops everywhere and uses old infrastructure, so it is slow. Traveling to Madrid is always fast with the newer AVE lines, and more are being built.
No idea about Portugal. I guess that it is the same situation, and those routes are covered by buses.