At 4000m, the outside pressure is _400_ atmospheres compared to just one inside. It's way, way harder.
Not quite. The composite overwrapped pressure vessels (COPV) we are talking about are easily pressurised up to 5000 psi [1]. That is about 340 atm. They use these tanks to contain helium as a pressurant, and nitrogen for the life support systems.
Elon talked in a tweet about higher numbers. 6k psi and 10k psi in this tweet[2] but it is unclear if he is talking about design works or actual pressures they have flown.
All in all you are right that the difference between the people tank and space is only 1 atm, but that is not where the challenge is in terms of space exploration and pressure vessels.
1: https://ntrs.nasa.gov/api/citations/20110015972/downloads/20...
No, they don't, not any more. That was decades ago.
High-end and racing road bike tires now are in the 40-75 psi range, and they're much wider than they used to be. They finally figured out that skinny, high-pressure tires are not only very uncomfortable, they're slower too.