It's not just scale. Even Intel's development fabs cost billions to build.

The equipment and facilities required for cutting edge fabs are very expensive.

Rockets turned out to be expensive not because the technology demanded it, but because the companies were intentionally inefficient, e.g. it was actually fine to build a rocket in a tent with simple tools than to setup a massive clean room facility. This was fine because the government would hand them a blank check with a guaranteed profit percentage, and the handful of commercial customers would just swallow the cost.

However, it isn't as clear if the same applies to semiconductors, since even a bit of dust is indeed a serious problem for producing a chip where the transistors are much smaller than most dust. Until very recently there were no blank checks, every improvement required mostly private investment, so it's less likely that the inefficiencies are intentional.

The big difference there is SpaceX was able to incrementally build up. Falcon 1, Falcon 9, Falcon heavy, Starship. Each step allowed them to prove that it worked and gain investors and government contracts.

Who in their right mind would invest $10b just to build the manufacturing capability without having a well established product? And if you have a well established product are you still a startup?

The equipment to manufacture rockets costs many orders of magnitude less than equipment for a FAB. It also does not get obsolete quickly.

Musk should have spent 44B on a chip company.