You need thousands of highly skilled engineers to operate an advanced fab. You need extremely advanced and expensive specialized equipment - look up ASML for a general idea of what this equipment does. You need a huge, purpose-built cleanroom that is many orders of magnitude “cleaner” than an operating room. You need a structure (and location) that has virtually zero vibration or noise. You need an constant, high-volume supply of a bunch of resources including water, power, and sand. And the list goes on.
Edit: Good video on ASML and semiconductor fabrication equipment in general: https://www.youtube.com/watch?v=iSVHp6CAyQ8
Even simple things like "water" is not just water but instead you need ultrapure water. The machines that purify water to the purity semiconductor industry needs it are not cheap.
* https://www.youtube.com/watch?v=C-52AI_ojyQ
* https://en.wikipedia.org/wiki/Long_and_short_scales
(Given a choice, I would go with the long system, but sadly that doesn't seem to be the timeline we live in.)
Ultra high precision set up of the equipment.
Storing, using and disposing of some very toxic chemicals.
Convincing ultra specialist, high demand/skill/income engineers to relocate to place X.