> why don't they start their companies abroad from the start?
Because it's easier to move a company to the US once it is funded and starts getting traction than to start it there.
Also, technical talent is much cheaper in France than in the US, so building the product in France makes sense. A lot of French tech companies that "move to the US" actually keep a subsidiary with their development team in France. And we see tech companies such as Facebook and Google open offices in France to benefit from the talent pool.
Young develpers are also probably more willing to try working at a startup instead of going for the safety of a big company because they do not have student loans to pay back...
France has been good with seed-stage startups for years now (incubators, accelerators, seed VC / angels, tax cuts & subsidies...) The problem is the growth stage and beyond. The French market is too large to force companies to think global from the start and too small to scale for most. VC is much harder to find beyond series A, and moves like the French government preventing exits to foreign buyers scares foreign capital.
But I am optimistic: France figured out the early stage just a few years ago, it will figure out later stages in time.