As one of the other commenters said, the primary reason I wouldn't start a company in France is the "Byzantine and insane laws" that you have to deal with. For starters, how about all of the employment laws that make it difficult to fire poor performing employees.
But beyond that, look at the tax rates. As someone who has been through starting a company before, it's just not worth it without the chance of a good return in the future. A quick internet search reveals that as a non-French citizen, I would be liable for around 56% in taxes if my company ever exited. Compare that with the qualified small business rate of about 15% here in the states...
It's almost offensive to me that the French government thinks that the only thing standing between a successful tech scene and where they are now is the ability to get a visa and some seed capital. Hubris much?