Coming under the French nuclear umbrella explicitly means that if a country is under extreme threat of nuclear attack, France will 100% retaliate. Unlike the current scenario where even if the country is an EU ally, France can still choose to refuse.
Later down the line, France could even choose to delegate nuclear hosting to those countries, like the US does with Turkey and the like. Obviously those nations don't get access to the codes.
At the end of the day, everything works on the basis of trust, and while the US has broken its covenant, the EU countries have not, especially not with each other.