A lesson I learned very early on: don't use -ffast-math unless you know what it does. It has a very appealing name that suggests nice things. You probably won't ever need it.
You can/would just use it in the translation units where you want it; usually for numerical code where you want certain optimizations or behaviors and know that the tradeoffs are irrelevant.
It's mostly harmless for everday application math anyway, and so enabling it for your whole application isn't a catastrophe, but it's not what people who know what they're doing would usually do. It's usually used for a specific file or perhaps a specific support library.
My understanding is that if you don't specify -ffast-math when linking then you shouldn't get crtfastmath.o linked in.