LTO is heavily used in my experience. If it breaks something that is indicative of other issues that need to be addressed.
Compilation happens once and then runs on hundreds of thousands up to billions of devices. Respect your users.
I would recommend only doing things that fit within the 'build > text > fix' loop.