Sure it depends how it is done but for most uses I'd say they are not appropriate - building tools with them is ok if you double check (though how many people will when the answers seem good enough at first?).

I'd find it really troubling if financial analysts are using them without knowing the deep limitations of the tooling (which the companies selling them will not highlight for you). They don't actually count or reason so they are liable to just make up figures based on their training dataset, not the data you give them.

Using them for actual financial analysis and generating reports based on data will lead to hallucinated figures which conform to what was asked for, not what the data says and silently fills in gaps in the data. It's extremely dangerous and not something they are good at at all.