That's a great use-case! But what about when GPT hallucinates - gets it wrong. What happens when you have sent a (potentially legally-binding) quote to your customers, and you have to back out of it?

Humans make mistakes - whether directly, or indirectly through the software that we craft. But when the mistake happens, we can reason about it, understand why it happened, and fix the problem.

Wouldn't it be embarrassing to not be in a position to reason about / explain a mistake to your customers caused by this "thing" we use that's been trained on all the random information available on the internet? Would that not reduce the value and standing of your business (which is ultimately about the people and expertise contained within) in the eyes of customers?

As an engineer, technologist, and CTO I absolutely love what I've been able to craft using generative AI under direct supervision.

But I cannot imagine a world where we give our carefully thought-through, easily changeable, algorithms and processes for a black box that cannot be reasoned about, no matter how good the output _sometimes_ is.

I have to think many people feel this way.