You can ask the same questions to ChatGPT and get the same or better answers.
I also know from personal experience with ChatGPT, that you can use it to:
- convert Python/boto3 to any language that has an AWS SDK
- convert CloudFormation to Terraform or the CDK
- write scripts that use the SDK
You will get the occasional hallucination.
I unfortunately don't think we'll be able to solve hallucination anytime soon. Maybe with the successor to the transformer architecture?
We’ve been testing LLM responses with a CLI, we’re using it to generate accuracy statistics, which is especially useful when the use-case Q/A is limited.
If ‘confidence’ can be returned to the user, then at least they can have an indication if there is a higher quality-risk with a given response.
ChatGPT: "I'm sorry, but as of my knowledge cut-off in September 2021, there was no service, tool, or product known as Amazon CodeCatalyst offered by Amazon Web Services (AWS). [...]"