My experience is that pretty frequently the LLM just refuses to actually supply json conforming to the model and summarizes the input instead. Even with several retries configured I still couldn't rely on it. I only spent an afternoon with it though so it's possible I'm just doing it wrong (either in how I'm prompting or in how I'm configuring pydantic-ai).