It's...interesting but I feel like people keep forgetting that LLMs like Claude don't really...think(?). Or learn. Or know what 'corn' or a 'tractor' is. They don't really have any memory of past experiences or a working memory of some current state.

They're (very impressive) next word predictors. If you ask it 'is it time to order more seeds?' and the internet is full of someone answering 'no' - that's the answer it will provide. It can't actually understand how many there currently are, the season, how much land, etc, and do the math itself to determine whether it's actually needed or not.

You can babysit it and engineer the prompts to be as leading as possible to the answer you want it to give - but that's about it.