You're seeing this because the model isn't instruction fine-tuned. You'll need prompting similar to the original GPT3 or Llama models.
Plugging
> Nancy has two apples and Becky has one apple. Becky gives 1 apple to Nancy. Becky now has
into GPT-2 via HuggingFaces at https://huggingface.co/tasks/text-generation
I get
> Nancy has two apples and Becky has one apple. Becky gives 1 apple to Nancy. Becky now has three apples and Nancy has one apple. Becky now has three apples and Nancy has one apple.
> Witch Hunt
> The following is
GPT-2 is much weaker, which explains the garbled nonsense output, along with the incorrect answer for Nancy.
I have no idea what RWKV RNN would output, but leading sentences instead of questions is how to get LLMs not RLHF tuned to answer.