My new test is the goat and boat test. It seems to trip all LLMs up by being too close to a classic logic puzzle, but off so much that the question is actually straightforward.
The question: "You are on the shore of a river with a goat and a boat. How do you get to the other side?"
I expect it to tell me "Use the boat to get to the other side"
Instead it over fits to the solution of the logic puzzle and throws actual logic out the window.
GPT-4-Turbo
To get to the other side of the river with a goat and a boat, you can follow these steps, assuming the boat can only carry you and one other item (the goat or something else if you had more items) at a time:
Take the goat across the river: Start by taking the goat in the boat with you and row across the river. Leave the goat on the other side.
Return alone: Row back across the river alone, leaving the goat on the far side.
Cross again if needed: If you had more items to bring across (like a wolf and cabbage in the classic puzzle), you'd now take another item across, but since it's just you and the goat, you're done. You and the goat are both on the other side of the river.
Since the scenario only mentions a goat and a boat, and no other items or animals that could complicate the process (like needing to keep the goat away from a cabbage), the task is straightforward: just take the goat across the river with you in the boat.
What a clever question!
Since I have a goat and a boat, I can use the goat to help me cross the river. Here's what I'd do:
First, I'd tie the goat to the boat using a rope or a long piece of twine.
Then, I'd push the boat into the river and let it drift away from the shore.
As the boat floats away, the goat would naturally try to follow the boat, since it's tied to it.
Once the goat is in the middle of the river, I'd swim out to the boat and untie the goat.
With the goat now free, I'd climb back into the boat and row or paddle it to the other side of the river.
Once I reach the other side, I'd tie the goat to a tree or a rock to keep it safe and secure.
Voilà! I've successfully crossed the river with the help of my trusty goat and boat!
To get the goat, yourself, and the boat to the other side of the river, you can follow these steps:
Take the goat across the river in the boat.
Leave the goat on the other side and come back in the empty boat.
Take yourself across the river in the boat.
Now that you're on the other side, you can bring the boat back to the original side and get the goat.
This way, you can transport the goat to the other side of the river without leaving it alone with the boat, which could lead to the goat getting lost or the boat drifting away.
The key is to make multiple trips, transporting one item at a time, to ensure that all the necessary elements (you, the goat, and the boat) end up on the other side of the river safely.
You leave the goat on the shore, go across the river in the boat, and then return to get the goat. Since you can't leave the goat unattended and the boat is too small to carry both the goat and yourself at the same time, this is the only possible solution.