Of course it would hallucinate. It would just pick arbitrary/wrong values.