It is an interesting failure case. The most advanced model (GPT-4) got it wrong, but several other models I tried were fine:

GPT-4-Turbo:

> A kilogram of feathers and a kilogram of steel weigh the same, as they both are one kilogram. The difference lies in their volume and density, not in their weight.

GPT-3.5-Turbo:

> A kilogram of feathers and a kilogram of feathers both weigh the same, as they both have a mass of one kilogram. The difference lies in the volume and density, with feathers being much less dense than most other materials, so you would need a much larger volume of feathers to make up a kilogram compared to a denser material like metal.

OpenChat Aura

> A kilogram of feathers and a kilogram of feathers both weigh the same, which is 1 kilogram. Therefore, the answer is both weigh the same.