If GPT-4 is really 2 trillion parameters, that implies it uses at least 32 A100-class GPUs (that's with 8-bit weights). Assuming they're each consuming at half their max TDP, that's around 6 kilowatts. It seems to spit out around 15-20 tokens per second, so sampling 1000 tokens probably consumes around 100 watt-hours of power. That's an entire high-end laptop battery, or enough power to move a Tesla half a mile.

OpenAI probably have some magic to batch requests under high load but still, pretty power hungry. A good amount of the $0.06/1k tokens they charge is probably just to cover electricity.