Yes, but for most things it's not significant.

For example Stack Overflow used to handle all their traffic from 9 on-prem servers (not sure if this is still the case). Millions of daily users. Power consumption and hardware cost is completely insignificant in this case.

LLM inference pricing is mostly driven by power consumption and hardware cost (which also takes a lot of power/heat to manufacture).

An "AI" box with a few high end gpu/npu cards takes more energy in a 4u box than an entire rack of commodity hardware takes. It's not nearly comparible... meaning entirely new and expansive infrastructure costs to support the high energy. That also doesn't count the needs for really high bandwidth networking to these systems. Not to mention the insanely more expensive hardware costs.

The infrastructure and hardware costs are seriously more costly than typical internet apps and storage.