It's partially related to the work needed to obtain them. There's very little edible meat in each one and the processes to obtain that seem to be largely manual.

Diamonds are treated as a luxury despite being "common."