The energy densities of modern batteries are a couple factors lower than that of gasoline, and much lower than TNT. Gasoline is incredibly energy-dense, but its energy density is comparable to butter. Do you worry about sticks of butter exploding on you? No, you don't. Gasoline is dangerous because it really likes to evaporate and the mixture of gasoline fumes and air ignites at the slightest provocation. Batteries are dangerous because they can go into thermal runaway from an internal short if the barrier between the electrodes fails. The energy density is not what matters here.

A lithium ion battery has about 2-3% as much energy per gram as a chocolate bar. The energy density is not what causes safety issues.