A type-writer or a piano is an application specific piece of hardware, yet people can still do an amazing variety of things with them.

In the last few decades we got really lucky that general purpose CPUs kept getting faster and faster. So many of us could get away with ignoring application specific computer hardware.

Btw, as far as I can tell, TPUs are really just a variant of GPUs with an emphasis on lower precision?

Have a look also at the kind of special purpose hardware people make to mine crypto-currencies. I think the challenge for your teenage hacker is not so much to make them do interesting things, but to design them? (Assuming the tools and simulators become cheap enough?) FPGAs seem like they are cheap enough for a teenager?