There's a company called Briel Computers that sells replicas of the Apple I, Altair 8800 and more.

The Maker Shed sells a 4-bit microcomputer trainer from Japan. It has a hexadecimal keypad, 1 7-segment LED display, some other lights and a speaker.

You could also go to eBay or a swap meet and buy an old microcomputer.

http://www.brielcomputers.com/

http://www.makershed.com/ProductDetails.asp?ProductCode=MKGK...

I don't actually but the 4-bit micro mentioned below sounds cool.

I am currently working on an atmega project and have hopes of bringing such a "self-hosted" programming environment to the device, which I'll post to HN when I get it working.

Probably the best way to get something like that would be to build it yourself. It may not be the kind of project you're looking for though.