Here is an idea of what you can do with a 84's Zilog z80 (warning: long animated gif): http://benryves.com/images/nostromo/bsp/2010.11.28/Walkthrou...
(corresponding blog entry: http://benryves.com/journal/3739423)
You can get a lot more out of an M68k. (Actually, to be fair, the M68k calculators' larger screen size works against them, but if you are willing to sacrifice graphics you can easily surpass that demo.)
Not bad but it is not a 3D Physics engine. Just a renderer. Not the same thing. Still pretty impressive.
(It is not in fact even a 3d engine, but rather a vector BSP 2.5D one that is technically similar to Doom. To the extent with Doom has physics (pretty much only momentum (seen prominently in how explosion knock back effects entities) and gravity, so not really) that demo kind of has physics. Actually I don't think he actually does momentum, and his gravity is primitive. He's pushing the calculator pretty hard, I'm not sure how much more you could add, but the 89 can do much better.)
Basically I think you could get something close to this on an 89 (with dramatically worse graphics/graphical abilities): https://github.com/id-Software/Quake/blob/master/QW/client/p...
Unfortunately I never had a TI-89. Would've been awesome if I had one back in the day, and then test these games: http://www.ticalc.org/archives/files/fileinfo/405/40593.html and http://www.ticalc.org/archives/files/fileinfo/323/32318.html
In hindsight, I wish I had spent that time learning Z80 instead...
The controls are pretty clever too.