- tex takes 0.15 seconds
- pdftex takes 0.37 seconds
- luatex (1.0.4) takes 0.46 seconds
All of these are reasonably fast, and I think getting a full book in less than half a second is acceptable. (My browser takes longer than that to render most things.)
What's the catch? This book is written in plain TeX. The slowness comes when you use LaTeX: it's not unusual to have even single-page documents that take longer for LaTeX to generate. The whole LaTeX approach is to have a package for everything, load everything you may want, providing all the features, etc — and all of this implemented in TeX macro expansion in painful and hard-to-refactor ways.
This is something LuaTeX will help with actually: LuaTeX makes TeX less monolithic by providing hooks (callbacks), and many things can now be implemented in easier-to-read Lua code as hooks into the right parts of TeX's execution, rather than setting everything up at the start so that after macros are expanded the right thing happens. If there's less use of LaTeX and elaborate macros, things should get faster.