GPU acceleration doesn't change the dynamics of caching glyphs. Existing text renderers already cache aggressively. Also, worth noting that caching the rendering of individual code points won't work effectively even in all cases for a terminal, because of diacritics, ligatures, etc.
Of course it's all pixels. You can do it all in pixel shaders. But of course, it's a lot more complicated than it seems. Supporting RTL requires some pretty advanced layout logic. Supporting OpenType ligatures also requires some pretty complicated, stateful logic. And you probably want to support "wide" glyphs even for a fixed width font, which are present in terminals where you are dealing with, for example, kanji.
If you want subpixel AA, that's another complicated issue. If you want to be able to do subpixel AA where glyphs are not locked to the pixel grid, you will need to do more work.
If you want to be able to render glyphs on the GPU purely, you'll need to upload all of the geometries in some usable form. Most GPUs don't render curves, so you will probably turn them into triangles or quads. That's a lot of work to do and memory to utilize for an entire font of glyphs.
You also might think you could utilize GPUs to perform anti-aliasing, but the quality will be pretty bad if done naively, as GPUs don't tend to take very many samples when downsampling.
Since a lot of the work is stateful and difficult to parallelize, doing it on CPU will probably be faster, that way you only pay the latency to jump to the GPU once.