Windows 95 rendered its UI on CPU and it was fast. You need GPU for big bitmaps and useless effects. For square with border and text CPU is more than enough. Then you send this small square bitmap to GPU to add shadows and whatnot to it.
Indeed correct unicode text rendering is one of the difficult problems. It's the same thing that is missing from basically all games. So when people say stuff like games can run at 120fps, how hard can UI be? it's basically the same Windows 95 argument. The UI work being done is a small subset.