You certainly didn't embed a >300KB font[1] in your few KB SVG image? If you don't embed the font, other people without the font will simply see something different.
The uncompressed 1.1MB export isn't optimized for size since it repeats basically the same CSS styles over and over for every token. When gzipped (default level) it's 302KB, and when brotli'ed it's 290KB, basically the font takes all the space.
Of course, technically it's possible to optimize the size by creating a custom font with only the glyphs needed, like what people used to do for icon fonts.