Gatsby doesn't offer a syntax highlighter out of the box. It has markdown (and MDX) components that generate pages, and those in turn can have syntax highlighting plugins. AFAICT prismjs indeed is the predominant one used by Gatsby sites (at least, in mine). But thanks to SSR prismjs will typically run at compile time.
Hey, OP/author here, I do use prismJS under the hood, a flavor of it: prism-react-renderer, so it might indeed remove the traces of Prism.
Agree that it's a mess of div and spans, but it does the job.