Even if this was faster than esbuild by whatever milliseconds, I would still probably go with esbuild for the api. esbuild is a delightfully raw, scriptable thing, that is as simple or complicated as you want, you just make the node scripts and fun the functions.
Config with swc looks to more closely resemble babel/webpack with extensive config objects, which is fine, but it can only mean that it loses out to esbuild for me.
But I am not a professional or anything like that, so very much YMMV!