To be fair though, even with Svelte et al bundle sizes can balloon quickly when using UI toolkits and other libraries.
This is just how Node works, it's not a compiled language and you are not tree shaking all the components from libraries so this is bound to occur.
You can solve this by making your own libraries or just copying the individual source code for specific components into your project instead of including the whole package.
With sveltekit(and maybe nextjs aswell) you are only serving the necessary files to the client instead of including full library source code.
And typically if you're just using a single component it's one that is more complex than a button, like a popover or colour picker. Which then pulls in other components, etc.