It is not only about how long it takes to download, it is also about parsing that amount of code in low end phones.
Still I do think if you are building a web application (and not a static site or a blog) it is expected it can take a second or two to load it for the first time. 500 KB bundle is nothing wrong in that case.
It's also expected that my train is going to be late, but we can lament this condition, instead of resigning ourselves to it.
Ignoring the impact of libraries you send to the frontend will lead to death by a thousand cuts.
I would hate to use something built by people with the attitude that compromising on performance when you don't need to is OK because it falls in line with current expected ranges.