Really depends. How big is the once CSS file you're loading and how much of the CSS is common from page to page? If 20kb is for a specific component on one or two, less-visited pages, then it probably doesn't make sense to lump everything together.
A fairly safe default starting point would be:
1. Tiny subset of inlined critical CSS (See https://www.smashingmagazine.com/2015/08/understanding-criti...) 2. Asynchronous + rel=preload loading of the main CSS of your site 3. Then, if there are hefty chunks of CSS for specific pages only, bring those in on those pages.
This changes a bit if you're using HTTP/2. Then the multiplexing makes it beneficial to break the CSS into a few smaller files instead of one big one.
In the end, you probably just have to experiment and test. :) No guaranteed "this is best" approach here.