I absolutely care about things that take hundreds of milliseconds for my builds. The faster I can build, the faster I can iteratively try new things things out. The goal should be to be able to see changes instantly after you make them, and Tailwind is usually just 1 part of a build pipeline.

Bret Victor's Inventing on Principle is probably the best demonstration of why this matters; in the first 10 minutes he shows a very concrete example of an insight that would not have occurred without instant feedback https://www.youtube.com/watch?v=EGqwXt90ZqA