The hardest part of building a design system is getting the organizational willpower to commit to it. There will always be another high priority feature to build, so to get it done you generally need executive level sponsorship.

Design systems are valuable but ultimately best tackled as an accelerant (/luxury) investment at scale. Your customers are unlikely to buy your product or churn from it due to not having a design system in place, so you really need to look at overall design/dev productivity as the benchmark of whether the investment is worthwhile.