It's hard to give really generic useful guidance, so I would suggest that first you should make sure to self-evaluate what kind of data you've got, how broad/comprehensive is it, what kind of query patterns you'll commonly use, and to be ready to profile those query patterns to see/understand/fix bottlenecks as and when they emerge. If you don't have enough data (and this is subjective and depends on the domain), it's probably not worth the effort to architect a feature repository system that only has 1000 customers and 10 features. OTOH, at small/tiny scale, you really don't have to worry much about performance or complexity, so maybe it's nice to try something simple out quickly, even if it doesn't provide much business value right away.

One milestone for success I think is, when your feature repository can be used to regularly gleam new insights that are surprising (even to your experts). So for example, imagine you are a typical retailer selling widgets, and you have 10000 customers who bought widget X, and you want to identify out of your remaining 10 million customers, which of those customers should you target as likely to want to also buy X? Now, of course, this is a classic problem and doesn't require a fancy feature repository; just the basic order information of who-bought-what is enough to create the classic collaborative filtering solution from.

But with a rich/useful feature repository, you can throw this problem in there, looking for the most similar other customers compared to the 10000 you know, and to rank which features are most powerful (in terms of, why it makes them similar), and you should see some interesting insights that you never would've guessed on your own.

Just to answer the "what really sucked" question a little bit, even if it's implementation-specific... man it really sucked to start out with 2-out-of-5 level of knowledge about Apache Spark, and to stumble through learning both the obvious lessons and the non-obvious/esoteric bugs/quirks, until after 1-2 years the team finally felt comfortable with like an 4-4.5/5 level of knowledge about Spark. This was also specific to the state of Spark back in ~2017 or so, and it's come a long way since then.