Well just bear in mind that this was about 4 years ago (we were fairly early adopters apparently), and I understand that my comments here are probably no longer fair, and probably don't apply to KC at it is today.
But since you asked, basically Keycloak was by far the single largest component of our stack, which didn't sit well with me because by any measure, our application was far more complex than Keycloak.
At the time we were transitioning away from JEE to JSE microservices, and Keycloak was a full-blown JEE application with all the heaviness that we were successfully leaving behind; it took over a minute to start, while our other Java services would launch in less than a second. Of course this didn't impact production at all - but most engineering time is spent in development and testing, where this was a real problem, especially if we were experimenting with and learning about new features.
All of this contributed to my perception of heaviness.