Crossplane maintainer here -- this has come up a number of times in the community, but ultimately we would prefer for the system to be able to handle an arbitrary number of CRDs, whether they are being used or not (as evidenced by the work outlined in this post). Moving to filtering CRDs that a provider installs can lead to a confusing experience where the presence of a given package does not mean the same thing across clusters, which also complicates the dependency model Crossplane packages implement. We view Composition + RBAC as the mechanisms to define what a given entity can access (i.e. the API line).