Yes we can, these things don't have to be bundled together.
However, I haven't pursued unbundling them so much:
1. A big use-case is right now cache.nixos.org is too big and the old sponsor for hosting dropped.
2. CA derivations have the potential to greatly reduce new store object churn because they normalize derivation outputs more strongly (two different derivations producing the same result now results in the same store object, not a different one.)
If someone were to say "no, we needs IPFS right away, please help" I would do that, but no one said so yet.
It's good question :)