For that reason, the usual trick is to use asymmetric encryption to protect a symmetric key (a fresh one per payload) that is then used for the actual payload. That way you get the speed of symmetric encryption with the benefits of asymmetric ones.
(Though evidently that's not what the PS5 uses here, otherwise it would be useless to read out one symmetric key).