See https://arxiv.org/abs/2312.00638 for one proposal to address this