Speaking of LoRA, what happened with ZipLoRA? It's supposed to be a better way of merging multiple LoRAs, and the results look good in their examples. Is it being used anywhere?
Not related to the article but tangentially relevant, would it be possible to train a LoRA or DoRA with a high rank, and then use SVD to see if the rank is too high and truncate to a better value of r? Maybe use different ranks for different layers after some training?
I haven't tried what you were suggesting, but that sounds actually plausible. Interesting idea!