I'm not aware of academic studies on the subject (I think that would be hard to do, because it's virtually impossible to know how many proprietary algorithms exist, how many are trivially breakable and how many have been broken).

However, this point is featured as 101 material in basically every cryptographic textbook. To put it very succinctly: there are conditions in which it can be beneficial to use proprietary cryptography, especially when you require very unique interoperability constraints. However it is almost never a benefit for the safety of the algorithm.

I've come across a proprietary algorithm and successfully broken it, in a black box setting, with differential cryptanalysis. This algorithm was deployed to disguise the sequential order numbers for a very large delivery company. It took me about a month, but it was done. The challenge in proprietary algorithms is shifted to figuring out what's going on because it's unrecognizable. That is a significantly easier challenge that identifying a vulnerability in an algorithm like AES, which has never had a meaningful vulnerability in a decade and a half of cryptanalysis.

If you use a proprietary algorithm it might be safer than a known unsafe open algorithm, but it's virtually guaranteed to be worse than widely studied algorithms, and most likely in a trivially breakable way. They can be safe, but that still means you're going to be working with professional cryptographers at a company like Riscure to assure it's safe.