The problem I think is that there are 25 cryptocurrencies bigger than it. Particularly with its form of mining, it's trivially easy for say a big player in the 10th largest currency to shift their mining power to a smaller one like Bitcoin Gold, overpowering everything else.

Normally the non-51% attack argument is that anyone who invests enough in 51% of the infrastructure and has sufficient coins to profit from double-spending, is very unlikely to do so because it would render the coins and mining equipment worthless or at least worth less than the investment had cost.

That'd be true for bitcoin, but not for a GPU-mined 26th largest cryptocurrency. You can completely destroy it, cash out and use your equipment elsewhere on coins in which people still have faith.