Oracle gives you a $300 free trial, which equates to running BM.GPU4.8 for over 10 hours - enough for a focused day of prompting
Not as fast as a GPU, but less than 5 seconds for a 250 token response is good enough for a Discord bot.
I'm not. The only way to know it is to try :) thank you for the link!
I might be suffering from FOMO to some degree, I've just got to tell myself that this won't have been the only time model weights get leaked!
This certainly sounds a lot like whining that others aren’t doing the work you yourself don’t want to do.
I'm not in a position to put in any meaningful work towards optimising this model for lower-end hardware, or working on the tooling/documentation/user experience.