Even with basic experiments I've done, it frequently introduces much more hallucination etc and allowing arbitrary steering...
Not to mention you just can't trust a model's judgement if the highest bidder chooses what it thinks
Even with basic experiments I've done, it frequently introduces much more hallucination etc and allowing arbitrary steering...
Not to mention you just can't trust a model's judgement if the highest bidder chooses what it thinks
But you already can't trust a model's judgement, and there's an entire industry around "GEO" or "AEO", which is basically poisoning training data so that AI mentions your products. The post above is the owner of the model taking a cut of that.
Yes, if you just hack steering into a model it's going to hurt performance, because doing so takes the model out of the regime it was trained for and validated in. But if that steering were to be accounted for (e.g. by rearchitecting the training process) there's no reason why it couldn't work. Diffusion-based image generation models, for example, 'by default' just generated random images out of the noise; steering (i.e. the user prompt) was added on as a secondary input, which models had to be re-trained in order to use.