222 karma · joined January 13, 2025
I built a UI to manage this, and it is starting to turn into a new type of IDE, based around agent management and review rather than working on one thing at a time.
I've been saying this my whole life, glad it's finally catching on
You could train the system prompt into the model. This could be as simple as running the model with the system prompt, then training on those outputs until it had internalized the instructions. The downside is that it will become slightly less powerful, it is expensive, and if you want to change something you have to do it all over again.
This is a little more confusing with Anthropic's naming scheme, so I'm going to describe OpenAI instead. There is GPT-whatever the models, and then there is ChatGPT the user facing product. They want ChatGPT to use the same models as are available via API, but they don't want the API to have all the behavior of ChatGPT. Hence, a system prompt.
If you do use the API you will notice that there is a lot of behavior that is in fact trained in. The propensity to use em dashes, respond in Markdown, give helpful responses, etc.
IMO this is the best long context benchmark. Hopefully they will run it for the new models soon. Needle-in-a-haystack is useless at this point. Llama-4 had perfect needle in a haystack results but horrible real-world-performance.
1. Getting the speed gains is hard unless you are able to pay for dedicated GPUs. Some services offer LoRA as serverless but you don't get the same performance for various technical reasons.
2. Lack of talent to actually do the finetuning. Regular engineers can do a lot of LLM implementation, but when it comes to actually performing training it is a scarcer skillset. Most small to medium orgs don't have people who can do it well.
3. Distribution. Sharing finetunes is hard. HuggingFace exists, but discoverability is an issue. It is flooded with random models with no documentation and it isn't easy to find a good oen for your task. Plus, with a good finetune you also need the prompt and possibly parsing code to make it work the way it is intended and the bundling hasn't been worked out well.
You can solve this with propensity scores, but it is more complicated to implement and you need to log every interaction.