I am seeing a lot of papers on arXiv where people do "prompt engineering" experiments, that is they set up maybe 1000 or 2000 sample problems, run them through the LLM, and see how it performs.
The results are usually sobering, frequently you find that the models get things right 70-80% of the time which is acceptable for a "chatbot" that is ultimately animated by the end user, but put it in a process where there are 10 steps that are 80% correct and you are talking about a process that works 10%. You can ask an LLM to "open the pod bay doors" but not expect it to fly the spacecraft to Jupiter.
The difference between organizations that apply ML successfully and those that don't comes down to two things: (i) luck, (ii) picking well-defined problems and developing appropriate training data. Luck comes in because once in a while you get handed an easy problem. If you don't have that you have to do the very difficult work of convincing management to invest in making a training set of 1000s if not 10,000s of examples. People are about as happy to see the ontologist as they are to see the oncologist because the ontologist makes them think rigorously and you will certainly have phone calls where you've figured it all out and then they call you back and they want to mess it up again and spend another few years pushing bubbles around under the rug. You've got to push back with tough love. People think the "new AI" eliminates the need for ontology but no, training is a "garbage in, garbage out" situation if your problem is not well defined you'll never get past a certain point in accuracy.
If you can master the black art of whipping real life problems into shape and the herding cats problems that get involved, that's a lot more valuable than being another statistic in the Kaggle leaderboard.
What could be a bootcamp like curriculum for someone who wants to learn only how to train a LLM (upto a sellable standard)?
Because it is such a new thing I don’t think there is any curriculum available, what you really can do is “learning through doing” and also reading papers both to get some idea of what people are doing (I like the papers where run of the mill researchers solve run of the mill problems because that gives me an idea of what to expect with my run of the mill problem) and also to pick out results to try to replicate.
My best advice right now is that it is important to scale down to something that lets you do a lot of experiments. For instance I have a classifier that takes about 30 seconds to train and I can rapidly iterate on it because I can run it 1000s of times a day. I have another that takes 30 minutes and I am loathe to put effort into it because it so easily becomes a tarpit. Thus start with something that gets meaningful results in a minimal time, get really comfortable with it, then scale it up deliberately until it is large enough. Feel free to send me an email if you want to talk more.
If I was to interview someone for Prompt Engineering at least the candidate should be able to look for information in the web with the least effort, that is you should be able to collect important information with the minimum number of queries.