Originally, I intended to produce an LLM that could be used for answering questions related to Australian law, potentially via retrieval augmented generation. The LLM I ended up building, however, is unfortunately too small to be useful for that. It could, however, be useful for autocompletion. Perhaps if it was further finetuned, it might also be capable of tasks such as text summarisation and question answering. Nevertheless, the model is also valuable in that it has taught me how to train causal language models and taught me that, if I were to do the same thing with a bigger model and more of my legal database, I could achieve much greater results than major LLMs which lack a sufficent amount of Australian legal material in their training data.
> 2) Was training your own LLM necessary? How did GPT-4 compare?
GPT-4 is certainly much better given that I only trained the smallest version of GPT2. I do think that there is still value in training a larger LLM on my database as, in my experience, GPT-4 lacks a sufficently advanced understanding of Australian law, although it has the best understanding of all the models I have tried (GPT 3.5 Turbo, Claude 2, Bard, Llama 2, Mistral, etc...).