813 karma · joined October 11, 2023
There is also the Open LLM Leaderboard by HuggingFace (https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...) which aggregates a number of benchmarks, some (eg, MMLU) more trustworthy than others (eg, TruthfulQA). There are real concerns, however, that ML practitioners are gaming the leaderboard by contaminating their training data with evaluation data.
There are a number of other leaderboards such as OpenCompass (https://rank.opencompass.org.cn/leaderboard-llm-v2) and Yet Another LLM Leaderboard (https://huggingface.co/spaces/mlabonne/Yet_Another_LLM_Leade...) that I have seen suggested although I personally have found the most success with Chatbot Arena and the Open LLM Leaderboard.
I would also suggest checking out the LLM Explorer (https://llm.extractum.io/), which has all of these benchmarks and more in single location and allows you to sort and filter by a wide range of variables. That has been particularly helpful for me when trying to find models that will fit on my GPUs.
N.B. I am not affiliated with any of the benchmarks and services mentioned above.
> The controversial CEO took to his social media platform X to say that unless he is awarded another 12 percent of the company, which would give him ownership of 25 percent of Tesla, he would be "uncomfortable growing Tesla to be a leader in AI & robotics."
Is this serious?
Just FYI, you might want to fix autocorrect on iOS, your textbox seems to suppress it (at least for me).
Also worth noting that that case involved the same plaintiff as this one, Dr Stephen Thaler, who has been on a mission to have a court recognise his AI system, DABUS, as capable of inventing patents.
Originally, I intended to produce an LLM that could be used for answering questions related to Australian law, potentially via retrieval augmented generation. The LLM I ended up building, however, is unfortunately too small to be useful for that. It could, however, be useful for autocompletion. Perhaps if it was further finetuned, it might also be capable of tasks such as text summarisation and question answering. Nevertheless, the model is also valuable in that it has taught me how to train causal language models and taught me that, if I were to do the same thing with a bigger model and more of my legal database, I could achieve much greater results than major LLMs which lack a sufficent amount of Australian legal material in their training data.
> 2) Was training your own LLM necessary? How did GPT-4 compare?
GPT-4 is certainly much better given that I only trained the smallest version of GPT2. I do think that there is still value in training a larger LLM on my database as, in my experience, GPT-4 lacks a sufficently advanced understanding of Australian law, although it has the best understanding of all the models I have tried (GPT 3.5 Turbo, Claude 2, Bard, Llama 2, Mistral, etc...).
Last month, I had the honour of seeing my article on how I built the largest open database of Australian law (https://umarbutler.com/how-i-built-the-largest-open-database...) reach the front page. Due in large part to the outpouring of support and encouragement I received for my work from HN, I became determined to publish the first open LLM for Australian law by training a model on my database. I am excited to share that I finally achieved that goal today with the release of Open Australian Legal GPT2, a finetune of GPT2 trained on 37,560 laws and regulations, comprising 635,482,112 tokens, taken from my database.
Although it may not be as large as I had originally hoped, I'm still quite proud of the model. It was a struggle to wade through mountains of options trying to find something that worked. And now I have code I can reuse for training any other causal language model and dataset. The model is thus a small but important step towards maturing the legal AI field here in Australia.
If you’re interested in playing around with the model, you can find it here on Hugging Face: https://huggingface.co/umarbutler/open-australian-legal-gpt2
It's possible that they're in my database. I have included the as made version of all bills on the Federal Register of Legislation. However, if they haven't had a first reading yet, then probably not.
For processing PDFs, I recommend using `pdfplumber`, which is what I used to build the Corpus. Happy to discuss further if you'd like.
> One last question/comment, have you considered adding some additional reference info like the federal list of entities?
Do mean adding additional metadata? At the moment, I've kept the number of metadata attributes as low as possible. Every attribute added equates to more work to keep it standardised across all the jurisdictions and document types. My plan is to slowly add more attributes as I have time. I'd really like to associate a date with documents but even that is a hurdle. I have to decide what date should be the date of a document (is it the time it was issued, the time it was published, the time it came into force, the time the latest version was issued, etc... and what happens when a document doesn't have a date? should I extract it from its citation? how do I preserve time zone information? etc...).
One other point, and this is not specific to CanLII (I haven't checked whether this is the case) but I've seen that a lot of legislation databases have poor SEO. In fact, AustLII is usually ranked higher than governments' own websites when searching for laws. I think it's something important to get right because a lot of people just use Google/Bing/Kagi to search websites nowadays rather than using internal search engines.
Thanks for the heads up! I've fixed that now.
> For someone interested in using the data (and help out with bugs/issues), where would you suggest starting?
I think the best place to start is by downloading the Corpus (visit https://huggingface.co/datasets/umarbutler/open-australian-l... , and then click "Files and versions" and then "corpus.jsonl"). You can then use my Python library orjsonl to parse the dataset (you'd run, `corpus = orjsonl.load('corpus.jsonl')`). At that point, there's any number of applications you could use the dataset for. You could pretrain a model like BERT, ELECTRA, etc... and share it on HuggingFace. You could connect the dataset to GPT and do RAG over it. Etc...