ubutler

So AustLII and Jade are what I was referring to when I say "While there were certainly a few free-to-access legal databases, none were truly open ...". They are free but not open in the sense of being licensed under an open source licence and being free to download in a raw format. Whereas everything in my corpus except decisions from the Federal Court of Australia is licensed under CC BY 4.0. And even decisions from the FCA are under a licence that permits both non-commerical and commerical usage.

In terms of completeness, however, AustLII and Jade win out. They seem to have almost everything if not everything. Their data is also much richer than mine. I must give props to AustLII for how they're able to hyperlink terms defined within legislation to their definition. I think they're an invaluable resource for members of the public.

The audience of my database was more so those who want to play around with raw legal data and want to feel secure that they are not breaching any laws in the process. The fact that it is stored in plain text is also beneficial for anyone trying to build ML models that only accept raw text.

Absolutely. Luckily, the websites I scrape don’t change very often (ie, every 5-10 years based on Wayback Machine), and also a couple of them all use the same underlying legislation management system and APIs.

Ultimately, however, I’m hoping that in the long term, the Australian Government will see the use in this project and decide to maintain it.

There is! It’s called the [Pile of Law](https://huggingface.co/datasets/pile-of-law/pile-of-law), and it’s actually what inspired this project. The only caveat is that it doesn’t appear to be regularly updated, and so is more of a snapshot of US law rather than a semi-live copy. Also I’m not entirely sure how comprehensive it is (ie, whether there’s anything missing). For those interested in building a true US equivalent of my corpus, I think it could be a great starting point, particularly since they published the code they used to build it.

> Fantastic work here! I was griping to my team just last night how painful developing a chunking strategy for Australian Legislation is that while there's (generally) layout consistency within a piece of legislation, that's not true across pieces of legislation... so I can imagine the pain of trying to collate legislation across jurisdictions.

Absolutely, there's a lack of consistency even within the same jurisdiction and document type. It only gets worse once you want to add multiple jurisdictions and different types of documents. My best strategy so far has been to use recursive chunking where you begin chunking at the largest section of newlines. Ideally though you want some form of semantic chunking where you already know what parts of the document represent Parts, Divisions, Schedules, Sections, Sub-sections, etc...

> I've reach out via your LinkedIn profile - would be great if there was an opportunity to collaborate.

Great! Always happy to connect.

Thanks for the suggestions! Distributing the Corpus widely will be my next focus.

As I point out in the introduction, there are a few free-to-access legal databases available in Australia, but none are truly open in the sense of being free from copyright restrictions. Neither AustLII nor Jade are licensed under an open source copyright licence such as CC BY 4.0 (which is what the majority of my Corpus is under).

Github might be difficult as they impose constraints on the size of repositories and the Corpus is around 5GB. The Internet Archive is a good idea, however, I’ll have a look into that. I’ve also been thinking about sticking it on Kaggle as well to increase its reach.

Hey HN, Over the past year, I’ve been working on building the Open Australian Legal Corpus, the largest open database of Australian law. I started this project when I realised there were no open databases of Australian law I could use to train an LLM on.

In this article, I run through the entire process of how I built my database, from months-long negotiations with governments to reverse engineering ancient web technologies to hacking together a multitude of different solutions for extracting text from documents.

My hope is that the next time someone like me is interested in training an LLM to solve legal problems, they won't have to go down a year-long journey of trying to find the right data!

You can find my database on HuggingFace (https://huggingface.co/datasets/umarbutler/open-australian-l...) and the code used to create it on GitHub (https://github.com/umarbutler/open-australian-legal-corpus-c...).

Does this support iOS WebKit? All I see is Tux sitting on a globe.