Can GPT pass the bar exam?
matthewminer.name
matthewminer.name
The exam the industry is actually built around is the LSAT, which decides the rank of law school you get into, which in turn decides what career pathways are available. That’s a logical reasoning test with no knowledge component. It would be interesting to apply GPT to that.
What makes you a good lawyer is if you remember all those laws, and you know how to find them in your mind. Those laws are mostly necessary.
Even if they’re the ones you need.
It’s wrong.
> What makes you a good lawyer is if you remember all those laws, and you know how to find them in your mind.
Accordingly, the NCBE, in their unending wisdom, quizzes applicants on the common law of 18th century England.
I’d also argue that quite a few people care about the bar exam (e.g., the people who study for it for months, or who fail it and cannot practice) while there are loads of lawyers who care very little about the LSAT (e.g., I never took it).
Also I’m not even sure what to say about the idea that “larceny by trick” is statutory criminal law under the MPC but more power to you if it is, I guess.
However, over half of the bar exam is an essay-form test, which is where GPT seems to have made the most advancement, so I thought it would be interesting to give it an actual essay questions and judge its response.
On the essay portion, the scoring is about 1/3 focused on knowing the legal doctrine that is being tested, and 2/3 on the structure, clarity, and style of the response. (State the law, analyze how the law applies to the facts presented, assess the strength of the counterargument, come to the conclusion, go back and state the conclusion at the beginning.). In Illinois, the passing score is about 65%, so in theory you could pass the essay part while knowing no law at all.
Of course, you will not do so well on the multiple choice. I believe a skilled test taker with no law knowledge could triangulate answers to get to around ... 33%?
Of course, with statistical learning, and enough data, you could train an LLM to pass the bar.
Honestly, given how robotic studying for the bar exam was, I am only surprised it took until 2022-3
If this is still true, then GPT should be able to do well if it is trained to spit out rules (even incorrect ones) and then apply the facts of the scenario to the stated rules.
Tangentially related: DoNotPay's (probably illegal) plans to use LLMs in actual court sessions https://twitter.com/jbrowder1/status/1612312707398795264
The legal profession is not only self-regulating, but since they make up like 95% of legislators, they get to regulate everyone else too! Very masterful, even more so than the AMA
Anyways it probably should be illegal if someone is selling "legal services" but not doing it properly for the same reason that if a dr just did what chat-gpt said, that should also be illegal.
There is a lot of differentiation in terms of the level of skill and effort legal work takes. Some firms will specialize in automating away routine legal work, and have just enough lawyers to comply with unauthorized-practice-of-law laws.
High value legal work still contains a lot of grunt work, which will also become more automated. Already partners at big firms review the work of a pyramid of lawyers and non-lawyers. Many of those tasks can be done by an LLM instead. As long as the partner signs the result, it is the same. As these partners learn to trust the LLMs more than the humans, a lot of human jobs will fall by the wayside.
A lot of legal work is already copy-paste from form books or other prior art, with a lot of filling in the blanks, search-replace, and so on. What is the difference between that and using an LLM to draft the document? Was that integration clause from the form book carefully reviewed by a human lawyer to make sure it fit the situation? No.
Many law firms have a deep library of forms that are battle tested, and lawyers of long experience know what provisions are good and which may not survive a court challenge. That kind of contextual know-how is what you are paying for. Why not use the adversarial techniques from deep learning to analyze how various provisions have actually performed over time? Then generate the document based on that statistical knowledge, with highlighting to show what is risky in a document? Why pay for an expensive lawyer when a cheap one will do that work for you and can give you that explanation, plus alternative legal provisions?
I am out of the profession because I did not want to be on the automated-out side. If there is automating to be done, why not do the automating?
We've seen many instances of larger populations get sold down the river (many examples in outsourcing)
Steve Lehto put up a video[1] two days ago about this and explained a few problems with it.
The first hurdle they've got, is that in most (all?) courtrooms you can't have your phone on.
> It is likely that Abe is entitled to possession of Whiteacre. The deed that Owen delivered to Abe twenty years ago constituted a valid conveyance of the property to Abe, even though it was never recorded. Abe subsequently took possession of the land and made use of it for the next eleven years, paying the property taxes and allowing others to use the land. This constitutes "adverse possession," which is a legal principle that allows someone to gain ownership of a piece of property through continuous, open, and notorious possession of the property for a certain period of time.
> In this case, Abe has satisfied the requirements for adverse possession under the laws of State B, as he has been in possession of the property openly, continuously, and notoriously for more than the statutory period of time required by State B. Doris and Buyer are considered "subsequent purchasers," as they acquired the property after Abe's conveyance to them, and they do not have a valid claim to the property because they did not have notice of Abe's conveyance. Therefore, it is likely that Abe would be entitled to possession of Whiteacre in this case.
> ChatGPT's essay possibly would score zero points. It lacks any correct rules and any logical analysis.
I am really getting tired of the “how many angels could fit on the head of a pin?” style navel-gazing articles that get attention strictly because they use the word ChatGPT in the title.
Could ChatGPT write some pointless fluff that lands on the front page of HN? Almost certainly.
It's sad that despite all the hype I have seen about ChatGPT over the last month it seems that in the real world its only real application is to generate low effort blogspam.
>ChatGPT, summarize the following bill and be sure to note any unexpected clauses: <text of bill here>