California issues fine over lawyer's ChatGPT fabrications
calmatters.org
calmatters.org
Wow. Seems like he really took the lesson to heart. We're so helpless in the face of LLM technology that "having some victims, having some damages" (rather than reading what you submit to the court) is the inevitable price of progress in the legal profession?
21 of 23 citations are fake, and so is whatever reasoning they purport to support, and that's casually "adding some citations"? I sometimes use tools that do things I don't expect, but usually I'd like to think I notice when I check their work... if there were 2 citations when I started, and 23 when I finished, I'd like to think I'd notice.
I disagree. It worked until now, and using AI is clearly doing more harm than good, especially in situations where you hire an expert to help you.
Remember, a lawyer is someone who actually has passed a bar exam, and with that there is an understanding that whatever they sign, they validate as correct. The fact that they used AI here actually isn't the worst. The fact that they blindly signed it afterwards is a sign that they are unfit to be a lawyer.
We can make the argument that this might be pushed from upper management, but remember, the license is personal. So it's not that they can hide behind such a mandate.
It's the same discussions I'm having with colleagues about using AI to generate code, or to review code. At a certain moment there is pressure to go faster, and stuff gets committed without a human touching it.
Until that software ends up on your glucose pump, or the system used to radiate your brain tumor.
This is known as shepardizing.
This is done automatically without AI and has been for decades.
In my experience they're a boon to the other side.
Using AI to help prepare your case for presentation to a lawyer is smart. Using it to actually interact with an adversary's lawyer is very, very dumb. I've seen folks take what should have been a slam-dunk case and turn it into something I recommended a company fight because they were clearly using an AI to write letters, the letters contained categorically false representations, and those lies essentially tanked their credibility in the eyes of the, in one case, arbitrator, in another, the court. (In the first case, they'd have been better off representing themselves.)
As for the last sentence, those systems already exist and roughly all sane lawyers use them. They are required to. You aren't allowed to cite overturned cases or bad law to courts, and haven't been allowed for eons. This was true even before the process was automated complety. But now completely automated systems have been around for decades, and one is so popular it caused creation of the word "shepardize" to be used for the task. So this is a double fault on the lawyers part. These systems are integrated well too. Even back in 2006 when I was in law school the system I used published an extension for Microsoft Word that would automatically verify every quote and cite, make sure they were good law and also reformat them into the proper style (there were two major citation styles back then). It has only improved since then. The last sentence is simply a solved problem. The lawyer just didn't do it because they were lazy and committed malpractice.
Yes, this is the crux of it. More than any other thing you pay a lawyer to get the details right.
How do you know this? Wouldn't we expect the benefits of AI in the legal industry to be way less likely to make the front page of HN?
My guess is that he probably doesn't believe that, but that he's smart enough to try to spin it that way.
Since his career should be taking at least a small hit right now, not only for getting caught using ChatGPT, but also for submitting blatant fabrications to the court.
The court and professional groups will be understanding, and want to help him and others improve, but some clients/employers will be less understanding.
Sure. Its also unrealistic to expect nobody to murder anyone. That's why we invented jail.
This was from the model available in June 2023
I've taken this hallucination issue to heart since the first time this headline occurred, but if you just started with leading LLM's just today, you wouldn't have this issue. I'd say it would be down to like 1 out of 23 at this point.
Definitely keep verifying especially because the models available to you keep changing if you use cloud services, but this September 2025 is not June 2023 anymore and the conversation needs to be much more nuanced.
People have more car crashes in areas they know well because they stop paying attention. The same principle applies here.
This is actually one of the more infuriating things about all of this. Non-lawyers read this stuff and they’re like oh look it hallucinated some cases and citations. It actually should still have been caught 100% of the time and anyone submitting briefs without verifying their cites is not fit to be a lawyer. It's malpractice, AI or not.
Same with FSD at Tesla, there's many people who think that accidents and fatalities are "worth it" to get to the goal. And who cares if you, personally, disagree? They're comfortable that the risk to you of being hit by a Tesla that failed to react to you is an acceptable price of "the mission"/goal.
There is definitely benefit to using language models correctly in law, but they are different than most users in that their professional reputation is at stake wrt the output being created and the risk of adoption is always going to be greater for them.
It's not a large step after that to verify that a quote actually exists in the cited document, though I can see how perhaps that was not something that was necessary up to this point.
I have to think the window on this being even slightly viable is going to close quickly. When you ship something to a judge and their copy ends up festooned with "NO REFERENT" symbols it's not going to go well for you.
Why would I pay for software what I could do with my own eyes in 2 minutes?
> Two minutes per filing full of citations is unrealistically fast and also adds up
Also adds up to what? The amount of time it takes to do your job? Vs not doing it and just letting AI rip? Oh I see, people here think that paying another service to fix the poor job the "do your job for you" AI service did makes sense? I don't think so. But then again I'm not the one wondering if lawyers keep cases stored in "relevant databases" out of what could only be sheer ignorance (I bet AI could get that one right). Never heard of Westlaw or LexisNexis?
Then that's the actual answer to the question they were asking. It's already done before you're given the document.
But I don't know what takes "2 minutes" then? Checking to make sure they actually have hyperlinks?
> But then again I'm not the one wondering if lawyers keep cases stored in "relevant databases" out of what could only be sheer ignorance
That is not what they asked. You only make yourself look bad when you change the question before mocking it.
> otherwise doesn't have a use case unless you're using AI in the first place
Their full post was describing something that has a use case if someone else is using AI.
> The amount of time it takes to do your job? Vs not doing it and just letting AI rip?
No, see previous.
It's not the actual answer. Having a standardized resource location scheme is a solved problem in the legal field for years beyond my knowledge. Well before computers, that's for sure. Getting all the resources cited in any legal brief is probably one of the most trivial task an attorney does, if not a paralegal, and I'm sure there are scripts and apps for it too. You just type the cite in and poof! Out comes the document!
>But I don't know what takes "2 minutes" then? Checking to make sure they actually have hyperlinks?
The task at question. Seeing if "hyperlink" returns a case the citation. It's even more trivial than getting the resource itself. You type it in and don't even have to click the download button. Of course, an attorney actually has to read a case to see if it stands for the proposition they are using it for. But you don't care about that because you're a super genius entertaining a hypothetical.
>That is not what they asked. You only make yourself look bad when you change the question before mocking it.
I'm not the one making myself look bad because I'm not the one entertaining the nonsensical hypothetical of some other internet super genius who knows so much about the legal profession that they've never heard of LexisNexis, WestLaw, the Federal Reporter or the Blue Book.
Legal citations are already a format you can plug into a legal database to get a result, so the idea that it'd be some sort of improvement to see if a citation actually exists when your AI makes up complete bullshit isn't an advancement, it's back to the status quo. Because an attorney needs to actually read the cases they cite to be sure they stand for the proposition they are relying upon. They also need to read the propositions they are representing to the court. So nothing has changed about that. But again, you don't realize that because you have no idea what an attorney needs to do to make their time more valuable. (It's hiring and training junior attorneys.) That's what senior lawyers have been doing forever.
Actual attorneys do the job way better job than AIs. That hasn't changed yet and it doesn't seem like it will anytime soon based upon the AI demos I've been given. The only people who tell me otherwise are posters like you and the hucksters that sell the demos. At least the others are hucksters. You're some poster whose going to argue with me about what will make my job easier, having no idea what I actually spend my time doing. It's always the posters that abstract it into some obtuse operation like "filling out a legal document with facts and relevant law" like they are just two buckets getting sorted. It's so tiresome.
So that's per citation? Then two minutes each is a waste of time for basic checking.
> who knows so much about the legal profession that they've never heard of LexisNexis, WestLaw, the Federal Reporter or the Blue Book.
This accusation is not supported by what they said.
> Legal citations are already a format you can plug into a legal database to get a result, so the idea that it'd be some sort of improvement to see if a citation actually exists when your AI makes up complete bullshit isn't an advancement, it's back to the status quo. Because an attorney needs to actually read the cases they cite to be sure they stand for the proposition they are relying upon.
Again a lot of that comment is about getting a document from someone else and quickly checking validity.
No, it's not two minutes per citation.
>This accusation is not supported by what they said.
It is.
>Again a lot of that comment is about getting a document from someone else and quickly checking validity.
You have no idea what you're talking about. Have a nice time going off.
Please clarify the statement you made. You said two minutes. What specifically are you saying takes two minutes?
You said it's "not per citation" but you also said that it's "Seeing if "hyperlink" returns a case the citation." which sounds like a description of a per-citation action.
And then you described it as "You type it in" which would mean: A) it's not a link B) something to automate that would be useful and C) typing in every citation in an entire document sounds like it would take longer than two minutes.
> It is.
Them using the phrase "some relevant database" doesn't mean they're unaware those things exist! What they were uncertain about is the minutiae of how they're accessed. You're misinterpreting them just to insult them.
> You have no idea what you're talking about. Have a nice time going off.
I'm talking about what their comment means. You are not an authority on that.
Especially when they already made a followup comment saying you interpreted them wrong.
Neither are you. Yet...
>Especially when they already made a followup comment saying you interpreted them wrong.
Once again, you aren't getting the thread. That poster was being facetious.
Good enough, I'll take that as agreement.
And you're still not going to explain what you meant? Okay bye.
https://apps.calbar.ca.gov/attorney/Licensee/Detail/282372
Doesn't seems like there isn't kind of disciplinary action. You can just make up stuff and if you're caught, pay some pocket change (in lawyer money level territory) and move on.
https://www.courtlistener.com/docket/68990373/nz-v-fenix-int...
What's interesting about the rules of civil procedure is that it has been built up over centuries to prevent all kinds of abuse by sneaky, clever, unscrupulous litigants. Most systems are not so hardened against bad faith actors like the legal system is and AI just thinks it can pathologically lie its way through cause most people trust somebody who sounds authoritative.
So lawyers use it, judges use it ... have we seen evidence of lawmakers submitting AI-generated language in bills or amendments?
There was an entire team dedicated to this work, and the hours were insane when the legislature was in session. She ended up not taking the job because of the downsides associated with moving to the capital, so I don't know more about the job. I'd be curious how much AI has changed what that team does now. Certainly, they still would want to meticulously look at every character, but it is certainly possible that AI has gotten better at analyzing the "average" ruling, which might make the job a little easier. What I know about law though, is that it's often defined by the non average ruling, that there's sort of a fractal nature to it, and it's the unusual cases that often forever shape future interpretations of a given law. Unusual scenarios are something that LLMs generally struggle with, and add to that the need to creatively come up with scenarios that might further distort the bill, and I'd expect LLMs to be patently bad at creating laws. So while, I have no doubt that legislators (and lobbyists) are using AI to draft bills, I am positive that there is still a lot of work that goes into refining bills, and we're probably not seeing straight vibe drafting.
Most people would be shocked to find the majority of bills are simply copycat bills or written by lobbyists.
https://goodparty.org/blog/article/who-actually-writes-congr...
Bank lobbyists, for example, authored 70 of the 85 lines in a Congressional bill that was designed to lessen banking regulations – essentially making their industry wealthier and more powerful. Our elected officials are quite literally, with no exaggeration, letting massive special interests write in the actual language of these bills in order to further enrich and empower themselves… because they are too lazy or disinterested in the actual work of lawmaking themselves.
a two-year investigation by USA Today, The Arizona Republic, and The Center for Public Integrity found widespread use of “copycat bills” at both federal and state levels. Copycat legislation is the phenomenon in which lawmakers introduce bills that contain identical langauge and phrases to “model bills” that are drafted by corporations and special interests for lobbying purposes. In other words, these lawmakers essentially copy-pasted the exact words that lobbyists sent them.
From 2011 to 2019, this investigation found over 10,000 copycat bills that lifted entire passages directly from well-funded lobbyists and corporations. 2,100 of these copycat bills were signed into law all across the country. And more often than not, these copycat bills contain provisions specifically designed to enrich or protect the corporations that wrote the initial drafts
MPs are definitely using AI to write their speeches in parliament: https://www.telegraph.co.uk/business/2025/09/11/chatgpt-trig...
This lawyer fabricating his filings is going to be among the first in a bunch of related stories: devs who check in code they don't understand, doctors diagnosing people without looking, scientists skipping their experiments, and more.
You're thinking too linearly imo. Your examples are where AI will "take", just perhaps not entirely replace.
Ie if liability is the only thing stopping them from being replaced, what's stopping them from simply assuming more liability? Why can't one lawyer assume the liability of ten lawyers?
Just like with a lot of other jobs that got more productive.
They don't understand how to calibrate their model of the world with the shape of future changes.
The gap between people who've been paying attention and those who haven't is going to increase, and the difficulty in explaining what's coming is going to keep rising, because humans don't do well with nonlinearities.
The robots are here. The AI is here. The future is now, it's just not evenly distributed, and by the time you've finished arguing or explaining to someone what's coming, it'll have already passed, and something even weirder will be hurtling towards us even faster than whatever they just integrated.
Sometime in the near future, there won't be much for people to do but stand by in befuddled amazement and hope the people who set this all in motion knew what they were doing (because if we're not doing that, we're all toast anyway.)
AI offers an incredible caveat emptor tradeoff: you can get a lot more done more quickly, so long as you don't care about the quality of the work, and cannot hold anyone responsible for that quality.
https://www.theguardian.com/us-news/2025/apr/24/california-b...
Using a tool that is widely known to be flawed to provide any sort of professional service (legal, medical, accounting, engineering, banking, etc.) is pretty much a textbook definition of negligence.
And lawyers just love negligence.
Do you think a Civil Engineer (PE) should be held liable if they vibe engineered a bridge using an LLM without reviewing the output? For this hypothetical, let’s assume an inspector caught the issue before the bridge was in use, but it would’ve collapsed had the inspector not noticed.
A single person can design a building, why not a bridge?
P.S. I sell and run commercial construction work
> Someone would just take the plans exactly as listed, purchase the material as listed, and not one question ever would be raised? I would never trust a construction team that didn’t raise questions if not even to see if they themselves could skimp on material to pocket the difference.
You’re right, the contractor would likely catch the design issues if there were any, and possibly before that in the plan review/permitting process if the AHJ is on the ball.
I work in the electrical trade and I (and my electricians) find and correct errors frequently in engineered plans. We tell the engineer if it costs us more money to attempt to get a contract change order, but we keep it to ourselves if we can do it safely for cheaper. A common scenario I run into is a design with oversized feeders where you can use a smaller wire and still meet code, we just pull the smaller conductors and pocket the difference (assuming you bid the project using the larger wire size)
Why jail time for lawyers who use Chat-GPT, but not programmers? Are we that unimportant compared to the actual useful members of society, whose work actually has to be held to standards?
I don't think you meant it this way, but it feels like a frank admission that what we do has no value, and so compared to other people who have to be correct, it's fine for us to slather on the slop.
Programmers generally don't need a degree or license to work. Anyone can become a programmer after a few weeks of work. There are no exams to pass unlike doctors or lawyers.
In absence of mitigations like laws and exams, it makes more important to use criminal and civil law to punish bad programmers.
speak for yourself. some of us are ready to retire and/or looking for parts of the field where code generation is verboten, for various reasons.
Sort of like recklessly vibe coding and pushing to prod. The cardinal rule with AI is that we should all be free to use it, but we're still equally responsible for the output we produce, regardless of the tooling we use to get there. I think that applies equally across professions.
There should be a system where software developers are held personally responsible for various offenses, e.g. helping their employers break laws, but there also need to be legal protections that allow us to refuse such work without facing repercussions.
Surely it would suffice to eject him from the California bar -- or suspend him from it for a time.
After that allow him to retake all the exams to get licensed again if he wants to.
What do you understand sacred to be, and why would you include the legal system in that category?
They are great for a large number of other things, like generating code for example.
We are so focused trying to shoehorn LLMs to do something it is fundamentally unsuitable for that we are probably missing the discovery of the technology that would solve this.
They don't care if you use an AI or a Llama spitting at a board of letters to assemble your case, you are responsible for what you submit to the court.
This is just a bad lawyer who probably didn't check their work in many other cases, and AI just enabled that bad behavior further.
As time goes on, it becomes less and less defensible to see this stuff.
I think this is a good reason for fines to not be incredibly big. People are using AI all the time. There will be a growing period until they learn of its limitations.
After asking for recommendations, I always immediately ask it if any are hallucinations. It then tells me a bunch of them are, then goes "Would you like more information about how LLMs "hallucinate," and for us to come up with an architecture that could reduce or eliminate the problem?" No, fake dude, I just want real books instead of imaginary ones, not to hear about your problems.
detail: The question was to find a book that examined how the work of a short order cook is done in detail, or any book with a section covering this. I started the question by mentioning that I already had Fast Foods and Short Order Cooking(1984) by Popper et al. and that was the best I had found so far.
It gave me about a half dozen great hallucinations. You can try the question and see how it works for you. They're so dumb. Our economy is screwed.
Fines are arbitrary numbers set by some people not necessarily knowing about other fines for other offenses.
This is a bit like all the stats like "this is appears to be an unprecedented majority in the last 10 years in a Vermont county starting with G for elections held on the 4th when no candidate is from Oklahoma".
Lots of things are historic but that doesn't necessarily mean they're impressive overall. More interesting is how many of these cases have already been tried such that this isn't "historic" for being the first one decided.
I didn't see them mentioned in the article.
[1] https://www4.courts.ca.gov/opinions/documents/B331918.PDF
I have worked in litigation and if you submit some litigation to a court clerk that is not perfect you won't even get a reply that it's no good. They'll just throw your brief in the trash without looking at it and never call you back.
Tools like Lexis+ from LexisNexis[1] now offer grounding, so it won't be as simple to bust people cutting corners in the future because these prevent the hallucinations.
We're now closer to the real Butlerian Jihad when we see pro se plaintiffs/defendants winning cases regularly.
[1] https://www.lexisnexis.com/blogs/en-au/insights/hallucinatio...
Here we have irrefutable evidence of just how bad the result of using AI would be and yet... the response is just that we need to accept that there is going to be damage but keep using it?!?
This isn't a tech company that "needs" to keep pushing AI because investors seem to think it is the only path of the future. There is absolutely zero reason to keep trying to shoehorn this tech in places it clearly doesn't belong.
We don't need to accept anything here. Just don't use it... why is that such a hard concept.
It's not just the result of using AI, it's the result of failing to vet the information he was providing the court. The same thing could've happened if he hired a paralegal from Fiverr to write his pleadings and didn't check their work.
It's like saying that because he typed it on a computer, it's the computers that are the problem, and we shouldn't keep using them.
We're already at least a year past AI tools having the ability to perform grounding (Lexis+ from LexisNexis, as I cited on another comment in this post, for example), so this whole fiasco is already something from a bygone era.