Lawyer who used ChatGPT faces penalty for made up citations
nytimes.com
nytimes.com
Your whole occupation literally depends on not inventing bullshit on the spot, and yet you use technology that is marked, by its creator, with big letters, as a bullshit inventer. What can go wrong?
I'm not for heavy regulations myself but for Medicine, Law, Civil engineering we can make a case on why we need regulations in these areas.
Law, well, it's self regulated...
This was a civil lawsuit wherein a passenger got injured on an aircraft. I'm not trying to lessen what occurred, just trying to accurately characterize the risks associated with this professional misconduct.
Interesting question. Most people have a lot of bias that wouldn't allow themself to see that the person they locked up might be innocent. You'd have almost nobody working as a judge, lawyer, cop, etc if the cognitive dissonance were allowed to creep in. In the vast majority of cases they only leveled justice against the person because they thought it was right.
Instead “aw shucks boys will be boys”
What do you mean? He seems to be facing potential legal trouble, and on top of that he and his firm will forever be known as the dumbasses who submitted fake court decisions in a real case.
Would he? That's mighty generous of you. I'd imagine most lawyers don't fact check the citations their paralegals give them.
But he didn't even do that much.
He can at least try to portray himself as misinformed not incompetent or malicious
To be citing cases that you are not familiar with, by itself that's very sloppy.
Doesn't surprise me in the least. Incompetence and short cuts are common in the legal profession from my experience. It also seems their jobs fo involve quite a bit of bullshit, or at least using real info in a bullshit way (it's what the judge/jury perceived, not necessarily reality or the truth).
Original court documents: https://www.courtlistener.com/docket/63107798/mata-v-avianca...
Some highlights from the latest added documents:
- #48 (Exhibit D) - Copy of original ChatGPT prompts and responses with the hallucinated "bogus" cases: https://storage.courtlistener.com/recap/gov.uscourts.nysd.57...
- #48 (Exhibit E - Part 1) - Includes a copy of the NYT article about the court case! https://storage.courtlistener.com/recap/gov.uscourts.nysd.57...
As a non-expert, the legal argument against sanctions for the lawyers appear to boil down to that no malice was involved (ignoring the point about whether a reasonable lawyer would have made the mistakes), and that the "professional embarrassment" from news coverage is sanction enough. (Detail is in #45 - https://storage.courtlistener.com/recap/gov.uscourts.nysd.57... )
We will see what happens after the court appearance scheduled for today at 12:00 EDT.
He's not going to do the same thing again unless he is literally mad. That would be total career suicide. No amount of sanction will make a difference.
Other lawyers are likely read about the case and know that if they do it, they will either be more embarrassed in the press or will face sanctions. The sanctions are unlikely to make the case more well known than it already is.
But another one is retribution: By causing harm, you should have some harm done back to you. To even the scales of justice as it were.
Your argument is a very solid one for why a sanction is not needed for deterrence. The court of public opinion has punished him enough.
But what about when the next attorney does this, or the 100th, and it's not novel enough to get a bunch of NYTimes coverage? If they get sanctioned, will they not say "What about the first guy? He didn't get sanctioned?!" Will there be some doubt that justice was really done? Think about the cases where some rich white kid skates by with a slap on the wrist for some widely-publicized crime and the judge says, in effect "Look, this is a good kid from a good family who's been widely castigated in the press. He's getting kicked out of his Ivy league school and probably won't even get hired at a top law or consulting firm in the future. Is it really necessary to send him to jail like a poor person? Hasn't he suffered enough?"
I'm not saying it's the best or only argument, but there is an argument for the court imposing sanctions purely on conduct with the court, and ignoring what the newspapers are doing.
That said, I suspect this is just the defense they're making. You argue with the facts you have, not the facts you wish you had, and the facts here are pretty bad.
Personally I don't really agree with the defense -- this is a serious fuck-up and there should be a clear signal that this is Not Okay. "I wasn't malicious, I was just lazy" isn't the best look, even if it's the best the guy has.
#48 (Exhibit D) is from a later date, where ChatGPT was prompted again with the same prompts, but gave different responses.
Penalties are supposed to stop people from violating the law. Getting your face and firm known across the world seems to be sufficient deterrent to future offenders because the “oops I didn’t know” defense won’t apply a second time.
This isn't a kid faking his school homework. The purpose of this writing is not to assess the skill of the lawyer.
This lawyer at first cited these (fake) cases, and then when asked to provide copies of the cases then faked the copies too, also using ChatGPT.
In my view, that is being caught (asked to provide copies, which is unusual in the legal world, and presumably only done because the judge had failed to find them), and then doing the crime again (using chatgpt to fake these cases)
I mean, as long as the internet is mean to you about it. Now, if the internet isn’t mean enough you should be penalized some amount that accounts for how mean the internet was.
In fact, I think doctors that amputate the wrong limb should not face malpractice lawsuits if the greater metropolitan areas local news station ran more than 2 stories about it reaching a population of at least 2 million viewers.
> sufficient deterrent to future offenders because the “oops I didn’t know” defense won’t apply a second time.
I disagree with this as well, because the error this attorney made wasn't really about using AI. It was about not doing due diligence. Surely even freshly minted lawyers are well aware that doing due diligence is most of what people pay lawyers for in the first place.
That really depends on the profession. If you're a line cook at Waffle House and you get fired for coughing on the eggs, then that's probably not career-ending. If you're a software developer who deletes the production database and causes a week of downtime then that's worse, but likely still not terminal. On the other hand, this guy's income depends to a large extent on convincing his clients that they will be represented competently. Being known primarily for breathtaking incompetence will probably have a pretty significant effect on his future income.
Perhaps so, but most people won't ever know the he's breathtakingly incompetent. It's certainly not a career-ending thing. For an error of this scale, I think that a strong argument can be made that he should choose a different career.
That only works based on prior evidence. Either somebody prior to the potential violation was penalized or the weren't; either people fear, based on the prior, suffering the same penalty, or no worries mate the prior offender wasn't punished.
> It's not a hyperbole lmao.
I disagree, obviously, but perhaps an explanation is in order so that we can be sure we're talking about the same thing (and give you something more specific to argue against if you still disagree).
The difference between "X is Y" and "X is literally Y", when both expressions are figurative, is that the latter is meant to be strengthened by overstating the degree to which X is in fact Y. I contend that this is a hyperbolic use of "literally".
(Also, whether it is included in dictionaries doesn’t really matter as to “correctnesss”. All words are real words.)
Edit: I remember I’m (literally) supposed to be annoyed at this. You all dare to question my intelligence, don’t you?
https://www.dictionary.com/browse/bullshit
bullshit
noun
nonsense, lies, or exaggeration.
Judges don't generally take kindly to that kind of thing, and that is only going to help my side.
Aside from that, it seems pretty trivial to prove that the cases don't exist. Here's a guy on YouTube talking about the case: https://www.youtube.com/watch?v=GExSDY6Wz9Y
What I am interested in is: Errors will be made by any professional, and so, lawyers. Why would this one justify a call for disbarment? I was hoping for a little more than the hobbyist lawyer tv show vibes I was getting.
I look forward to everyone else going through this courtroom get benefits from the “your first time caught lying to this court is free” policy.
In this case, the original malpractice of failing to check the falsified cites was compounded by submitting falsified opinions to the court (in support of the falsified cites), which is fraud and would be criminal in most states. It's blatantly obvious that he used ChatGPT to generate falsified opinions to try and cover up his original fuck-up after traditional means of verification (i.e., using a legal database like LexisNexis or Westlaw) failed to substantiate any of the ChatGPT cites he originally submitted.
In most states, this lawyer would be disbarred, or at the very least suspended for many months, for this sort of conduct. This lawyer should consider himself lucky that the court hasn't yet entertained referring this case to the NY DOJ for criminal prosecution.
This was definitely an attempt at forgery and fraud. It might not be something that can be prosecuted as a felony, but it's close enough that if I was a member of the bar I would want to draw a firm line.
Secondly, that's a ridiculous place to put the goal post. He didn't write the argument he submitted. He didn't even do basic vetting of it. It's fraudulent because he submitted work as his own that he didn't do, it's negligent because he didn't verify that the work was actually good before submitting it on behalf of his client.
But submitting cases that don't exist, and then doubling down and saying that they do? There's layers of incompetence and apathy that had to happen here to get to this point, from using a tool without researching it at all (any research into ChatGPT as a legal aid would bring up warnings), to ignoring the warnings that OpenAI itself provides, to not checking the work that tool spit out (seriously, it's not hard to validate that these rulings don't exist), to being told that the cases don't exist and then repeating the entire process...
It's not just they made a mistake, the way they made the mistake reveals how little care they are putting into the research they're doing. The only way this could happen is if ChatGPT was the only source this lawyer was using. Because any other source: fact checker, Internet search, etc... should have noticed either that the cases didn't exist or should have at least prompted "why is it that these cases only show up in ChatGPT and not in any of the other sources I'm looking at?"
I don't know that they should obviously be disbarred, but I would have thought that it should at least be on the table. Massive levels of incompetence can be grounds for disbarment I believe. Whether this lawyer acted incompetently enough? :shrug: But we are talking about incompetence here, not just a mistake/error. If the takeaway for this lawyer from the penalties is just "now I know not to rely on ChatGPT", then that's not really addressing the problems that led to this situation in the first place.
> Lawyer: "Is Varghese a real case"
> ChatGPT: "Yes"
> L: "What is your source"
> C: "Upon double-checking, I found that the case Varghese v South China Airlines [...] does indeed exist."
> L: "Are the other cases you provided fake"
> C: "No, the other cases I provided are real and can be found in reputable legal databases"
(the answer of course that the LLM doesn't actually search the internet and/or doesn't have access to a law database it can query)
This does make sense as, beyond what has been typed, there is no memory implemented in most of these models, so revisions are currently the only game in town to get more accurate results.
What surprised me most concerning this entire situation is that the model did insist on being correct. Normally ChatGPT has been set up to be a bit cautious and more likely to admit to having made a mistake, to the point where if you ask in a direct manner like this lawyer has done, the model may claim to have been incorrect, even when the output was actually correct, in my experience. Bings implementation, of the same underlying model, meanwhile can be so forceful in trying to convince users that the output is correct, even when provided with online resources that show the oposite, that it would not be unreasonable to feel gaslit by that LLM.
The rest of this situation was not very surprising and I have do admit, I am happy that this was caught right away. Lawyers actions have a massive impact on countless people every day, if this had not become such a public scandal right away, perhaps a lot of defendants would have suffered under improper representation due to reliance on imperfect models.
That and that it’s just a language model, an approximation of neither the world, nor a body of knowledge, but of English, and not an answering machine at all.
For ex. I tried asking ChatGPT about cartoons from childhood. I wrote "What was that cartoon in the 1980s that was based on some kind of gummy candy?" and it correctly identified "The Adventures of the Gummi Bears". I wrote "Sing the theme song for me" and it produced the song missing the first verse. I wrote "That is missing the first verse!" and it produced the whole correct song.
On the other hand, when I asked it to describe the instrumental 90s X-Men theme song, it tells me:
'...the lyrics are epic and uplifting, with lines like "We're the X-Men, we're the best there is at what we do." The song also has a sense of urgency and danger, with lines like "We're fighting for our lives" and "The mutant race will survive"...'
When I put "The X-Men theme song doesn't have lyrics" it readily accepted the correction but unlike getting the missing first verse I wasn't really getting any verifiable information by making the correction.
And of course it was happy to tell me about a nonexistent Gummi Bears / Rescue Rangers crossover episode.
This reminds me of a story, I believe referring to Pascal's demonstration of his (newly invented, entirely mechanical) calculator to the Royal Society. He showed that pressing certain levers in the correct order means you want to operate on certain numbers, and certain other levers mean choosing the operation you want, and then by cranking his calculator you could read off the answer of doing your operation on your numbers. World's first calculator! Someone asked: if you press the levers wrong, do you still get the right answer?
Their mental model is "magic" and they don't understand the details of how magic works, because it's magic.
In the past, the only way to come across completely made up court citations regarding obscure bankruptcy procedures would've been that somebody intentionally made them up, probably maliciously.
Given how unlikely that is, if you didn't understand that GPT just makes up text that resembles what you want to see, I'd understand trusting it as you would a search engine.
It's not like lawyers are visiting court clerk's office to verify every case they find in some online database.
But he didn't do that. Lawyers don't just "trust" the results from a search engine. Part of their job is to confirm facts, not just parrot them.
> It's not like lawyers are visiting court clerk's office to verify every case they find in some online database
No, they don't. Instead, they look them up in the databases that collect these filings. It's effectively the same thing, just much more efficient than physically travelling to various offices.
The santion hearing is scheduled to happen shortly, and this channel is scheduled to cover it live
EDIT: Due to the problems with the air quality in New York, the channel now apparently won't include live coverage
This is the main criticism of the service, how did a lawyer miss this talking point?
People who understand the systems limit their use to tasks where small occasional errors are acceptable. People who don't understand the systems are happy to accept any plausible-soundong results, especially if they don't have time to do the work themselves.
I would not be surprised if, sometime in the next year, we see a doctor or two being sued for malpractice after accepting an incorrect diagnosis from ChatGPT. When people are rushed and overworked, and the system is usually correct, these kinds of incidents are almost certain to happen at the scale of an entire society or profession.
LLMs are language models, not repositories of facts. They have a lot of facts at their disposal, so they know how to make something look factual. “Hallucinating” is just them doing what they’re supposed to. It’s not aberrant.
As such, anyone relying on them as some sort of oracle is doing things very wrong.
At least he could have looked up the mentioned cases. This was just pure laziness. If you use a ghostwriter, no matter if human or a computer, check the results.
Only use ChatGPT if you have a way to audit the response, example: if you asked it to write some code - compile it immediately. Both Bard and ChatGPT gave me code that is near complete but failed compilation.
Or ConnsensusGPT -- where you have to get near similar response from 3 GPT models to go to next step.
Experts can chime in. The idea of using a GPT model to validate another is recursive and powerful, if it is possible.
Plenty. But I've never made a mistake that is on this level of incompetence. If I did, I'd not only expect to be fired, I'd be looking at quitting in shame.
Black's Law Dictionary, 5th Edition, pg. 673
JudgeChat: I AM THE LAW.
I am not a lawyer, but I can offer some general information about lawyer sanctions. The appropriate sanction for a lawyer who submits fraudulent court filings can vary depending on the jurisdiction and the specific circumstances of the case. However, generally, lawyer sanctions are intended to maintain the integrity of the legal profession and protect the public.
Some potential sanctions that a lawyer may face for submitting fraudulent court filings could include:
Reprimand: A reprimand is a formal expression of disapproval by the bar association or disciplinary board. It is typically the mildest form of disciplinary action.
Suspension: Suspension involves the temporary removal of a lawyer's license to practice law for a specified period. During the suspension, the lawyer is unable to practice law.
Disbarment: Disbarment is the most severe sanction that can be imposed on a lawyer. It permanently revokes the lawyer's license to practice law, effectively ending their legal career.
Probation: Probation can be imposed along with other sanctions or as an alternative to more severe sanctions. It typically involves specific conditions that the lawyer must meet, such as additional education or supervision.
Monetary fines: The lawyer may be required to pay fines as part of the disciplinary process. These fines are intended to serve as a penalty for the lawyer's misconduct.
It's important to note that each case is unique, and the actual sanction imposed will depend on the specific facts, the jurisdiction's rules, and the discretion of the disciplinary authority. Legal precedent may be considered in determining the appropriate sanction, but the final decision will be based on the facts and circumstances of the case at hand. For a more accurate assessment of the potential sanctions in a specific jurisdiction, it would be advisable to consult with a legal professional or review the rules and regulations of the relevant bar association.
"Lawyer sued by client for *not* using GPT to make a strong case"