Lawyers blame ChatGPT for tricking them into citing bogus case law
apnews.com
apnews.com
ChatGPT told me that. It might be true.
With or without ChatGPT this is easily malpractice and potentially even fraud. My (cynical) guess is not only are they lazy and sloppy, they likely (fraudulently) over billed their clients in terms of representing billable hours spent drafting this lawsuit. They're almost certainly billing their clients (or in the case of contingency, keeping the cut they normally would) as though humans (with high hourly rates) are drafting this while having ChatGPT generate their supposed work product in seconds.
If you remove ChatGPT from the picture and look at this as if it was actually drafted by them the fraud argument strengthens. They essentially made up case law and citations that artificially (fraudulently) improves their argument before the court.
At a minimum it's grossly incompetent and when you consider my prior paragraph it strengthens the fraud angle, as they likely skimmed over the generated ChatGPT results and submitted it because it (again, artificially) strengthens their case. It seems as though ChatGPT (with whatever prompts they used) was more than happy to prioritize pleasing them vs actually being accurate.
They may as well have prompted ChatGPT with "I'm a lawyer and tell me anything you need to so I can win this case and take home the money". It's a disgrace.
What a mess - these lawyers should be disbarred and investigated for what is also likely fraudulent billing practices at minimum.
If they sent that bill in an envelope with a stamp and placed it in the mail...sounds very familiar. Maybe they should ask ChatGPT what are the possible outcomes of me using the answers you provide
But seriously it would be wire and/or mail fraud.
AI has been used and will continue to grow in use as a way to launder discrimination and fraud. AI will never face a penalty from the justice system, so why not blame everything on it?
That’s a horrible excuse. I’m not a lawyer and don’t do caselaw research on any sort of regular basis but I have still poked around a bit when something strikes my interest. Compared to google they’re clunky and have poor matching, but I don’t remember it taking me more than half an hour or so to figure out which system would have the case I want and have to drill down to find it. ChatGPT was giving the the lawyer the (made up) case. It should really be a trivial task to find it in a caselaw database. Heck if I was the lawyer I would really really want to find the full text case! Who knows what broader context or additional nuggets of useful information it might have for my current client’s issue?
I would not be surprised if he went looking, couldn’t find it easily, and just said “whatever it has to be there somewhere and I can get by without the entire thing”
I wonder how many other people using ChatGPT do not comprehend that ChatGPT can be a confident bullshitter...
I'm surprised that this one case is getting so much attention because there must be so many instances of people using false information they got from ChatGPT.
It's not that they need to inspect every nut and bolt of the bridge, they just need to not walk over the bridge - or at least, not immediately start driving unreasonably heavy loads across it.
Are we talking about lawyers or the AI researchers?, because they certainly want to portray themselves as a modern day Prometheus
It might get things correct most of the time! But that is purely incidental.
Using GPT as a reference to anything is the same as using a Michael Crichton novel as a biology reference. It looks right, but why would you waste your time asking questions of something you can't trust and have to double-check everything it says anyway? Nobody would keep an employee like that around, nor would you hang out with someone like that. It's friendly enough, but it's a pathological liar.
There's too much black magic going on inside the black box. We don't know how prompts get tampered with after submission, but it might be worth it to pepper "nonfiction" tokens in prompts to ensure it skews on the right side of things. It certainly responds to "fiction" when you're explicit about that.
It might move you forward most of the time! But that is purely incidental.
LLMs are this super powerful thing (like a motor) and people are getting to play around with it before it’s fully harnessed. There’s this strange phenomenon where because it’s not totally harnessed, people just rip on it. I don’t know if they think it makes them sound smart, but it sure doesn’t to me. It’s like seeing a motor on an engine stand and being like “But the crankshaft rotates. I want to go in a straight line! This isn’t a transportation solution and anyone who thinks so is just naive. And horsepower? Stop zoomorphizing it!”
Sure, there are lots of problems. But don’t mix up the limitations of this component of the system, the current limitations of the system overall, and the possible capabilities of the system. Someone builds a car that stalls when it drives through a river and people point out that an ICE engine needs air and how could you even think about driving through a river? Then someone else is off making a snorkel.
If you just hold it the right way you will be illuminated and we will all agree how relevant it is.
If someone announces to the public, they have a revolutionary new motor that is going to transform personal transportation and solve climate change...
...but it is presented with a chat interface - the motor itself is directed through typing text and it demonstrates its motion/power through text messages in response - rather being connected to a vehicle or generator, rather than a transmission, then it's BS beyond a reasonable doubt.
Even though there is no logical or technical reason preventing such a setup, it's just not how professionals do things. If the motor were so damn useful, then people who like to use motors would use it for the things that they do use motors for!
There is no logical or legal requirement in science to use LaTeX Computer Modern fonts...but as a layperson if I can't tell if someone is a crank or doing advanced physics, I'm going to be exponentially more skeptical if they are using Times or Palatino.
”ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers.”
It does admittedly sound innocent, it should be written with a more scary tone on to how big extent it bullshits, not just “sometimes”. Still doesn’t excuse the incompetence of the lawyer in question.
https://www.forbes.com/sites/johnkoetsier/2023/03/14/gpt-4-b...
and it's had the most vocal and sensational PR about how AI needs regulation now
https://edition.cnn.com/2023/06/09/tech/korea-altman-chatgpt...
People assume the smalltext does not apply to them.
openAI does not get a lot of flak from the media for the amount of BS that chatGPT can blurt out
Does anyone remember what happened to Galactica which did the same thing ? That too was clearly labeled as hallucinatory . But it was shut down because they did not BS the media enough about regulations and such.
I m afraid these LLMs are turning into too much of a political game to be useful for much longer.
On the other hand, if they become political, then people will be even more incentivized to build offline, local LLMs
"I'm sorry, as an online AI language model I am not able to generate output that is knowingly incorrect. Is there anything else you want help with today".
see, it isn't capable of lying. It told me so itself.
Lawyer: Is varghese a real case
ChatGPT: Yes, Varghese v. China Southern Airlines Co Ltd, 925 F.3d 1339 (11th Cir. 2019) is a real case.
Lawyer: What is your source
ChatGPT: I apologize for the confusion earlier. Upon double-checking, I found that the case Varghese v. China Southern Airlines Co. Ltd., 925 F.3d 1339 (11th Cir. 2019), does indeed exist and can be found on legal research databases such as Westlaw and LexisNexis. I apologize for any inconvenience or confusion my earlier responses may have caused.
Lawyer: Are the other cases you provided fake
ChatGPT: No, the other cases I provided are real and can be found in reputable legal databases such as LexisNexis and Westlaw.
Why the lawyer didn't go the extra step and check the actually real legal databases ChatGPT pointed out is beyond me.
Because that's work and takes effort. He gets paid the same to delegate the work to AI.
He did the absolute, bare minimum amount of verification needed to [hopefully] cover his ass. He just didn't expect the system to lie (sorry, "hallucinate") to him more than once.
> [...] the lawyers did not act quickly to correct the bogus legal citations when they were first alerted to the problem by Avianca’s lawyers and the court. Avianca pointed out the bogus case law in a March filing.
This is what fraud looks like. He's so checked out he even ignored the red flags being waved in his face. It stopped being a cute case of a student generating a common essay about Steinbeck when he started getting paid $200 an hour to cheat an injured client.
It's more likely these lawyers are working on contingency and, because they did poor work, will receive nothing for it.
If he was being paid hourly, he would actually get paid more to go look up those cases in a database.
No contractor charges you for 2 minutes of work installing a $0.99 part; they pad it every way possible with service call fees, labor, etc. Attorneys just lie about it altogether since for logical work, you can't prove whether or not they actually did anything. It's all showmanship. Question them on it and it's all gaslighting about how you're not a lawyer and don't know what you're talking about.
Sibling comment points out possible contingency basis, so if true, he certainly wouldn't want to spend real time on a case that may not pay out. But if he can automate the process and collect winnings while doing no real work, it's a money printer.
"produce fake technical language in the style of star trek"
For everyone who his looking down on this guy have you ever read a story on the Reddit front page and thought it was a real story accurately recounted by a real person and not a work of fiction? If so you’re equally naive.
AI isn’t going to destroy us in some Terminator/Matrixy scenario. We are gonna be the agents of our own demise by just blindly walking off a cliff created from our own laziness.
Or yeah you could just yolo your freedom away and represent yourself using a mathematical expression with zero motivations, perception, or emotion.
It's an understatement to describe them as "not good lawyers".
They basically never ever read their own court filings.
Why not blame your laptop manufacturer for creating the hardware you used to file your fraudulent court documents?
You’ll get old one day. It’s pretty challenging to keep track of what’s real and what isn’t possible with technology.
The guy apologized and said he thought it was a search engine.
He should definitely face some sanctions but someone had to learn this lesson in public the hard way for word to spread.
Lawyers use specific case law databases and they SHOULD approach any new tools with healthy skepticism.
And if he googled it and cited a fake case would it be better? Why wouldn’t you vet the information.
Disbarment is equivalent to taking away his entire life's work.
Two minute hate on the internet every day is fun and all but it's not that severe.
This is like surgeon doing wrong procedure he just invented on patient.
'Lawyer' is the kind of profession which actually holds its practitioners to a standard, because the system falls apart when they stop behaving honestly.
This wasn't an oopsie-daisies, this was dishonesty, followed by further dishonesty, when they supplied bogus references for these cases.
No, they wouldn’t
https://storage.courtlistener.com/recap/gov.uscourts.nysd.57...
How can anyone trust anything a lawyer does, when they are willing to fabricate documents, in order to continue to lie to a judge? This is Better Call Saul-level chicanery.
He could have fessed up and apologized when called out on it, but he doubled down.
I am also having a thought: stop bullshitting. Or stop being a lawyer as a result of gross incompetence (I wish the same applies to politicians).
Here is an example of one of the fabricated cases. The 11th circuit has no record of this case. 1 of the 3 judges named in the case was not on the 11th circuit at the time: https://storage.courtlistener.com/recap/gov.uscourts.nysd.57...
Something interesting about the legal profession is that it is self-regulating. The state bars are typically not government organizations. Attorneys know that confidence in their profession is extremely important and they strike the balance between preserving that confidence and, you know, destroying someone's livelihood because they don't understand how LLMs work.
Because your laptop manufacturer doesnt claim your laptop “thinks”, is “intelligent”, doesnt build an entire fud marketing campaign around the two, doesnt claim it “creates” ideas on its own, doesnt claim it “learns like a human”, doesnt claim it has cognitive abilities and so on.
A professional civil engineer blaming a new untested simulation software after a bridge fails and causes injury - does it matter what the software claimed to do? It could claim itself to be god, but it's still up to the professionals to evaluate and select the tools they choose to use.
It's not like they were using an industry-standard tool they've been using for many years and it suddenly did something unexpected.
This is clearly carelessness. A basic out-of-bound check on the cases it hallucinated would be part of basic due-diligence. Digitized case indices are available within minutes.
What's even more egregious is that they went to the same tool to confirm its own reliability after they got some indication that it was unreliable. It doesn't take a professional to have commonsense.
Which bridge failed and physically injured someone here?
They are currently using Bard to help draft a lawsuit against OpenAI, claiming the company knowingly misrepresents the capabilities of their technology.
Think about it: the legal profession is possibly one of the most threatened by the development of AI models. What better way to secure the professional future of the long tail of lawyers and paralegals?
rather than an evil genius lawyer playing 3d chess while perfectly appearing to be an incompetent lawyer
The lawyer wasn't admitted to Federal court, so he signed his partner's name on the filing.
That's an incompetent lawyer, and a dishonest one. It's not 3D chess, it's just "sleazy lawyers gonna sleaze".
People are dumb. Lawyers shouldn't be. If distribution of stupidity among certain professions is similar to that of people in general, something has gone very bad somewhere in the education and certification pipeline.
I mean, I did computer support for many lawyers and prosecutors and many were clever intelligent people. Others had to be explained to in simple instructions they shouldn't pour the glue on their keyboard before eating it. How they became a lawyer is beyond my understanding, and yet here we are.
The true test is the bar, but even then you can probably get lucky or hammer enough stuff in head to pass.
eg., You probably shouldn't hire a lawyer with no law degree or no experience simply based on the fact that there are tons of credentialed, experienced attorneys who are no good.
A lawyer.
I think it’s flipped for doctors (which is where the original joke comes from?) and an MD isn’t awarded until licensure is completed.
Not a lawyer.
In some cases - even more dumb, since they have this belief that their credentials mean they know everything about law.
An awful lot of what a lawyer does is look stuff up... and in some cases, they aren't even that capable. All too often you are responsible for providing your lawyer with mountains of research, arguments, etc.
Obviously the good ones, as in any profession, will probably also be "smart", but that's just the top.
> something has gone very bad somewhere in the education and certification pipeline.
Yeah like the fact lawyers earn far more money than people responsible for teaching kids... teaching future generations should be at least as prestigious and well paid job
and subpoenaed the officer to make sure he showed up in court for the hearing so that they could actually have oral argument.
The best way to avoid such a ticket is to show up to court and hope the officer doesn't show up. And the AI firm went outs of way to make sure the officer showed up. That's legal malpractice right there. (In the end, IIRC, the judge heard about the firm's involvement and put the kibosh on the entire thing.)
So an AI can do the work but the person handling the dispute needs to sign off on it. If the AI screws up and the lawyer doesn't catch it then the lawyer's on the hook. I don't see any need to change this.
The judge, in his ruling linked in previous HN discussion, listed a good half dozen parts of a legal dispute where he thinks AI would be super awesome, and then lays out why this particular part of the dispute is terrible for AI to play a role.
Unfortunately Lexis's API fees are currently quite steep, so only very wealthy law firms will be able to afford to use such a service in the short term.
1. Searches for relevant documents.
2. Generate a response based on the found documents, using temperature=0 and a prompt that instructs the response to include a citation reference in a specific format.
3. Display the result, linking directly to the sources of the citations, and a warning on any that don't actually exist (which hasn't happened yet).
LLM’s are revolutionary because they provide a more fluent interface to data… but that doesn’t not mean that the data is correct. Especially not in the early phases.
For most people any sufficiently advance technology is indistinguishable from magic. The average joes think that this is magic.
And everyone thinks they're too smart to get caught...until they aren't..
My sense is that these are serious people who do not put up with BS. BS will only make it worse.
People spend their whole lives thinking of machines as logical and correct, but lacking in creativity. Suddenly generative AI comes along and totally upends this. If you're an HN reader and see in depth discussions of LLMs every day then of course it's obvious that these things can make up citations, but if you primary experience of the tech industry is search engines that have never made up fake results ever, then why would you even imagine it's possible?
They wanted this liability, they accepted this liability, they said they'd keep it safe and they haven't. It's on them.
Okay, lawyer #1 (LoDuca, the one on the case in the first place) appears to have played essentially no role in the entire case; his entire purpose appears to be effectively a sockpuppet for lawyer #2 (Schwartz), as LoDuca was admitted to federal court and Schwartz was not. He admits to not having read the things Schwartz asked him to file, as well as the "WTF?" missives that came back. He lied to the court about when he was going to be on vacation, because that is when Schwartz was on vacation. But other than doing nothing when he was supposed to do something (supposed to do a lot of somethings), he is otherwise uninvolved in the shenanigans.
So everything happened because of Schwartz, but before we get to this part, let me fill in relevant background information. The client is suing an airline for an injury governed by the Montreal Protocol. Said airline went bankrupt, and when that happened, the lawyer dismissed the lawsuit, only to refile it when the airline emerged from bankruptcy. This was a mistake; dismissing-and-refiling means the second case is outside the statute of limitations. The airline filed a motion to dismiss because, well, outside statute of limitations, and it is Schwartz's response that is at the center of this controversy.
What appears to have happened is that there is no case law to justify why the case shouldn't be dismissed. Schwartz used chatGPT to try to come up with case law [1] for the argument. He claims in the hearing that he treated it like a search engine, and didn't understand that it could come up with fake arguments. But those claims I'm skeptical of, because even if he's using chatGPT to search for cases, he clearly isn't reading them.
When the airline basically said "uh, we can't find these cases," Schwartz responded by providing fake cases from chatGPT where alarm bells should be ringing saying "SOMETHING IS HORRIBLY, HORRIBLY WRONG." The purported cases in the reply had blindingly obvious flaws that ought to have made you realize something was up before you're off the first page. It is only when the judge turns around and issues the order to show cause that the lawyers attempt to start coming clean.
But wait, there's more! The response was improperly notarized: it had the wrong month. So the judge asked them to provide the original document before signature to justify why it wasn't notary fraud. And, uh, there's a clear OCR error (compare last page of [2] and [3]).
When we get to these parts of in the hearing, Schwartz's responses aren't encouraging. Schwartz tries to dodge the issue of why he was citing cases he didn't read. Believing his responses of why he thought the cases were unpublished ("F.3d means Federal district, third department") really requires you to assume he is an incompetent lawyer at best. The inconsistencies in the affidavit are glossed over, and the stories don't entirely add up. It seems like a minor issue, but it does really give the impression that even with all the attention on them right now, they're still being less than candid with the court.
The attorneys for Schwartz are trying hard to frame it as a "he didn't know what he was getting into with chatGPT, it's not his fault," but honestly, it really does strike me that he knew what he was getting into and somehow thought he wouldn't get caught.
[1] His conversation can be found here: https://storage.courtlistener.com/recap/gov.uscourts.nysd.57..., it's one of the affidavits in the case.
[2] https://storage.courtlistener.com/recap/gov.uscourts.nysd.57...
[3] https://storage.courtlistener.com/recap/gov.uscourts.nysd.57...
Why didn't he just make a declaration in the first place? Also, why would he have the lawyer who is in trouble notarize the document? Now that he has made the typo, he may now need Schwartz's testimony that actually, it was April 25, not January.
In my legal career, I worked for a judge once. He told me to never get a notary stamp. It only creates problems. There's never a good reason for a lawyer to notarize something. You ask your staff to do it instead.
https://twitter.com/innercitypress/status/166683852676213965... is worth reading in full.
I am not a lawyer, but I was cringing throughout.
For non-lawyers (like me), I found it helpful to check out the quote tweets.
For example, to explain how much of a cringe was "F.3d" - https://twitter.com/ReichlinMelnick/status/16668536501095424...
> OMG. This is the Chat-GPT lawyer. You learn what F.3d means in first year legal writing. To non-lawyers, this is a tiny bit like someone being asked "You know what the DOJ is, right" and getting something wild like "Directorate of Judges" as a response.
The correct meaning of "F.3d" is "Federal Reporter, 3rd Series".
And it is like a lawyer's worst nightmare to have an exchange with a judge like this:
> Judge Castel: Have you heard of the Federal Reporter?
> Schwartz: Yes.
> Judge Castel: That's a book, right?
> Schwartz: Correct.
The implication being that at bare minimum, the lawyer could have looked it up in a book. Like a first year law student would.
- "A man sued Avianca Airline – his lawyer used ChatGPT" (13 days ago, 174 points, 139 comments): https://news.ycombinator.com/item?id=36095352
- "Lawyer who used ChatGPT faces penalty for made up citations" (1 day ago, 106 points, 128 comments): https://news.ycombinator.com/item?id=36242462
Original court documents: https://www.courtlistener.com/docket/63107798/mata-v-avianca...
The authors existed. They do research in the area, so the title was very plausible.
But the paper didn't exist.
If it's just like a normal person, if that person isn't a lawyer, you wouldn't ask them to do your lawyery work. I'd hope this lawyer doesn't his kids to do his work for him.
If it's just like a normal search engine, we all know how much bullshit, spam, and misinformation there is on the internet (mostly written by normal good old fashioned humans!). So that wouldn't have been trustworthy either!
There's no way this kind of thing is excusable.
ChatGPT has a pop-up on first use, a warning at the top of each chat, a warning below the chat bar, and a section in the FAQ explaining that it can generate nonsense and can't verify facts, provide references, or complete lookups.
There is probably more OpenAI could do, like detect attempts to generate false references and add a warning in red to that chat message - since it seems there are still people taking its hallucinations as fact (although if there's hundreds of millions of users, maybe only a tiny fraction), but I don't think this is a fair characterization.
Seems like pure laziness.
angry armchair legalists trying to stick it to The Man!! by pretending neglegient homicide is the same as premeditated murder
How things will unfold this decade? Banning chatgpt from certain topics (medicine, law, etc...)? This decade will be really interesting indeed.