Hacker News Reader

Comment by ivraatiems | Hacker News Reader

I know someone who works in law and deals particularly with an area of US benefits and healthcare law. One of their workflows for lower-level employees at their firm involves taking in documents from healthcare plans and organizations, analyzing them for certain kinds of data, and then importing that data into an internal system they use to analyze and provide guidance on plans. The internal system can contain hundreds of documents for an individual client. All of the documents have the same information (roughly) but in totally diverse formats and styles. Once it's in the system, it's easy to compare and analyze across documents and the research process is much faster.

They recently bought a Claude subscription and began using Claude to do the initial read of the documents and output JSON they can import into their internal systems. The work still must be reviewed by an attorney - Claude is nowhere near making the kinds of judgments a lawyer would make about this content - but it has increased their throughput from 2-3 documents an hour to 8-10 documents an hour by killing the busy work.

LLMs have great advantages for this kind of work - but not for decision-making. I just don't see OpenAI ever admitting that.

(I've left some details intentionally vague because this is a very specific area of law and I don't want my friends to be identified without their consent.)

View on HN

I just realized how refreshing it is to read an honest take like "from 2-3 documents an hour to 8-10 documents an hour" instead of "it's doing the work of a month in 5 minutes!!!!1".

People will eventually realize that this is exactly what AI does for most workflows. The propaganda is "synthetic humans", "obsolete mathematicians" and so forth, but what we will likely see is a fantastic time saver for things we didn't like to do to begin with.

If mid-2026 LLMs were the last ones to ever exist, sure.

You're also assuming the next step towards AGI is an LLM. It's probably going to be another architecture entirely

and everything else is still speculation

Sure, nobody knows the exact future. But you still plan for what's probably going to happen. I don't know if social security is going to be around when I retire, and all the experts are saying it's probably not going to be. So, I'm planning a retirement without. Similarly, all the experts are saying AI (<- that word is not "LLM") is probably going to improve. I personally think it's irrational to ignore experts.

I think that a person (me) who has worked directly with frontier AI for over a year now for many hours a day may have gained some expertise in what this technology is capable of and thus what is dangerous about it, but I wouldn't speculate too hard on what superintelligent AI would or could do because we have not yet seen such a thing nor is there even an established criteria as to what that would look like or how it would be valued and in what capacity (hint: humans are still the final evaluators of what is valuable)

The simple fact that I have to write every control scheme in the book to keep the thing on the goal path (and not lapsing into laziness or dishonesty) is itself possibly a safety feature, if you look at it that way

> The simple fact that I have to write every control scheme ...

Look three years ago for a reality check of the progress. The LLM you're using now isn't the LLM you'll be using in 5 years. It probably won't even really be the same architecture, just like those from 3 years ago are VERY different than the full systems we use today (harnesses and all). What you're doing was literal science fiction not long ago.

The fact that you work every day with frontier AI doesn't make you an expert on the future technological developments.

People with experience in every other field have a better handle on what's coming than others, and yet experience with AI is apparently the sole exception to you LOL

That's a common misconception.

See e.g. https://scholarlycommons.law.northwestern.edu/facultyworking...

It's not a time saver in that you, as an employee, don't get more time to yourself. You get to work more. Can you walk up to your boss and tell them that you are going to start working 20 hours a week instead of 40?

I use these things daily, not against this kind of "AI" at all.

You're not wrong of course, but removing the tedious parts of a job can be really nice. I still work the same hours of course, but now I get to work on things that are more meaningful, more design and architecture oriented, and less about the grind of tedium. I'm sure there are plenty of exceptions out there, so I don't want to generalize, but I do suspect there are a number of people in a similar situation as me.

>removing the tedious parts of a job can be really nice

Sure. For me AI removes the parts of programming work that I like the most, and "gives" me more time to do thing I consider tedious, like solving human problems, achieving consensus in the team, attending meetings.

How can we deal with this mental overload? There are only so many things that I can keep in my head.

Your job was to deliver Project X and you did it in half the time. What do you do with the other half? Your job was to deliver X. Now what? Extra vacation time? Just musing.

Honestly, I know several devs who do chores around the house or even play video games while AI does the bulk of the heavy lifting. If nobody cares or even realizes, does it matter? (To be clear, they all work remotely.)

There's nothing that will save your employment time because any employer will want to maximize what they get from you. This applies to literally any innovation past, present and future.

For self-employed people, it allows them to focus on other problems. I know people who would be bankrupt without AI now because development time was crushing their company and they couldn't afford to hire others. Now they can spend more time in marketing and sales. AI will make some business that weren't profitable in the past viable.

For people who program for leisure, self-consumption or otherwise don't have a major income pressure, it is objectively a huge time saver.

For people who program for leisure, self-consumption or otherwise don't have a major income pressure, it is objectively a huge time saver.

I've had much enjoyment recreating my frequently used utilities on macOS. It's not about saving money, it's costing me more than to keep my subscriptions active, it's more about tailoring things to myself and keeping BPMs up.

There's nothing that will save your employment time because any employer will want to maximize what they get from you. This applies to literally any innovation past, present and future.

A personal anecdote from 2 decades ago which hunts me till today.

I was at a smallish firm in NYC, less than 50 people. I ran the "IT department" which included a couple of devs, DTP folks and one customer support underling who I backed up when needed.

We fixed all the printer problems, all the network shares, everything was quiet. I get called into the corner office and am told, "tell the help desk dude to sweep the floors as they are not doing anything". And, I really liked that boss. It's a different mindset.

If you’re a contractor, sure, same as always.

If you’re an employee, you are selling a quantity of your time, not your results. Of course you will have to work the same amount of time.

Your issue is with employment as a general concept, wherein the employer reaps any efficiency gains, not the worker.

You can if you don't tell them :-D

I was thinking about this today. I am definitely making things easier but.. if I was paying the actual costs of this service, could it even be close to worth it?

Well now that Astra for Law is here I give it about five minutes before social media grifters claim to be 10x lawyers

But wouldn't that be great if true? Why all the negativity over the prospect of leveling the playing between people lucky enough to be born with high IQs and average people? High IQ is a completely unearned gift and AI might reduce that fundamental source of unfairness in the world.

But my point is it won't be true, and people like to lie and exaggerate to get clicks and sell people courses

Which will end hilariously, let's face it

Agreed. However:

That's a factor of 3 to 5 improvement reported on here. If you keep stacking these kinds of improvements, you end up at month to minutes pretty quickly.

That doesn't take into account dependencies between tasks, where bottlenecks can happen. In any case, Amdahl's law applies regarding what kind of speedup you can get in such a case.

Compounding like this ignores fundamental friction. You can attempt a huge document processing factory but quality will go into the toilet.

10 women can't make a baby in a month

Throughput-wise, they can. Latency-wise, you are right.

Sorry for being so detached from the discussion and your actual point but if someone had made a dystopian sci-fi novel with the tagline following your sentence with "...or can they?", and made it all cheesy, I'd read it in one sitting.

Brave New World (and probably many other sci-fi) basically skips this problem entirely by having babies be mass-produced in bottles.

Since we are already on a tangent:

In one part of Brave New World they bring up that they tried to do more automation, but people got restless without the drudgery. In another part, they bring up that they tried to have a society with only Alphas, but they didn't want to do the drudgery.

I wonder why they never tried combining both?

You want to read a novel about ten women making cheese in one month?

That wasn't what I was going for but, why not?

You mean each women all producing parts of the baby one then one women/machine putting all the parts together?

In the future, because of bad things we did to the atmosphere and stuff, miscarriages are the common ending of a pregnancy, instead of being the exception. One evil company finds a way to accelerate the process by using 10 women, lowering the chances of miscarriage simply by reducing the time in womb.

I thought, something like a medication helping the women develop the baby in 30 days, but seriously damages the host in a couple of days, so the fetus gets transferred from host to host. Some die, but the evil rich don't care. Your idea could also work though.

I realize that this is totally the wrong corner of the internet to post this but I guess today I'll just go with it.

Yeah, that plot might actually work and is more believable than mine. (some women actually do feel like a baby is a parasite growing inside them - this would be a extreme version of it)

As a prediction, it doesn't sound particularly likely.

As a premise for science fiction, it sounds like it might be interesting.

I'm too bad of a writer to be scared of giving my ideas away, but I'd explore a pupae stage, where (layman disclaimer, if it isn't abundantly clear from my incoming butchering of the science) the butterfly is basically melted down into a goop with no "brain part" or "feet part" of the goop existing until reformed, yet retain their memories. If you divided that goop up and imagined there was a gestation time reduction, before combining it for the final stage of formation, you'd have some "plausible" (loose definition here) venue for it.

Interestingly, real butterflies don't turn into goop, that's more of an 'urban legend'. Nature's more interesting than that.

Thanks for the clarification. I'm very happy with my disclaimer then.

Don't leave it on a cliffhanger, what really happens?

It is a goop. They basically eat themself, except for some predefined core parts, which will then use the rest to grow and connect.

Eh, once you have the pointer, you can just Google for the details (or ask Claude, if you ain't so old old school.)

Dont say this too loud... :-X

If humanity lives long enough, someonen will came up with this idea and implement it?

The AI can't make a reliable judgement, so now you have to hire 3x to 5x more to make those judgements if you have the documents coming in at 3x to 5x faster. Or you'll have one severely overworked lawyer. It's the same problem developers have with thousands of AI generated PRs. Just because you speed up one thing doesn't mean the whole system works faster, and yes a human really does need to be in the loop or you end up with even bigger problems, and lawsuits.

No, that'd only be true the judgement was the full work. If the judgement was 33% of the work and the rest was just old plain e.g. typing, that is now fully automated and instantaneous, then you automate 2/3 of the work, getting 3x in productivity.

Why can't it make a judgement grounded in real data points - then the human can accept / deny that proof much quicker?

Surely there's a middle ground / this is faster than reading all papers front to back.

> They recently bought a Claude subscription and began using Claude to do the initial read of the documents and output JSON they can import into their internal systems. The work still must be reviewed by an attorney - Claude is nowhere near making the kinds of judgments a lawyer would make about this content - but it has increased their throughput from 2-3 documents an hour to 8-10 documents an hour by killing the busy work.

At my company we're also in the process of deploying a system that does exactly that. And what's interesting is that you absolutely do not need a frontier model for that, a small VLM (vision-language model), with optionally a little bit of fine-tuning, gives you the same output quality at a fraction of the latency and cost.

This is really where Gemini still excels. Honestly from my experience the others aren't as good.

Yes, we used Gemini before and it worked fine in terms of output quality.

But between 10 and 20 seconds to analyze a document with only a bunch of pages just isn't good UX.

Which Gemini? 3.8 Flash is fast and very nearly as good as Pro.

So which one do you use now?

And what are you using now?

I'm wondering if the bottom is going to fall out of frontier models when people start to realize this. Sure, as an experienced programmer you can tell the difference between a compact free model and the ones that are 100x bigger and cost billions to train, but 90% of busy work is dead simple: turn a PDF into json fields, or do the inverse and fill out a form. As the tools for this conversion become more widespread you can also imagine an economic shift toward efficient information brokers that make frontier models less necessary.

It's also not really clear if everyone is going to want a frontier model when the real implications sink in. Maybe we'll get sick of incomprehensible code optimizations and people wile tire of reading AI prose that feels ever-more-human. There might be a few use cases, but who is going to pay for this when providers start charging enough to be profitable.

I think this is the long-term reality of LLM tech and one of the objectives of engineers implementing solutions, finding the right fit for the job - the right model and cost to achieve high enough accuracy at the lowest price.

But that's phase two, phase one is finding localized problems to solve using LLMs and productize them. I'm reminded of cloud tech, where phase one was changing software to run in the cloud, and phase two was optimizing costs.

Yep this is one of the best use-cases for a fine-tuned SLM/SVLM.

I’m curious how this increased throughput happens.

You’ve accurately stated that AI isn’t as rigorous as a trained attorney. Doesn’t that mean that every single datapoint must be confirmed by a human?

How is that quicker than just using a human to read the content and make the call? Data entry savings?

Yes, remember that these are effectively random PDFs in various different designs and formats, some of them not editable or even OCR'd.

It took a human attorney 20-30 minutes on average to manually copy-paste data from these PDFs into a spreadsheet (while also fixing any errors they found in the document and re-checking for quality).

Now, the AI copies everything into the spreadsheet in a small amount of time, and then the human reviews it. It takes maybe ~5-7 minutes to scroll to the appropriate pages in the document, read the lines vs the spreadsheet, and make corrections. So you've gone from 2-3 items an hour to ~8-10 items an hour.

Maybe you could pay someone to develop an OCR/ML application that could do this. But that project would never be profitable, even with the time savings. At the cost of a couple Claude subscriptions, it makes sense.

Does the human find enough bugs that they stay on guard, or just rubber stamp everything without really looking at it? It’s hard to stay vigilant when stuff looks plausible.

Complete speculation: you could instruct Claude to hide one random mistake in every document.

And Claude should write down the mistake in a sealed envelope, so it doesn't make into the database.

A review that doesn't find the mistake counts as invalid.

This is what bag scanners at airports do - the hit rate is so low and the job so boring the software projects fake contraband onto the imagery. Fail to spot the knuckledusters and expect a chat with the manager.

You could do this for code reviews. Have Claude include one subtle bug into each PR. If the reviewer does not spot it, then the PR is not mergable.

Yes. Or you want to have a potentially more than one flaw planted, so people don't get lazy after the first one.

Don't forget that humans have a not insignificant error rate when copy/pasting or copy/typing data.

And it's possible to run each document through the LLM pipeline multiple times, using different models and/or prompts each time, to check for errors and inconsistencies. That will take more time and cost more, but it can reduce the error and hallucination rate significantly.

Thanks for sharing the details. Does the attorney check that the AI copied the data accurately? Or is it just assumed to be correct?

Your experience mirrors my own. AI is great for parsing data that can take up a huge amount of time. My only concern is whether or not it’s done accurately. I wouldn’t use it for anything where mistakes cause serious consequences.

I'm doing some public court records processing for bankruptcy cases (interested mostly to seek out corruption in big national cases), and yes, the "variousness" of random PDFs is exactly the issue. Trying to get the cost for a whole case down to a minimum.

Sample is around 300 court dates, shy under 1k files.

At best I'm building a claude skills file.

> Maybe you could pay someone to develop an OCR/ML application that could do this. But that project would never be profitable, even with the time savings. At the cost of a couple Claude subscriptions, it makes sense.

A better use of these Claude subscription would be to develop the app (which it can pretty much do at that point) and you could iterate to make the workflow even more efficient than your current one.

Nobody working there has the requisite experience to do this in a reasonable amount of time. These are not particularly tech-savvy folks, Claude use aside.

> These are not particularly tech-savvy folks, Claude use aside.

The difference between a tech-savyy person, and a non-tech-savyy person has always been mostly in the later's head, but this is even more true now that we have pocket assistants who can answer pretty much all of our questions in a language tuned to our level of understanding.

Yes. And it might not even be worth it, as the AI agents gets cheaper and cheaper.

Keep in mind that the task is fixed, so as the frontier of AI advances, you can switch to a cheaper trailing edge system and still get the same or even better performance for this task.

You don’t need a trained attorney to schematize data. The LLMs are used to make the data easier to understand and manipulate.

They'll also hallucinate and change meaning in the process of extraction and "schematization"

Not necessarily. Depends how you use it.

"Write a python script that breaks down this PDF by X feature" would not hallucinate anything in the PDF. Certainly you could trivially double check that all text in the extracted JSON was in the text layer of the PDF.

How much experience do you have with LLMs exactly? It would be consistent with my experience if Claude stuck in a line of python that just emits a JSON literal with no justification, potentially buried in a large program where an untrained person might not notice it. I don't even trust them if the output consists of structured data paired with source images from the PDF, because I've experienced LLMs fabricating the source rectangles to match the output. I only use tools like this by asking for programs, because as you note LLMs are good at that, and the verification process consists of tool calls to legitimate PDF manipulation tools so I have some confidence everything is above board. Even then I only do this for hobbies, not anything that matters.

Lawyer here. I used to trust Claude as hallucinations are near non-existent now. However for large volume tasks such as due diligence exercises, they still happen. We also tried Legora's tabular review, there were also numerous halucinated provisions in our due diligence exercise.

Is it possible to catch those hallucinations using another LLM with a strong fact checker prompt with sources provided in output for human validation?

Junior associates hallucinate too...

And when they do, you can train them or fire them, and they learn not to do it.

LLMs change not a whit, and there's no one to take responsibility for the failure (and thus no way to fix it).

As the new variation on the old theme has it, "A computer can never be held accountable, and so very many people are trying to get them make management decisions."

You can’t train people to never make a mistake, particularly when doing highly repetitive work like this. You must build your systems to account for that regardless.

Yes, exactly. Humans are non-deterministic as well, just in different ways. A tired human can make all sorts of errors for example, regardless of how much training they've had.

For sure.

But they do learn and improve.

The models don't (yet).

The models improve in the sense that GPT 5.6 succeeds at things GPT 5.5 fails at.

It might be that the models have been improving in this sense faster than a human child improves.

> LLMs change not a whit, and there's no one to take responsibility for the failure (and thus no way to fix it).

LLM output is nondeterministic and humans take responsibility for the failure the same way they take responsibility of a photocopy is too dark.

> humans take responsibility for the failure the same way they take responsibility of a photocopy is too dark.

You mean, they notice it's too dark right after making it, change the settings, do it again, and give you the good copy?

Because yes, that's my experience of humans.

You can scold juniors and they will learn. You can't scold Claude.

Surely the rate of improvement in new LLM models is the equivalent mechanism?

You can scold Claude. Just doesn't make a difference.

... until you hit Claude's risable "model welfare" protection.

That's a recipe for disaster in my experience. I tried it (with Claude) on a simple tabular bank statement PDF, and it transposed two amounts, placinh each against the other's description. And the bot assured me the result was cotrect. The chance of a human checker catching such corruption is low.

Interesting. Did that PDF have a text layer or did you ask Claude to OCR it? If the latter I'm not surprised at all.

Text layer.

Doing similar-ish things with Claude, it's helpful to have something to ground it.

For instance, if you can say:

"Refer to the database schema in x.sql as your source of truth for the database structure we want to import into*. Do not invent data, tables or columns that do not exist. Carefully match all output against this database schema and do not create output that doesn't exist if it does not match the schema, simply skip it."

You will end up with a far better result in my experience.

Gotta treat it like a child.

> Gotta treat it like a child.

"Sorry for that, Your Honour, but we gave that case prep work to a child."

They do, sometimes. That's why the review still has to happen.

But now it's comparing already filled columns on a spreadsheet, not copy-pasting every single thing from an (often uncopyable) PDF.

> But now it's comparing already filled columns on a spreadsheet

... with a PDF, right?

> not copy-pasting every single thing from an (often uncopyable) PDF

Obviously the PDF is copyable, else your bot would not be copying it.

"Copyable" in this case means you can highlight and copy paste text instead of manually retyping it.

But it gets it right like 99% of the time so human attention can be put towards catching the 1%, not entering data from one table to another and then catching that human’s mistakes.

Reviewing something takes less time than producing it.

I...don't think that is universally true.

Few things are. It also doesn’t require standards. “Stamp this diff” culture is everywhere even before AI. A stamp is literally easier than anything else.

Whether that is useful measurement I suppose depends on the circumstances.

Yes, but empirically in this case, it seems to be true.

LLMs are still absolutely horrid at analyzing PDFs so the results they are getting must be chock full of errors..

Are they? I've had excellent success. The confusing part of this is that there are two types of PDF. The first is a "normal" digital PDF. The second is a scanned PDF. The first can essentially be read like a document. LLMs have no issues with this. It's the second kind of PDF where the constraint becomes the vision capability, and this is very impressive with Astra. I've had no issues with either. I imagine there could be issues with unusually dense and/or misaligned text on scanned PDFs, but I have not tested this.

The bottom line, though, is that PDF OCR is usually regarded as a solved problem. LLMs won't usually do the recognition itself. It will farm it out to established tools which are very good.

Well, I would argue about the first part. Even if they contain "native" text that can be extracted, in most cases their order will be messed up and it is often crucial for correct parsing.

So in many cases the visual way is the only one that works correctly, the textual one is just a shortcut that may be walkable in certain cases.

It depends on what you called solved.

If the goal is to only extract the unstructured text from the document, it is definitely solved. Extracting a more natural structure like paragraph separation, tables, header, footers (what is referred as document intelligence) is much more complicated and not fully solved, but I would say almost.

Yup, this.

It is actually one of my test cases for LLMs: take the weekly discount PDFs of all the big supermarkets and process each of them, creating a nice table per supermarkt, converting discounts like 1+1 and only listing discounts that are interesting value. I then share that with a bunch of people.

All models fail this, even the really expensive ones. Even with harness, examples and proper insistent instruction, they'll mix up items and their related discount, which category the item should be in, which page they are on, skipping over items etc.

As said above, you can OCR it, but at that point you're not processing a PDF, you're processing an image.

And yes, I know the underlying raw PDF data is messy, but that's why it's such a good test.

The more data in the PDF, the more nines you need in the OCR accuracy. Plenty of 5/S, O/0 and other issues exist at frequencies that cause problems.

Yes but this is more an artifact of how much PDF sucks as a file format than LLMs being crap.

Which models are you using.

All the models output mixed data (errors). The larger is the context, the bigger the probability to happen. And this is an inherent property of LLMs (the output is a statistical concatenation of text, the LLMs are an vectorial database filled statistically).

If you connect an LLM to pilot a plane with passengers, prepare the obituaries, because would happen sooner or latter, an undesired index got mixed and produced a mixed/wrong output. But here, the bigger mistake would be to let an LLM execute commands without human revision, aimed by the deceptive marketing around the A"I" (LLMs).

they were in 2023

I'm a lawyer myself.

I see problems in LLMs doing research, making drafts, etc.

I see 3 kinds of potential users.

1. Non legal background people trying to avoid going to a lawyer. They should definitely NOT use an LLMs.

2. Fresh out of law school graduates.

They think they can do fast work, draft quick replies, do fast case research, enh. They should go the traditional route of interning, learning the "art", they probably won't need it because they will learn "how the law works"

3. Established attorneys. They usually have templates made for most things, over their career, they have established routines of making things fast, maybe it can help them but to say replacing good chunk of work or offloading to LLMs isn't probably going to go well.

There is a reason why interns are made to do grunt work. The "chain of command" is built so they learn on basic stuff, learn to make decisions, etc. Without these basic decisions under their belt, an intern can't take bigger decisions later on. They just won't know....

>Non legal background people trying to avoid going to a lawyer. They should definitely NOT use an LLMs.

I used LLMs to replace a lawyer this year to great effect. I successfully advised myself as to the right strategy and drafted an immigration petition in the correct language. Granted it wasn't 100% LLMs (I also supplemented it with "traditional" online research and wrote the letter myself using the LLM version as rough guidance), but still, I certainly would've gone to a real lawyer in 2024 for this stuff but didn't.

Professional advisory work can be divided into two types:

1. Once you know the rules of the game, the next move is immediately obvious.

2. Knowing the rules is not enough to know the next move -- that requires judgement and experience.

LLMs in the hands of someone smart and savvy can easily replace the first type of work.

If it is at all possible to replace a lawyer with an LLM, it should always be the preferred choice. Using a lawyer is not a zero-sum game, using an LLM is.

Lawyers nearly universally tend to: convince clients they need a lawyer; promote extreme views; charge legal fees (that ultimately result in financial damages to the client). There are no winners in this game, except for lawyers, who are having a lovely time. Even if you won the case, you've lost, because the pie got smaller due to lawyer fees.

You’re managing risk and what you pay for is derisking yourself. You be the judge of what that is worth to you, but it’s not zero and it’s not always your attorney’s fee either.

Retaining an attorney is a little bit like feeding a shark in order to catch a ride on its back and do some jousting with your opponent, while your opponent is entertaining a possibly bigger shark or two, and has possibly infested the place with piranhas.

There is a chance of de-risking yourself, but the chance of a bloodbath is even higher.

And yeah, the sharks are trying to convince you that they are "your" sharks, while it is quite clear that their mates are the other sharks.

Is that based on personal experience, or just anecdata?

BS. Last time I hired a lawyer he advised me on settling rather than going to court, even though he'd have made much more money going to court.

Might be true today, but I still feel pareto principle applies. These points also sound like gatekeeping a bit and I hope future AI versions prove that law is also essentially something that can be made as programmable as possible for most common answers if the underlying principles and constitution are organised such.

If there is no precedence for X happening when Y , Z, P, Q, R are special combinations , the judge decides and sets precedence. But if another judge interprets it differently with similar set of circumstances it is not considered incorrect I presume especially if in different jurisdiction unless overridden by a higher court. This is my layman understanding but I think this design is wrong and essentially where human subjectivity controls the system.

Subjectivity is not totally avoidable in law but the goal of the legal system should be reduce it as much as possible along with ambiguity.

Your understanding is correct, but where your understanding goes astray is thinking that that's a bug. The subjectivity is a feature. It is not possible to specify what should legally happen for all possible circumstances the entirety of humankind faces. Like, literally not possible. We humans can't even create rigorous specifications for what our programs should do that keep up with their evolving requirements, and you want to make life-ruining decisions set in stone based on a rigorous set of predetermined rules about all possible circumstances the entirety of humanity could face any time in the future in an evolving world?

Law is intentionally not code. The world is not something you can program.

Isn't achieving a rigorous, unique specificarion what civil law tries to achieve? Not that it is a success, but the idea behind it. Sorry if this is a silly question, but this is how I understand it.

No, it doesn't. It aims to provide a fairly comprehensive framework within which judgments can be made, but it is absolutely expected that there are edge cases and ambiguities which are resolved by human judgment of the circumstances.

Since this is a thread about using ChatGPT for law, I should note that this is actually one of the most important reasons you'd rather hire a lawyer than ChatGPT to represent you. A good lawyer or firm will know the local judges and how they tend to lean on ambiguities in their area of expertise, and will be able to use that information to both advise you and tailor their arguments to appeal to the judge's sensibilities. There is an element of unfairness to this, to be sure, but you'd find much more unfairness really quickly with machine-generated judgments because of how messy and non-compressable real world circumstances are.

I understand, thank you so much for the detailed answer.

Write me the code that says how the Lindsay Clancy case should have come out.

if murder: verdict = guilty

Now define murder!

> Non legal background people trying to avoid going to a lawyer. They should definitely NOT use an LLMs.

In my experience, it is incredibly hard, practically impossible, to find a lawyer who will even talk to you. I think this is a valid use case to get at least some understanding what the next steps might be and if it even makes sense to seek legal representation.

Agree. Now imagine the same thing in less developed and/or less well off countries.

There was a post here a few weeks ago about a UK court being inundated with some kind of claims (unfair dismissal?). To submit a claim you do not need a lawyer, but the process is (intentionally?) vague and very complicated, so it takes a lot of effort to figure it out yourself. But AI makes that kind of stuff easy.

AI is not going to replace layers billing big corporate clients millions a month, but it will replace things that are fairly standardized processes, that don't require a lawyer, but today are just too complicated for regular people.

> To submit a claim you do not need a lawyer, but the process is (intentionally?) vague and very complicated, so it takes a lot of effort to figure it out yourself.

I'm not sure there's quite accurate. The form is really not that difficult to complete.

It's basically: Party details, type of claim, particulars, damages

Obviously knowing these details requires some legal knowledge but I actually think ET claims are easy enough to bring if you've got the time to do a little research and build your case. Almost all ET judgments are public so there isn't the usual difficulty with research as you get with law reports being restricted either.

https://assets.publishing.service.gov.uk/media/65bcbd214a666...

> AI is not going to replace layers billing big corporate clients millions a month, [...]

On the margin, AI will drop these guys' billable hours.

> In my experience, it is incredibly hard, practically impossible, to find a lawyer who will even talk to you.

Really? I find them quite talkative. But I'm not sure what kinds of questions you are pestering them with?

In any case, the whole discussion reminds me of people using LLM to replace medical professionals.

For both: unless you have a doctor or a lawyer always on standby, you have to make a decision on whether to even go and seek out one of these professionals. By definition, you make that decisions without professional help. I think LLMs can help you there with the initial research to decide whether it's even worth it to contact the professionals.

I do think non legal background folks can use LLMs today to sense check legal ideas, like for instance, "what are my legal rights in this situation?"

These situations usually are not ones that an individual can justify the time or money to contact an actual lawyer, but then if they do decide to contact one they will come in with better questions and more of a sense of what they are expecting.

This is similar to medical. Should you use LLM to diagnose yourself, treat yourself with prescription drugs you buy from shady gray market online sellers? No. But you can use it very well to know when it's time to go to the doctor and what to ask.

Yes. Unless you have a lawyer or doctor on standby, you have to make the decision whether to contact them. And by definition you have to make that choice without professional help.

LLMs can help with that. I don't think they are worse at this than me trying to figure this out all by myself.

You sound much like software engineers sounded when the first coding agents dropped.

"You can't trust the output, it doesn't understand bigger systems"

"Its an art, you need to learn the ropes of it to truly write good code"

Its a very dangerous line of thinking. Software engineering will never be the same, as writing code has basically vanished from the daily workflow. Not for every specialized usecase, but for many.

You can replace "laywer" with "software engineer" in your post and it holds equally true. I think this also applies to other factions.

> There is a reason why interns are made to do grunt work. The "chain of command" is built so they learn on basic stuff, learn to make decisions, etc. Without these basic decisions under their belt, an intern can't take bigger decisions later on. They just won't know....

That might be true, but that doesn't mean you benefit from training up juniors.

There are 3x as many lawyers per capita in the US as we had in the 1970s. Credit agreements that used to be 50 pages are now 500. This increase in volume has not lead to any increase in actual value. The legal profession is a metastatic cancer on our society. Un-fuck this situation and maybe I'll think about taking your advice of not using a LLM.

Another thing that happened in 1971! https://wtfhappenedin1971.com

That's website is a great showcase of how to lie with statistics.

That very first image uses a different inflation indices for each of the two lines in the graph.

They also only post pictures without linking to sources.

Once again, LLMs comes to solve a problem that should not be there in the first place.

>I see 3 kinds of potential users.

3 kinds of users that don't scare the shit out of you.

My buddy works with lots of folks that have gone all in on Harvey. There are other kids of users.

As someone who has privacy concerns this is horrifying to me.

Both companies have settled lawsuits for illegal use of data. I would be extremely upset to learn that anybody is passing my private information to these companies for any reason.

I don't think the details of the company's benefits plan is considered your private information. But I am concerned about errors in this parsing leading to incorrect results (and that compounding everywhere throughout society when this sort of workflow becomes ubiquitous).

These documents are not personal private information. They might be client privileged in some cases? But I don't think so.

Happy to see measured results that isn't breathless glazing, and I suspect this is the kind of thing happening everywhere all at once. The other thing to consider is "burnout", in quotes because I mean that the work you describe sounds kinda boring, so people would resent doing it after a while and productivity would slow right down. Automation is the solution to tedium / toil, has been long before AI.

Yeah, it's sort of odd that even here people are pushing back like it couldn't possibly be doing this, you must have misunderstood it or are overhyping it.

I come neither to praise AI nor to bury it. This is a case where the tool is useful.

The key is that this is work the attornies themselves do not want to do and which does not require their brains or expertise. It's not just "boring" but a bad use of their time.

"Sounds kinda boring" - many do say this about lawyers.

this is one of the largest tasks of any professional services support firm like big law, audit, tax. if you can automate this monkey work you have to cut teams at least in half to stay competitive (probably more).

yeah, i think this is probably the more interesting use case. i do a bunch of boring data wrangling as a solo dev and the amount of time that disappears into "read this thing, find these 5 bits, put them somewhere else" is ridiculous. if you're going from 2-3 docs/hr to 8-10, even with someone checking the result, that's a pretty substantial improvement. i don't really care whether we call that decision making or not, i'd be happy to never do the copy/paste part again.

How do these use cases stack up (for real) in legal AI tools like Legora and Harvey?

Disclaimer: I used to work in legaltech, but not those two companies.

Wait what, the conversion is so lossy you use an LLM to analyse it? Or you get the claude analysis into JSON and put that into the system too? I'm not seeing what's causing the 2-3x efficiency bump.

They'd be better off outsourcing to lawyers in another country.

My dog can review documents at an even faster rate.

You havent given any proofs or even comments that the work is the same level of quality or accuracy.

I'm thinking back to some of my interactions with corporate lawyers and you know what? My bar (<- pun) for trusting Astra more than a lawyer is pretty low.

Visit a developing or corrupt country and you will realise the LLMs of today analyse better than the judges from purely precedence and literature review POV

The proof would be that the attorney who did the work before, and who still reviews all ingested data, says it is.

Exactly.

The statements in the post are opinions, there is no actual PROOF they are true, and thats my point.

You should free your dog then because a creature that intelligent should not be anyone's pet.

And free the biglaw junior associates while we're at it!

They're lawyers. They don't deserve freedom.

Reply on news.ycombinator.com