Hacker News Reader

Comment by cj | Hacker News Reader

I can see a world where "ML Engineer" (or similar) is someone that's hired to solve a known problem (whether it be with classifiers, LLMs, neural nets, etc), whereas a "AI Engineer" (or whatever the title) is hired to figure out how the hell to capitalize on the AI hype, without a specific problem to solve.

IMO right now we're entering the "Peak of Inflated Expectations" in Gartner's hype cycle model. https://en.wikipedia.org/wiki/Gartner_hype_cycle#/media/File...

Lots of companies want to jump on the AI bandwagon, but they don't really know what to do or how to leverage it.

What the LLM community needs now is for companies to leverage and productize these LLM models for truly game changing use cases. If that doesn't happen soon, the hype will start to fade and lose momentum ("trough of disillusionment").

I'd really love to see some killer use cases emerge soon.

View on HN

(author here) There are at least 4 killer apps ($100m/yr revenue potential) so far:

1. Generative Text for writing - Jasper AI going 0 to $75m ARR in 2 years

2. Generative Art for non-artists - Midjourney has by some accounts $80m ARR

3. Copilot for knowledge workers - GitHub’s Copilot has roughly 50-80m ARR as well

4. Conversational AI UX - ChatGPT probably has >$100m ARR by now, Bing Chat has brought $m's worth of attention to Bing

(More on agents as the immature #5: https://www.latent.space/p/agents)

what else do you need to see to believe? (geniune question)

The longer HN continues to doubt in AI's deliverables, the bigger our revenues and our moats can grow. Don't tell them.

(It's not that hard to hit $1M ARR with a good AI product. So many classes of new products and solutions have opened up.)

I’m sure I could find a similar comment about cryptocurrency or NFTs. There’s always snake oil salesmen.

I remember thinking this about Tesla too. They built a fairly massive automaker while everyone kept saying their whole approach was fundamentally doomed.

> what else do you need to see to believe? (geniune question)

It's not that I don't believe in LLMs.

It's that the level of media attention AI is getting isn't backed by the same level of real world use cases, yet. (All of the use cases you listed are awesome, no denying that, but I don't think those alone justify the amount of hype in mainstream media)

i mean, theres nothing i can do about mainstream media hype, but i guess my main point is this is a growing field with real money and utility behind it, and so will professionalize. if I am correct on that then AI Engineer will be a thing (because it is Least Bad title for the thing)

AI Engineers are just software engineers who use specific tools. If we want to use a fancy title then that's fine, I guess. But let's not pretend that a typical dev can't easily learn vector DBs, data pre-processing, fine tuning, etc.

None of these things require specialized knowledge in the way that say being an AI researcher would.

You could say the same about things like SREs or Devops. In fact, many of them transitioned from regular "software engineer" to these roles simply by learning closely related skills.

That won't stop the industry from inventing new, useful titles.

You are actively contributing to the hype cycle lol.

The tech problems will eventually get sorted out I'm sure, the most uncertain problems are legal. The world's copyright law is not even up to date enough to deal with the internet without crappy patchwork workarounds (like Youtube's contentID nonsense), much less LLMs.

Steam just banned AI art and text assets and built in generators because they don't want the liability of hosting it. Midjourney is getting sued by Getty, Copilot breaks GPL licenses and was even facing class action lawsuits, OpenAI trained using copyrighted and even pirated content, LLama models are unavailable for commercial use, AGI alarmists want to ban the whole thing altogether, etc.

A lot of end usage depends on how all of this plays out in courts over the next few years. Why would people dump capital into something that will be declared illegal?

Because if they don't, their competitors will, even if those competitors are overseas in a more lax environment.

Either IP laws will catch up, or the countries with better IP laws will become more competitively successful.

If people get used to LLMs governing their lives, those copyright laws will "just" be changed. People will not accept a legal challenge taking away their toys. Granted, we're not at that point yet. But the window for the legal stuff really making a difference is slowly closing. Meanwhile, the legal system is pretty slow.

75+80+80+100 is £335m per year in revenue. For a point of comparison, that's approximately 11 hours of Google's revenue.

but rate of growth is likely 100-1000 times higher

On a smaller time scale, is rate of growth that meaningful?

I guess I'm just thinking, until we see the how this all pans out over the next decade, we don't really know if the current rate of growth will hold, or if it'll plateau/slow down.

Yes, it’s meaningful. If Google doesn’t fix search in the next decade then Bing (or some other winner) will be the service earning Google’s ARR in 11 hours.

> On a smaller time scale, is rate of growth that meaningful?

revenue in initial stage of extreme growth is much less meaningful.

I find myself using Bing Chat more and more instead of googling specific questions. And no, hallucinations are not a problem, because the questions are concrete and the answers immediately verifiable.

The innovator's dilemma on display.

Consider the possibility that may be a point in a technology's evolution where you can "look at what is being done with it" and conclude that it's useful from that, rather than just comparing revenue figures across domains.

Yeah the dollar values here are a distraction. I use copilot everyday at my Data Science job. Its useful!

What we are not considering is integrating a custom trained LLM into our work because the tech just isn’t there yet.

LLMs today are great for certain use cases, which is amazing when it suits your needs. Need to extract a hotel name, city, address and confirmation number from emails and return it as JSON? Not a problem. Need it to validate postal codes in Canada? It's the wrong tool.

I'd add these use cases:

5. automated processing of unstructured paperwork and ingestion into ERP systems. Basically, upload any kind of bill and get all of the information into the system, not just "find out the total amount". That can save so much in accounting it's not even funny any more.

6. related to this, something that sorts incoming emails. Classify stuff into "look into it now" vs "look into it later" vs "yet another bullshit marketing email".

My problem with this is that AI at the moment is a kind of 80% thing.

And you want to plug that into ERPs and accounting?!?

We don't even really understand its failure modes.

Yea but humans manually retyping things is also a 80% thing.

The thing is it can be used to get rid of the tedious and error-prone busywork. Instead of having highly paid accountants manually type monetary amounts into SAP, the accountants can now just look over and check if the AI was accurate in transcription.

Why AI, though? Why not OCR and regular software? What does AI bring to this table?

Except for VC money.

OCR can't grasp context, it needs incoming bills to be formatted in exactly the same way or at least in somewhat consistent area patterns.

AI doesn't have that problem, so you can use it to detect areas of interest and then follow up on that with classic OCR.

I’ve been thinking the leading edge is already in the trough of disillusionment! At least on HN etc.

We see lots of reports of limitations on HN. Those in the know don’t trust it nearly as much as the public and all the CEO’s of businesses drooling to install AI and have the money go up. Going to have to dig our way through those issues to get real value.

Well you have to consider the inherent bias on HN, in that the trough of disillusionment is HN's default state of mind regarding literally every topic. It's really rare to see a thread where most of the comments aren't negative.

Ha, when GPT 4 launched the default HN post seemed to be saying LLM's were about to exceed human intelligence and the singularity was upon us.

The disillusionment seemed to happen pretty fast.

The right way to leverage AI is to use it to do ye olde engineering, faster. Actually, most AI use cases are in fact that! Ye olde engineering - data pipelines, automation. Stuff we've been doing since before I was born.

Everything else - transformative tech, will hit the open domain fairly quickly, is my bet. Similar to databases.

We are indeed at the "Peak of Inflated Expectations" in the Garter hype cycle. Everyone is screaming that we are out of the AI winter and throwing LLMs at every problem.

> What the LLM community needs now is for companies to leverage and productize these LLM models for truly game changing use cases.

The killer serious use-case has always been summarization of existing text. That's it. Everything else is a constant flow of creative bull-shitting from a black box AI requiring triple checking everything before using the output, meaning that the output can't be trusted.

> If that doesn't happen soon, the hype will start to fade and lose momentum ("trough of disillusionment").

I think they will realize that there is more to "AI" than LLMs, just like the hype with CNNs and the like.

Not just summarisation, information extraction in general. LLMs are great data normalisers.

No. No they are not. Do not keep proliferating this idea, as there are real consequences. Your input only informs the likelihood of the output. There is no model actually extrapolating rules from the information, so the premise of LLMs being extractive is 100% verifiably false. They summarize well due to being handed a roughly correct arrangement of tokens to mimic the order of. This should not be confused with extraction.

To put it simply, keep falling for that, and it'll bite you in the ass.

You are talking from a top-down point of view. Of course they can make an error any time, after all they stochastically sample the output. All approaches to information extraction are upper bounded at 90-95% accuracy, I have extensive experience with this task and run many evaluations on invoices, receipts, forms and other doctypes. Human in the loop is still required.

But in practice you can rely on good copying and excellent ability to parse names, addresses and various values as good or better than other approaches. LLMs reduce the number of corrections, and they are easier to deploy - instead of labeling hundreds of documents you can just query with the field name.

I'm talking about an "at scale" point of view. A 90-95% success rate would be at worst 9:1 odds and at best 19:1. That means your best case scenario would be a likely failure on every 20th inference. This may be passable on an adhoc individual basis, but why do this when a deterministic solution can achieve beyond 99%, with proper error handling? Data normalization tools are rigid as a feature - LLMs are not, even at a temp of 0

Thanks for the info z3c0. Have you implemented open source solutions? Do you mind sharing some deterministic solutions to detect entities without huge dictionaries (NER)? Or maybe extractive QA if you used that instead?

Deterministic? Not anything that is "one size fits all". If your documents are of an unpredictable shape, then ML is your best bet. For both NER and QA, BERT models are very capable.

To your first question, most of my ML work is proprietary, unfortunately. I am hoping to change that in the near future.

Edit: spaCy is a great library for NER, if you're hoping for an open-source solution that can have you hitting the ground quickly. SQuAD for QA.

Reply on news.ycombinator.com