Hacker News Reader

Comment by tracerbulletx | Hacker News Reader

There's a tremendous amount of useful things you can do with Transformers and diffusion models. It's kind of fascinating how businesses are not able to turn those straight forward things into businesses and instead think that they should run commercials insisting people will want to have an AI write heartfelt thankyou letters or do creative tasks for them. Whoever is driving these initiatives is so lacking in aesthetic and cultural awareness it's insane.

View on HN

I wonder if it's because it's hard to do most things with LLMs in a reliable way when not supervised.

As programmers we complain about the ~1% from copilot-type models where the code is terrible. It is annoying but you can live with it.

A 1% error rate for many things, and with no bounds of how terrible the hallucinated error is, perhaps that's unworkable.

For example, Air Canada ended up liable for a discount which its AI chatbot made up.

Just thinking of it. 1% error rate, say 1 in 100 customers gets some wrong information. And they go to place trusting it. Just to hear that AI lied to them. Say you have 1000 or 10000 customers using system, now you potentially have 10 or 100 one star negative reviews... And this might be just answering to simple queries like a restaurant menu or opening time.

No decently coded chatbot is going to respond with an incorrect restaurant menu or opening time. You'd call a function to return the menu from a database or the opening time from a database. At worst, the function fails, but it's not going to hallucinate dishes.

Exactly this. Even for internal use. Our corp approved a small project where NN will do the analysis of nightly test runs (our test suite runs very long). For now it does classification of results into several exiting broad categories. Technically product type failures are the most important usually and this should allow to focus efforts on them. But since even 1% false rate (it is actually in double digits in real life) would mean that we, QAs, need to verify all results anyway. So no time saved, and this NN software is eh... useless.

There are other ideas how to make it more useful, but my point is that non-zero failure rate with unpredictable answers is not applicable to many domains.

Yeah, a 1% error rate (IME in practice the error rate is _much_ higher than this if you care about detail, but whatever) just won't fly in most use cases. You're really talking about stuff which doesn't matter at all (people are rarely willing to pay very much for this) or where its output is guaranteed to be reviewed by a human expert (at which point, in many cases, well, why bother).

indeed, reviewing notoriously takes just as much effort as producing the artifact under review!

Curious—what useful things are you looking to work on right now? I'd love to learn more and help out.

In my opinion, this AI development stalemate is more layered. Big companies set such broad targets in a race to catch up with OpenAI that they lose focus on real use cases. So the loudest voices, those good at navigating internal politics, end up in a good spot to push their own ambition over actual customer needs or technical practicality. They set goals that sound just a bit more exciting than their peers, which pulls resources their way. But the focus shifts to chasing KPI's rather than drilling into real problems. Even when they know going smaller is smarter, knowing and doing are two different things.

It’s still a great time for small AI startups. My favorite kind is a team that quickly learn a business’s needs, and iterate toward the right interaction points to help. I think just staying focused on solving a lot of small related problems very fast, you can create something that feels like a real solution.

If I were Google, I would offer small businesses built-in chatbots for their Maps listings.

If I were a small business, I would not be excited about having a magic robot 'hallucinate' things about my business. Like, this is the company that brought you glue on pizza.

I think the main issue is that the average consumer does not know what to do with a raw transformer model.

While the base technology is now there and is rapidly improving, a lot of the "glue" and "plumbing" is still missing. What is the best way to integrate these tools into our normal workflows / daily lives and so on?

It will take time...

Articles like the one above are not very useful as they completely miss the big picture.

Reply on news.ycombinator.com