Hacker News Reader

Comment by pgroves | Hacker News Reader

This is sort of why I think software development might be the only real application of LLMs outside of entertainment. We can build ourselves tight little feedback loops that other domains can't. I somewhat frequently agree on a plan with an LLM and a few minutes or hours later find out it doesn't work and then the LLM is like "that's why we shouldn't have done it like that!". Imagine building a house from scratch and finding out that it was using some american websites to spec out your electric system and not noticing the problem until you're installing your candadian dishwasher.

View on HN

> Imagine building a house from scratch

Thats why those Engineering fields have strict rules, often require formal education and someone can even end up in prison if screws up badly enough.

Software is so much easier and safer, till very recently anonymous engineering was the norm and people are very annoyed with Apple pushing for signing off the resulting product.

Highly paid software engineers across the board must have been an anomaly that is ending now. Maybe in the future only those who code actually novel solutions or high risk software will be paid very well - just like engineers in the other fields.

> people are very annoyed with Apple pushing for signing off the resulting product.

Apple is very much welcome to push for signing off of software that appears on their own store. That is nothing new.

What people are annoyed about is Apple insisting that you can only use their store, a restriction that has nothing to do with safety or quality and everything to do with the stupendous amounts of money they make from it.

It's literally the case of Apple requiring signing the binary to run on the platforms they provide, Apple doesn't have say on other platforms. It is a very similar situation with local governments.

Also, people complain all the time about rules and regulations for making stuff. Especially in EU, you can't just create products however you like and let people decide if it is safe to use, you are required to make your products to meet certain criteria and avoid use certain chemicals and methods, you are required to certify certain things and you can't be anonymous. If you are making and selling cupcakes for example and if something goes wrong you will be held responsible. Not only when things go wrong, often local governments will do inspections before letting you start making the cupcakes and every now and then they can check you out.

Software appears to be headed to that direction. Of course du to the nature of software probably wouldn't be exactly like that but IMHO it is very likely that at least having someone responsible for the things a software does will become the norm.

Maybe in the future if your software leaks sensitive information for example, you may end up being investigated and fined if not following best practices that can be determined by some institute etc.

...but the EU is one of the entities forcing Apple to allow other stores.

It turns out that Apple is not, in fact, the government.

That's not a very compelling counterexample, when you consider how often countries with governments force other countries with government to do as they want, often with nothing but economic or soft power.

> Maybe in the future if your software leaks sensitive information for example, you may end up being investigated and fined

This is already the case in the UK, and the EU too as far as I’m aware.

> Apple is very much welcome to push for signing off of software that appears on their own store.

Just to be clear, apps have to be notarized/signed to run on an Apple device. For macOS, notorized apps aren't required to be distributed in the App Store. Due to sandbox restrictions, some dev tools are distributed independently.

Or there are two versions: a less capable version for the App Store and a more capable version distributed independently.

Software developers being paid well is result of demand, not be cause it's very hard.

Skill and strictness required is only vaguely related to pay, if there is enough people for the job it won't pay amazing, regardless on how hard it is.

> Software is so much easier and safer, till very recently anonymous engineering was the norm and people are very annoyed with Apple pushing for signing off the resulting product.

that has nothing to do with engineering quality, that is just to make it harder to go around their ecosystem (and skip paying the shop fee). With additional benefit of signed package being harder to attack. You can still deliver absolute slop, but the slop will be from you, not the middleman that captured the delivery process

I don't understand why the experience you describe would lead you to conclude that LLMs might be useful for software development.

The response "that's why we shouldn't have done it like that!" sounds like a variation on the usual "You're absolutely right! I apologize for any confusion". Why would we want to get stuck in a loop where an AI produces loads of absolute nonsense for us to painstakingly debug and debunk, after which the AI switches track to some different nonsense, which we again have debug and debunk, and so on. That doesn't sound like a good loop.

> This is sort of why I think software development might be the only real application of LLMs outside of entertainment.

Wow. What about also, I don't know, self-teaching*? In general, you have to be very arrogant to say that you've experienced all the "real" applications.

* - For instance, today and yesterday, I've been using LLMs to teach myself about RLC circuits and "inerters".

I would absolutely not trust an LLM to teach me anything alone. I've had it introduce ideas I hadn't heard about which I looked up from actual sources to confirm it was a valid solution. Daily usage has shown it will happily lead you down the wrong path and usually the only way to know that it is the wrong path, is if you already knew what the solution should be.

LLMs MAY be a version of office hours or asking the TA, if you only have the book and no actual teacher. I have seen nothing that convinces me they are anything more than the latest version of the hammer in our toolbox. Not every problem is a nail.

> LLMs MAY be a version of office hours or asking the TA

In my experience, most TA's are not great at explaining things to students. They were often the best student in their class, and they can't relate to students who don't grasp things as easily as they do--"this organic chemistry problem set is so easy; I don't know why you're not getting it."

But an LLM has infinite patience and can explain concepts in a variety of ways, in different languages and at different levels. Bilingual students that speak English just fine, but they often think and reason in their native language in their mind. Not a problem for an LLM.

A teacher in an urban school system with 30 students, 20 of which need customized lesson plans due to neurological divergence can use LLMs to create these lesson plans.

Sometimes you need things explained to you like you're five years old and sometimes you need things explained to you as an expert.

On deeper topics, LLMs give their references, so a student can and should confirm what the LLM is telling them.

Self-teaching pretty much doesn't work. For many decades now, the barrier has not been access to information, it's been the "self" part. Turns out most people need regimen, accountablity, strictness, which AI just doesn't solve because it's yes-men.

> Self-teaching pretty much doesn't work. For many decades now, the barrier has not been access to information, it's been the "self" part.

That’s a complete bogus. And LLMs are yes men by default, nothing stops you from overriding initial setting.

They're fundamentally trained to agree and don't do well when they require challenging ideas they're not "confident" about

It's not bogus at all. We've had access to 100,000x more information than we know what to do with for a while now. Right now, you can go online and learn disciplines you've never even heard of before.

So why arent you a master of, I don't know, reupholstery? Because the barrier isn't information, it's you. You're the bottle neck, we all are, because we're humans.

And AI really just does not help here. It's the same problem with professor Google, I can just turn off the computer, and I will. This is how it is for the vast majority of people.

Most people who claim to be self taught aren't even self taught. They did a course or multiple courses. Sure, it's not traditional college, but thats not self taught.

I think LLMs lower the barrier for learning certain topics. For instance, without them I wouldn't have tried to learn about RLC circuits at all.

Why would you think that a machine known to cheerfully and confidently assert complete bullshit is suitable to learn from?

Because you can independently check anything it tells you. You understand there can be independent sources of validation?

Why not just search out the independent sources and ditch the middleman?

Because verifying something is easier than finding it in the first place. It's in some way the difference between P and NP.

It’s somewhat delusional and potentially dangerous to assume that chatting with an LLM about a specific topic is self-teaching beyond the most surface-level understanding of a topic. No doubt you can learn some true things, but you’ll also learn some blatant falsehoods and a lot of incorrect theory. And you won’t know which is which.

One of the most important factors in actually learning something is humility. Unfortunately, LLM chatbots are designed to discourage this in their users. So many people think they’re experts because they asked a chatbot. They aren’t.

> It’s somewhat delusional and potentially dangerous to assume that chatting with an LLM about a specific topic is self-teaching beyond the most surface-level understanding of a topic

It's delusional and very arrogant of you to confidently asserts anything without proof: A topic like RLC circuits has got a body of rigorous theorems and proofs underlying it*, and nothing stops you from piecing it together using an LLM.

* - See "Positive-Real Functions", "Schwarz-Pick Theorem", "Schur Class". These are things I've been mulling over.

I think everything you said was true 1-2 years ago. But the current LLMs are very good about citing work, and hallucinations are exceedingly rare. Gemini for example frequently directs you to a website or video that backs up it's answer.

Thinking about this some more, maybe I wasn't considering simulators (aka digital twins), which are supposed to be able to create fairly reliable feedback loops without building things in reality. Eg will this plane design be able to take off? Still, I feel fortunate I only have to write unit tests to get a bit of contact with reality.

Simulations in general are pretty flawed, and AIs will usually find ways to "cheat" the simulation.

It's a very useful tool of course, but not as good as the software situation.

You can install a Canadian one just fine; authorities might not like it in some jurisdictions though but it's safe and might even be code.

I literally just had this exact argument; biggest issue is they can be tested for smaller amperages but you just downside the breaker.

It's more like you're installing the dishwasher and the dishwasher itself yells at you "I told you so" ;)

I think of it as you say "install dishwasher" and it plan looks like all the steps but as it builds it out it somehow you end up hiring a maid and buying a drying rack.

Reply on news.ycombinator.com