Hacker News Reader

Comment by geraneum | Hacker News Reader

There is research [1] that suggests using GPS navigation could impact your brain function negatively. I wonder how this code generation “assistant” will impact the coding skills of its users in long term.

As for the LLMs, stochastic parroting (hallucination), which is inherent to its design, makes it untrustworthy for precision work. Maybe some other architecture will prove more useful. LLMs can generate BS authoritatively which is easy to detect for toy examples and forgivable by the AI hype train but can be very costly or hard to fix in a complex system with a lot of moving parts.

I’m actually optimistic about the future of AI. At the same time I believe we should know the capabilities and understand the limitations of our tools in order to use them more effectively. Unfortunately people are misunderstanding tools like ChatGPT, etc. for we are human and vulnerable to anthropomorphism.

1. https://www.nature.com/articles/ncomms14652

View on HN

I wonder how this code generation “assistant” will impact the coding skills of its users in long term.

My brain is already negatively impacted having to juggle 3 separate package managers, know the syntax and inputs for 17+ different dotfiles in a repo and then actively context switch between at least 3-4 languages on a daily basis. I counted it up recently and I've built products in at least 20 languages over the past two+ decades. At some point you really just can't effectively compartmentalize it all and context switch perfectly. I try by using different fonts+colors per lang etc but it still is hard. AI has been a net win for syntax recall just by giving it gentle nudges towards intent.

Why are you touching 17 different dotfiles on a regular basis to require knowing their syntax/inputs ?

I've done a similar number of platform jumping but I honestly don't remember when I've used more than 3 or 4 at the same time. Like I was really into Clojure for ~1 year back around ~2010 - if you put me on a clojure project I'd be lost for a month. I wouldn't need to get over LISP and functional programming mindset again but the details are lost.

At this point I rarely see big paradigm shifts and it's just picking up the details + connecting with things I already know. I've outsourced remembering details to Google :)

I didn’t mean just any negative impact like getting tired from context switching or feeling exhausted because the workload is high.

What I meant was: Will you lose it if you don’t use it?

Maybe you won’t, I am not certain. Or maybe you will and you don’t see it now because the “net win for syntax recall just by giving it gentle nudges towards intent” relies on your hard earned ability to understand the code which is gained partly through exercise. You’ve already done your exercise.

I wouldn’t form an opinion just yet. For now we can ask the questions.

Edit: wording of the first sentence for clarity

But losing it may free up resources to gain other things. Like how "losing" assembly skills allowed people to dedicate more capability to higher-level approaches.

Sure that could happen. Nothing against that. But I am not fully confident. As a counter example, we have ORMs. They bring value but I have seen enough devs who are too reliant on it and don’t develop a good understanding of how the underlying database behaves. This could be actually detrimental to the work.

Maybe we will have a mixed result similar to ORM example. It can lower the barriers to entry, which is good, but won’t eliminate the need for deep expertise. At least initially.

On the contrary, I'd argue all the switching probably kept you sharp

I sincerely doubt it, humans are historically poor at multitasking, being able to focus on solving problems at an architectural and algorithmic level requires concentration - not having to worry about some bull crap DevOps cloud formation nonsense.

>people misunderstand and anthropomorphize LLMs

I agree but there is a certain "god of the gaps" process that's happening here.

To me what's been achieved so far is encoding text into natural (and programming) languages. These models can't reason and e.g., can't pass a Turing test. For example, if you play with GPT-3 or ChatGPT enough you can definitely get it to fail on a task a human would succeed at.

When models so far have achieved "reasoning" behavior, usually it's in conjunction with a classical algorithm -- for example AlphaZero used a model to evaluate chess positions but then ran a tree search algorithm to decide what move to play.

I think if LLMs can be combined effectively with some kind of computation process like in AlphaZero or some kind of "chains of thought" [1], then we might see them do things we'd truly call "reasoning".

Some "gaps" have been (1) "(encoding into) images is hard" (2) "(encoding into) natural language is hard" and now (3) "'reasoning' is hard" and afterwards (4) "subjective experience is hard".

Whether to be optimistic or pessimistic about AI depends on the final "gap" -- whether there is an irreplicable/noncomputable/etc. property of human consciousness other than qualia. That is, if "understanding" is hard (e.g., via some voodoo like [2]). If not humans will become obsolete, so I hope the answer is yes.

[1]: https://ai.googleblog.com/2022/05/language-models-perform-re...

[2]: https://en.m.wikipedia.org/wiki/Orchestrated_objective_reduc...

> There is research [1] that suggests using GPS navigation could impact your brain function negatively.

Wrong, that research only shows that when you use a GPS, you don't learn routes as quickly. That is only equal to "impacts brain function negatively" in a narrow sense, which you should specify rather than use such a broad and scary-sounding phrase!

> … could impact your brain function negatively.

Brain function being negatively impacted is not scary on its own and not necessarily permanent.

I didn’t mean to scare you. It’s likely you are confusing the above phrase with “your brain will be altered forever” which is not what it means.

Who said I was scared? I'm just pointing out that the phrase you used it very vague and some interpretations can be scary. It pays to be specific, especially in tech and medicine, you know? Winkyface.

> As for the LLMs, stochastic parroting (hallucination), which is inherent to its design, makes it untrustworthy for precision work.

To be fair, this could be said of a surprising number of the developers I've interviewed too, and I checked -- they were hired by someone.

> for we are human and vulnerable to anthropomorphism.

Stop anthropomorphizing computers, they hate that!

Cupcakes and clouds feel them same. Stop excluding them!

> There is research [1] that suggests using GPS navigation could impact your brain function negatively. I wonder how this code generation “assistant” will impact the coding skills of its users in long term.

I wonder how code completion and syntax highlighting have altered the brains of programmers negatively as well.

Have you never seen a project where exactly one use of a term is mistyped across whole project? Guess what tool is responsible for that.

Also there's distinct project structure difference between users of tree style project navigation vs find-anywhere or editor tabs vs modal windows/buffers. I mean, the influence is not negative, at least not obviously, but the deformation is there.

Interesting research but I don't really see any reflections on "GPS navigation could impact your brain function negatively" there

Most programmers wish to be stupid. They willingly embrace philosophies predicated on their inability to be conscious and learn.

Goddamn do I hate this industry. I work hard to sow dissent in my blue-chip, VC startup though. Guerrilla tactics. And I get to do it raking in a quarter mil a year in my underwear.

It's a real struggle, the company I work for was just bought out by a megacorp (in the top 1,000 in the world by employee count, and most posters here would recognize their name) and the sheer lack of motivation to learn of many of the devs both at the acquired company and the acquiring company is staggering. Really basic stuff like "instead of package management let's pass DLLs around over email/Slack/wherever and then complain about all the time we have to spend debugging because of version mismatches." It's like they intentionally make things harder on themselves through willful ignorance.

I just tried ChatGPT today after weeks of hype on Twitter, and I was surprised to NOT be impressed?

First I tried it on a bunch of programming topics -- it definitely gets in the right ballpark, but then the details are frequently wrong.

I asked which languages have the F# |> operator, and it told me that OCaml, Elixir, Haskell and Scala do. 2 of them are correct, and 2 appeared wrong, so I googled it.

After Googling, then I asked it specifically if Haskell does, and it says no. I ask it if Scala does, and it apologizes for its earlier mistake.

It gave me a good although slightly shallow answer about how PyPy works, but then fell down hard for a prolog type checker, and a few other programming queries. It doesn't understand the question.

---

It is generally very mealy-mouthed and vague. It often doesn't answer the question directly, but just tells you what it happens to know in the ballpark.

I like another framing I saw on Twitter -- it's "mansplaining as a service". If you want someone to confidently explain things you didn't really ask about, and be frequently wrong, that's kinda what it does!!

---

After programming, I asked it about Black Sabbath, UFC, travel and biking, health, and got similarly vague/slightly incorrect answers.

It confabulates A LOT.

The best I can say is that Google is filled with spam now, so chatting with ChatGPT was a relatively fun and focused experience, after I took into account the fact that it is not "honest" and just trying to "please me" with closely related words.

I honestly don't get the hype, and I feel like there must have been some coordinated marketing aimed at me ...

If this chat is representative of what it can do for programming, then I think this article saying that "programmer salaries will be affected by LLM" is very weird.

It is assuming something that simply has NOT happened!

---

I mean even Geoff Hinton (who I respect) said several years ago that "we should stop training radiologists", and that plainly did not happen. Likewise with Elon and the fleet of self-driving taxis.

I'd honestly be surprised if someone wants to argue that a tech company staffed with LLMs + a few smart people is NOT like those predictions. Of course the blog post doesn't give a time frame, so it can't be falsified. But it appears to be nonsense.

(FWIW it doesn't matter to me personally, as I left the industry several years ago. Also FWIW I did work a little bit on deep learning, when the hype was in its upswing.)

I agree that a lot of it is anthropomorphism -- projecting some kind of intelligence, knowledge, or intention on what is really a big, expensive bag of data organized to respond to inputs.

Maybe there are other models that are better, but I was really, honestly not impressed with ChatGPT. Maybe people were hyping it too much for the last several weeks, and inflated my expectations.

This is not what it's good at at all. Not sure why people expect this to be it's forte.

It's like a really smart sheep dog or a junior junior carpenter. It's currently helping me refactor a .NET app to a trending pattern, not because it know how to do that exactly or which pattern is better, but because it doesn't tire out at the "boring" parts like I do, and it knows the difference (most of time) between IQueryable and ISomething that I just can't bother to google for the Nth time.

I asked ChatGPT if it wrote this ^ comment, and it claimed that yes, it did. For the record, I don't believe it.

expectations are interesting things... i was impressed with chatgpt, though not blown away as it's simply the latest in a chain of increasingly impressive research demos. substantially though my experience was the same as yours, though perhaps i'm more forgiving (haskell does have a reverse function application operator, just using a different symbol, and you can get |> in scala via a commonly-used library).

i guess one difference might be that i also tried a variety of tasks where confabulation was called for, like making up games and getting it to play them with me.

all of this certainly does seem farther from productizability than the optimists admit... it tantalizingly 'almost works' in so many domains, but despite the rate of progress, its not obvious at all to me that it will soon converging on the reliability which is necessary for robust application.

playing with the tech in more creative areas has certainly expanded my mind as to what is and what may become possible, and i think this is true of the general reaction as well... it's the flashes of strangeness and potential, the weird use cases like simulated command lines and ascii art games, the cat-and-mouse of prompt injection... not hard to let your imagination run wild... depending on how much disbelief you're willing to suspend.

Yes, I too have found it to be completely unreliable when it comes to specifics. However, I've been finding it really helpful in a few areas:

- taking point form, or messy notes, and converting it to full sentences/paragraphs. Useful for things like writing blurbs, longer emails, proposals. Naturally, it does veer towards cliche, so you kind of have to massage it back after.

- getting a bash script started in the right direction. I'm always fuzzy on bash syntax because I don't use it all the time. ChatGPT will usually screw up some details, but those can be fixed

- getting a general overview of a new topic area like a new software library, or say a historical period

That vibes with the images from what I saw with Dall-E. The rendering/styling was superb, however, the logic and scene understanding were toddler-like.

Still impressive, but if you had a picture with dogs sometimes a dog head would be pasted on top at the wrong angle. Confabulation is a great term for it IMHO.

That's basically right, AI of the type we're discussing has high technical ability like ability to draw and make grammatically correct sentences, without understanding the higher level structure of what it's doing. A bit like an enthusiastic teenager.

But I can't see how people aren't impressed, we couldn't do this a few years ago, and surely we will build on top of this in coming years.

I hate the AI hype. It ruins the fun. People always respond in the same way.

I tried it without reading anything about it and I freaked out about losing my job. Then I tried it for any practical programming and realised it was no immediate threat at least and I calmed down abit.

I mean "generate BS authoritatively" seems fairly similar to "fake it till you make it".. Some people writing code now generate BS authoritatively, some don't.

Reply on news.ycombinator.com