Hacker News Reader

Comment by mk_stjames | Hacker News Reader

This is the same answer format of essentially any deep technical question I've given it.

Essentially:

1. Do the thing that is the first thing that is mentioned and that any first year student learning about thing hears about the thing.

2. Do some more complicated things in a hand-wavey manner that solves the problem.

3. Check to see the problem is solved. If it is not, do some more of the hand waving in step 2.

4. There may be other problems to check for in the future even if this solution works. Who knows! But I solved your problem.

View on HN

Sure, but because it's a chat bot, you can then ask: "Cool, so how would you do the thing you just told me to do?"

I just pasted the manifest and the main script and asked it how to port it. It was basically a one-shot. Here's the port: https://github.com/unflxw/chat-gpt-google-extension

The README includes the conversation I had in order to port it. Close to zero thinking necessary. I have never worked on a browser extension before.

I'd really love to see a shootout between two people who weren't familiar with extension programming (but who have some programming knowledge) be given that task (port this github repo for a chrome extension to work and be installable on Firefox) with one person just using normal google-fu and the other person doing what you did, and watch how long it takes them.

I actually wrote a few VS Code extensions recently starting from zero knowledge of the package formatting for them, and I wonder how much different my time spent doing it would have been if I would have just asked this bot to help. There was a lot of regex involved and hell just using chatGPT to explain to me some of the ways I could be matching strings with whatever format of regex tmLanguage uses would have probably saved me a lot of googling.

This is super cool. Curious how much work it was to discern the `[blabbery omitted]` sections. Were you generally able to tell right away that they were nonsense? Or did you have to spend significant time following dead ends it recommended?

Most of the "blabbery omitted" bits are just one-paragraph summaries of the rest of the answer above. It wasn't nonsense, it was just repetitive fluff, and the README was already way too long.

ChatGPT likes mimicking its own previous style, so after it ended its first message with a paragraph like "Overall, porting a Firefox extension to Chrome is doable, but it requires knowledge of the differences...", it kept closing its following messages with similar ending summaries.

The one bit where I refer to it as "nonsensical modification suggestions" was where it suggested that, in addition to changing `chrome.runtime.connect()` to `browser.runtime.connect()`, it told me that:

- I needed to use `document.querySelector`, because Firefox does not support `document.getElementByID` (very clearly not true if you're familiar with DOM APIs)

- I needed to change `classList.add` to `classList.add`, because Chrome supports `classList.add`, but Firefox supports `classList.add` instead (yes those are the exact same thing)

It did make me think for a moment: "what, really, Firefox does not support `getElementByID`? wait, no, of course not, the bot's just saying words."

In this case, it did not get me stuck debugging anything. In other conversations, where I've tried to "play dumb" and follow exactly what it tells me to do, it has sent me down the wrong path. But if you keep telling it "no, that's not working, how can I solve <original problem>", or you give it additional information, it usually corrects course.

I've also seen it say, "well, I don't know what's going on, but the problem must be around this line of code, insert a console log here to see what's going on with this specific value", which was actually good debugging advice, and then you can paste the output of the console log to it, and it takes it into account.

The concerns I have about this is definitely being fed potentially outdated information. DOM APIs change, and Firefox may not have supported that at some point, so here we have some suggestion that's a blatantly wrong now. Same goes for any suggestions that are against best practice now, or class components or function components in React, etc.

(These are just examples of things, surely it would suggest FCs for React at this point)

Is it bad I think this is a decent high level algorithm? As in, the user then filling in details or asking the assisting model to fill them in via subsequent prompts seems like a solid start to a problem solving session.

The problem is the part that I denoted as "hand waving". If you are asking it things about coding or other details it seems pretty good at, it can get pretty deep into it and thus that 'hand waving' can be resolved.

If you are asking about aerospace engineering concepts at the level of a PhD thesis... it just hand-waves away problems in ways that, when pressed, it cannot give detail on.

So, as far an a algorithm goes, the problem becomes determining the computational complexity of the 'hand waving' part. For writing reactJS frontend APIs it seems like it's pretty damn low and thus the AI can spit out code and even fix bugs for you. For developing something that actually took a human 7+ years of undergrad, graduate, and PhD work to ascertain and work out.... no chance.

But yes that general algorithm works for all walks of life essentially and I keep getting a format like that for everything setup I ask about, as if it was trained on 'How to answer a hard technical question in these easy steps...' and it is sticking to that.

I'm not well versed in this but IIUC the less context/data that exists, the less it should afford the creation of a succinct response, right? This would make sense for the gap between "frontend js api" responses, and "aerospace thesis" responses.

Would it be safe to assume the limiting factor with this or any algo of this type might always be availability of context/data, and these niche questions might always be an edge case?

Are there alglrithms that can reliably extrapolate "new" data/context that does not exist yet and is accurate IRL?

Sorry if my terminology or understanding is way off, I don't know much about this field.

No I think that is pretty much it- it is deep learning after all, the scope of the training data limits the model. It's so good with code because not only is there a TON of code in the data set, there is a TON of data of tutorials and examples and documentation EXPLAINING the code! I've long been amazed at the computer programming world at the amount of content programmers write about... programming. When it comes to other engineering fields, people don't write nearly as much on the internet (and, ofter, are not allowed to write as much, publicly) about the topics. So of course these large language models are simply not going to be able to produce super detailed analysis of those fields.

There is something a bit more needed - Meta did a deep model recently with training specifically more on scientific papers, and it is able to 'talk the talk' better but unfortunately the 'understanding' just isn't there.

I think at the end of the day, programming languages are languages after all, thus work well with these tools, but there is something harder about scientific reasoning and interdisciplinary planning that is going to take something a bit more. This is the main goal of various research into bring symbolic methods back into interface with deep learning models.

Reply on news.ycombinator.com