Comment by afro88 | Hacker News Reader

Comment by afro88 | Hacker News Reader

The article shows everything that works for this approach. But it's a bit disingenuous. At the end:

> Once this is working, Xu Hao can repeat the process for the rest of the tasks in the master plan.

No, he can't. After that much back and forth and getting it to fix little things where it gives responses with the full code listing again, he would have easily hit the token limit (at least with any chat LLM capable of this quality code and conversation - ChatGPT). The LLM will start hallucinating the task list, the names of functions it wrote earlier etc. and the responses would get less and less useful with more and more "this doesn't work, can you fix X".

So anyone following this approach will hit a footgun after task 1.

For anyone that really wants to follow this approach, the next step is to start a new chat and copy/paste the inital requirement prompt, put the task list in there, any relevant code, adjust the instruction (ie "help me with task 2") and go from there.

It is of limited utility though. By step 3 (or even 2) you end up with so much code that you're at the token limit anyway and it can't write code that fits together.

Where I've found ChatGPT 4 useful is getting me going on something, providing boilerplate, and unblocking me.

If you don't know how to approach a problem like the "awareness layer" (like I didn't before reading the post), you can get a great breakdown and starting point from ChatGPT. Similarly, if you're not sure how to approach that view model, or write tests etc. And if you want a first draft of code or tests.

All that said, I'm looking forward to much larger and affordable token limits in future.

Your experience matches mine closely. I've had ChatGPT-4 do great and then it just gets confused after a while. I can literally tell it "task X is done" and it'll apologise and show me a list of tasks where X is still not done - this is clearly not just a context window issue, as I have repeated variations of my statement over and over in the same session and the issue persists.

I have ended up using it the same way you have - it's honestly the best anti-procrastination tool I've ever used because I can tell it my intentions, what I've thought of so far... and it'll spit out a list of bite-sized chunks that get me going. I find myself looking forward to telling the AI I've completed a task.

Similarly, if I'm facing a tricky design decision, I find that just writing it out for ChatGPT is extremely helpful for clarifying my thought process. I actually used to do this conversational decision making process in a text editor long before ChatGPT, but when I know there's an AI on the other end my thinking becomes clearer and more goal-oriented. And unlike talking to myself or a human friend, it's happy to just say "well if these are your concerns, let's start HERE and then see what happens".

Good rule of thumb with ChatGPT: you can’t exit loops. Once you’ve gone A > B > A, your best move is to start a new chat. Even then it may reproduce and you should do some similar but different task. Remember that it’s a prediction engine, weighing heavily on the existing prompt. So you say B again, or B1 and it’s like, I know what to do! A! Cause last time was A->B so let’s do it again.

In your case this would be “[]Task1”, “Task1 is done”, “[]Task1”, [here is where you start a new chat or fix it yourself if possible].

Instead of starting a new chat, why not change the prompt higher up in the conversation with the relevant detail that you have gained through the responses?

I think that would work and be better in some cases. The thing is you want to change things up, lose some context. I may not be able to go far enough back without losing progress to do that. But I could copy just the current state of the task (last response) to a new chat.

Ooh! That's a really good point - ChatGPT is effectively rubber-ducky as a service =)

This is exactly how I've been explaining LLM tech to my "non-geek" friends and family. I start by explaining rubber ducking, and how I now use chatgpt as a more advanced version of the process.

Hmm, I also use ChatGPT as an anti-procrastination tool and task manager, and it's never made a mistake with keeping track of my task list (except that when it sums the estimated times of subgroups of tasks, sometimes those sums are wrong).

Note that it outputs my updated task list every time I add or remove a task (I only asked it to do that one time), so even if old messages go outside of the context window, it's not a big deal because the full updated state of the list is output basically every other message.

Interesting. What's your prompt?

Details here, https://news.ycombinator.com/item?id=35390644

Here's a web app I found recently that should work way better for you. Idk what model it uses (its also free, it feels like chatgpt3.5, so I guess they are funding it out of pocket?)

https://goblin.tools/

You iterate on your plan after it is generated step by step. You go and edit the prompt chain you started working on step 1 on, and modify it to start working on step 2 (including any ideas or fixes you have identified while implementing step 1. Repeat until complete.

You can still absolutely hit the context limit, but you are far less likely to do so if you go back and start a new prompt chain for each different thought process you are going through with it.

Great idea. But does it get hard to navigate back to something in older chat histories though?

I find a new separate chat with the revised initial prompt to be easier.

I’ve been using another call to an LLM to write or rewrite code that is separate from the main “conversation”.

What I mean is that I’ve got a dialog going with an LLM and I’ve trained it to call a build() function with instructions that then returns the function, with the text of the function kept out of the dialog with the main thread.

It's great to see that there's now a term for the type of prompting, “generated knowledge”. I've been experimenting with this technique since the beginning, and I've noticed a significant improvement in version 4. The process involves outlining the project, creating tasks, and feeding them back to chatGPT as you progress. This approach has helped me complete projects that would have otherwise taken me much longer to finish.

It's also useful for creating practical tutorials. While there are plenty of tutorials available online, sometimes you need guidance on a specific set of technologies. By using generated knowledge prompts, you can get a good outline and tasks to help you understand how these technologies interact.

One thing to keep in mind is to avoid derailing the conversation with questions that are not relevant to the core tasks. If you get stuck on something and need to debug, it's best to use a separate conversation to avoid derailing the project's progress and the allucinations & forgettingness

Absolutely, and same here. I've done multiple tools that would have taken 2-3 days each in 2-3 hours each.

> One thing to keep in mind is to avoid derailing the conversation with questions that are not relevant to the core tasks. If you get stuck on something and need to debug, it's best to use a separate conversation to avoid derailing the project's progress and the allucinations & forgettingness

Definitely. Great advice.

Another tip: don't bother asking it to fix small things. Just mention you fixed it in the next reply and move on.

Something must be wrong with me. I could never get anything useful from Martin Fowler's writings, and coincidentally I cannot get any functional code out of ChatGPT. Even the boilerplate it produces for me needs to be corrected. I still use chatGPT to produce examples of abstract things but was not able to get any working code that matches concrete problems or even compiles.

Are you using the GPT4 model? There's a very significant improvement between 3.5 (the free one) and 4.

I am supposedly on GPT4 via GPT+. I try using it for boilerplatey things, like terraform, and the results are simply incorrect. It seems more helpful in providing examples, even for some far more complex tech - like rust code.

Does it say GPT-4 at the top of the screen?

It does. One example of incorrect TF got produced is splitting the dynamo table, and it's gsi into two distinct resources

he would have easily hit the token limit

That was my first question. Do all these tasks fit within a 4K or 8K buffer?

Wouldn't be surprised, though, if it works in a 32K GPT4 token limit. Amazing things are possible.

This may be a dumb question, but do you know if this something that LLM frameworks like LangChain can (or others) can help with? Aren't they designed to help with more complex prompts/logic/outputs? Or will they run into the same token limits?

I completely agree with this take.

Reply on news.ycombinator.com