Comment by baby | Hacker News Reader

Comment by baby | Hacker News Reader

Not sure. But maybe you can answer my questions. I’ve had issues with trying to tell the LLM how long the answer should be. It doesn’t really seem to understand X number of words, or pages, or paragraphs. But I had some success with things like “short story”.

The other thing I’ve been struggling with is to have the AI keep track of what’s important. For example, when the AI learn something from you it should add it to a list (if producing a json output, the object can contain a list of things it knows about you). But it doesn’t always seem to understand it learned something personal from you, and has trouble carrying a list forward without losing items.

The last one is about correcting the user. I want to speak chinese to the AI and I want it to correct me. And if I use english words within my chinese I want it to help me translate them as well. It can’t do none of these things. It’s like it doesn’t seem to realize that chinese and english are two different languages.

I don't know the real answer to your question, but on local models you have a parameter you set that controls how many tokens to generate. It doesn't always follow it, it can end early, but sometimes it just keeps going. Usually though I can set it to generate 700 tokens and it will generate about 700 words.

I wonder if the online chat models have a similar value somewhere.

---

If you want the AI to remember something you will unfortunately have to keep reminding the AI of it in the prompt. With explicitly or you might refer to the previous generated text if it fits into the context. However, in local models the context can be limited (eg 2000 tokens). If the conversation goes above that 2000 tokens then the model will discard stuff from before. There are models with larger context sizes though. Lengthy prompts will cause the same issue though.

The way things like SillyTavern role-playing work is that the model will constantly be reminded of some important attributes of the character that it's role-playing in the prompt (but it's done for you).

That's what I do BTW, for example I say: "these are the things you've learned about the user in the past: ..." but I couldn't get it to use these things in the output object so that the list can only grow.

It'd be cool if the API of LLMs would also allow for structured state like lists

> understand X number of words, or pages, or paragraphs.

LLMs do not have the ability to reason with numbers. Most of the time they are hallucinating. One good strategy is to make it output in list and define the structure for each item of the list. If you give an example of what your list should look like, it will give you something close it.

> has trouble carrying a list forward without losing items.

This is the fundamental problem with these models because of the context limit. When you are prompting always remember that is processing a huge paragraph and emitting the next sentences of the paragraph. If you want information to be carried onwards, you have make it output on every prompt or you can also try to use specific identifiers. LLMs are good at in-context learning. It will not work 100% of the time, but it is usually good than having nothing at all.

> I want to speak chinese to the AI and I want it to correct me.

Give it a role of tutor and describe the instructions what the tutor should do.

> LLMs do not have the ability to reason with numbers

I’ve found ChatGPT pretty good at estimating long division

> LLMs do not have the ability to reason with numbers

Interestingly I get good results when I say "ask me 10 trivia questions"

> Give it a role of tutor and describe the instructions what the tutor should do.

I did do that, it never worked

Reply on news.ycombinator.com