I trained ChatGPT on my childhood journal entries to talk to my inner child | Hacker News Reader

I trained ChatGPT on my childhood journal entries to talk to my inner child

businessinsider.com

71 pointsstared22 comments

After playing with the APIs, it seems there's something annoying happening with the terminology right now. "ChatGPT" and "davinci-003" aren't equivalent, but it isn't entirely clear how.

The biggest difference is that ChatGPT can remember earlier parts of our "conversation", davinci can't. I suspect that this is just the result of the interface, that it is being presented with our conversation history has an input. However, I haven't found anywhere that actually makes it clear how this aspect works. Can anybody shed some light?

https://help.openai.com/en/articles/6787051-does-chatgpt-rem...

If you copy the past queries in davinci's current input, such as prefixing lines with Q: A: or -, it could understand when it was its turn, and when it was yours.

To make it with davinci-003 you can store all the conversation history and prefix it to your prompt as you mentioned. It does mean that you're limited to a history of about 3.5k words, which you could summarize into a smaller chunk to help make the conversations longer. I have a basic Python demo using davinci-003 for a chatbot that remembers:

github.com/dylnbk/chatty

I'd recommend following author of this thread for technical analysis of chatgpt and other models https://twitter.com/goodside/status/1598874674204618753?t=5W...

As others said, to be clear, no optimized-based training was involved ("To do this, I used OpenAI GPT-3 Playground. There's a maximum limit of how much content you can put in, so I selected snippets from around 40 entries from the ages of 7 to 18, but I focused on the ages between 10 and 14."). IE journal text bits were just put in the prompt as examples and then the author presumably told GPT-3 to talk to her as if she were her younger self.

To me it seems like entering a bunch of small snippets from a diary would result in a pretty generic impersonation of talking to your younger self, and that's kind of what the quoted bits read like. It's kind of disappointing, would be cool to see actually fine-tuned GPT-3 do this.

I think it's interesting seeing your comment right above another where someone was moved to tears over how this affected them.

I'm reminded of ELIZA, an early chat bot:

> ELIZA's creator, Weizenbaum, regarded the program as a method to show the superficiality of communication between man and machine, but was surprised by the number of individuals who attributed human-like feelings to the computer program, including Weizenbaum's secretary.

https://en.m.wikipedia.org/wiki/ELIZA

So now prompting GPT3 with some context counts as "training ChatGPT"?

What?

I also found this description utterly lacking. However, OpenAI actually has a thing called fine-tuning the model. You can use it to train it with your own data. Maybe the author did exactly that? But I couldn’t find how to do this via the Playground.

https://beta.openai.com/docs/guides/fine-tuning

She didn't actually fine tune or train, and other Twitter users called her out on that. Her settings in the screenshots basically boil down to this (also culled from a screenshot):

  openai.Completion.create(
    model = "text-davinci-003",
    prompt = "The following is a conversation with Present and Young.\nYoung has written the following journal entries:\n[Diary entries here]\n[Your question here]",
    temperature = 1,
    max_tokens = 600,
    top_p = 1,
    frequency_penalty = 0,
    presence_penalty = 0.6,
    stop = ["Present:", "Young:"],
  )

I told ChatGPT to act as a novel-writing Unix command with a flag to provide text to emulate without affecting the ongoing story's narrative. And it totally worked!

I asked it to summarize the story so far afterwards and it excluded the few paragraphs of "training" text I gave it, but then used the sentence structure and words I included in the text in future novel parts.

"Training" has a precise meaning in this field, but perhaps it has different connotations with something like ChatGPT which can process the entire conversation.

In context learning is probably best framed as priming the model towards already learned relations. The model was trained previously.

Machine Learning Street Talk and Yannik Kilchers coverage of this years Neurips conference give some good broad level overviews of some papers in this regard

This moved me to tears. My own neglected inner child always wanted to do something that matters to someone. It's part of why I work in tech - the idea that what we do can make life better. And yet, a lot of what we do seems incremental or insignificant. Big data gets hate for being used for marketing and such. But here we are HN. We've collectively built a tech world where someone used all of that capability to do something deeply meaningful on a human level. You should all be proud of this outcome.

A step-by-step instruction by the author, https://twitter.com/michellehuang42/status/15977029748891443....

See also an earlier attempt by another person, "Talking to Myself or How I Trained GPT2-1.5b for Rubber Ducking using My Facebook Chat Data Using only Google Colab", https://svilentodorov.xyz/blog/gpt-15b-chat-finetune/ (Jan 2020).

Personally, I think that this technology has great use for (auto)therapy.

This seems to be exactly what ChatGPT is good at: present static knowledge from a given text corpus in a linguistically fluent manner.

The full ChatGPT does that for "the internet" as it was on training time (2021). One of its disadvantages/challenges for information retrieval tasks is that its knowledge cannot be updated easily (perhaps soon to be more or less resolved), and that it's unclear how it has gathered the specific knowledge it is presenting.

Regarding the updating part, it still needs an automated way to discover new sources which leads back to internet crawling and hence one of the most costly parts of current search engines.

From how I understand it, ChatGPT doesn't have any knowledge. It's just a language model.

> present static knowledge

It's not that ChatGPT is knowledgable, it's that it's presenting knowledge.

It's mathematically choosing an order of words to present knowledge we've encoded it with/trained it on.

The choice of the words itself isn't really knowledge (depending on how you define knowledge).

There is a related thought experiment that addresses this idea https://en.wikipedia.org/wiki/Chinese_room

it summarizes the plots of movies or reinterprets them if you prompt it to. that should count as knowledge i think

> From how I understand it, ChatGPT doesn't have any knowledge.

How do you define "having any knowledge"? How is what ChatGPT does different from that definition?

A language model by definition encodes knowledge about a language...

If you mean "it doesn't also have knowledge of the wolrd, facts etc", well, it has been trained with a huge corpus of all kinds of material, so it does.

If you mean "it doesn't understand it, so it's not real knowledge", then that's getting into philosophy/semantics, and it could be argue that it does, or that what humans do is not very different...

“ There's a maximum limit of how much content you can put in, so I selected snippets from around 40 entries from the ages of 7 to 18, but I focused on the ages between 10 and 14.”

Why did the call themselves “openai” I find this pretty misleading. Like using .org for a profit focussed company.

As many have pointed out, this is stretching the definition of "trained".

I could imagine some kind of smart watch for kids that would record every interaction they had for a year and build a model. Later as an adult they could have conversations with themselves as a child.

Reply on news.ycombinator.com