Comment by sheepscreek | Hacker News Reader

Comment by sheepscreek | Hacker News Reader

Great work! Thanks for taking the time to build this and open sourcing it. For me, it scratches on a very real itch. I’ve toyed with the idea of a personal AI assistant to act as a second brain, and help me prioritize and remember things with human intuition (“you can’t afford to delay X, do X today and maybe reach out to Z to set the expectations for delaying Y?”).

To me, the greatest strength of LLMs is not their knowledge (which is prone to hallucination), but their ability to analyze ambiguous requests with ease and develop a sane action plan - much like a competent human.

One a side note: wouldn’t it be significantly cheaper and as effective to use ChatGPT 3.5 by default, and reserve GPT 4 for special tasks with explicit instruction (“Use GPT 4 to…”).

For most chats, GPT 4 would be incredibly wasteful (read: expensive).

Also - it would be very cool to experiment the use of GPT 3.5 and GPT 4 in the same conversation! GPT 3.5 could leverage the analysis of GPT 4 and act as the primary communication “chatbot” interface for addressing incremental requests.

GPT4 is so much better than 3.5 at virtually everything that I don't think it's worth trying to figure out which ones 3.5 is almost adequate enough for.

Also, the pricing is per token, so even with 4 it is close to negligible unless you are loading in a lot of context or your conversation gets very long.

The real problem with 4 is that OpenAI are having trouble keeping up with the demand.

So it might not be an option until 4 turbo comes out in 37 hours or whatever their development cycle is these days.

Yea, I find it a fair bit annoying that I pay $20 and have a 25 message cap every 3 hours.

I use TypingMind.com and the GPT4 API. Seems an overall better experience.

GPT-4 API still not open. I should really try Evals to get early access, been dying to give it a spin (outside of ChatGPT)

It works for me.

Because you were one of the blessed lucky ones to get access to GPT-4 API? I'm still on the waitlist :(.

I pay for it and do not have access to the GPT4 API

I'm just excited to have access at all.

I don't necessarily mind the cap, but I am a little annoyed that it was 50 messages every 4 hours when I first began my subscription and they lowered it without so much as an email telling me.

I guess you have experienced something different, as I got an email saying this:

> GPT4 will have a dynamically adjusted usage cap. We expect to be severely capacity constrained, so the usage cap will depend on demand and system performance. API access will still be through the waitlist.

And then the actual 'sign-up' or marketing page before subscription didn't even say anything about getting GPT-4 (just about getting priority access to the standard chatgpt product).

Then at the bottom of each page above the box when you open GPT-4 as long as I remember it has always said something like "Current limit is this, capacity limits for GPT4 will be changed as we adjust for demand."

well it was something like 100 before that even.

Seriously their release cycle has been so rapid that I am building stuff with the idea that AI will be better and cheaper by the time I am ready to release it.

Right now it is a bit of a blocker because you can easily get a single prompt to cost you .005c - .01c, which would crush you if you ever had any kind of scale.

The one exception is probably speed. GPT-4 is noticeably slower than GPT-3.5.

https://twitter.com/natfriedman/status/1639029709395886080?s...

This by defsult is using the zero shot agent attached to Google, it is burning trough tokens.

Maybe it is just my niche use cases but after spending a good few hours with both, 3.5 for me has actually produced more coherent outputs for me which is a little confusing to me. Maybe I need to rethink my prompting or something.

GPT4 is definitively the future, but GPT3.5 is the present :-).

In addition to being more expensive, GPT4 is a lot slower. For most casual things I use gpt3 and upgrade to GPT4 as needed. I've actually had a couple of days where I spent > $1 on GPT4. It's hard to do with every day chat, but easy to do when you get it to look/improve large amounts of code.

This is all from the API/CLI not the web interface.

>a personal AI assistant to act as a second brain, and help me prioritize and remember things with human intuition

Everyone wants this, but this is not the product.

The current AI offerings are information in --> information out.

It is not meant to keep state long term, and it is not meant to be your friend. It is meant to answer questions with the information available to it.

You can even see in the example screenshot they showcase the fact that it is not designed to be asked follow-up questions.

You can use Langchain / Plugins to get around that issue

That's just the case for this bot, but is obviously not the case for ChatGPT.

I have a long running conversation with ChatGPT that I use to keep track of a verbal to-do list. I tell it my items with categories (e.g. work, personal, etc.) and estimated times, and then it outputs my complete task list, grouped by category. I then just tell it when I add tasks or complete tasks, and it continually keeps track of and outputs my current outstanding task list.

I've been using this for weeks now, and since it's all in a single conversation ChatGPT can keep track of the entire state over time.

I don't have access to plugins yet but it would be trivial to implement a personal AI assistant with ChatGPT if it could, for example, look up flight times and prices.

Oh dang, that sounds awesome. So a ChatGPT conversation doesn't have a historical limit? I guess I assumed it would start having to forget things at a certain point.

As far as I’m aware it will definitely forget things as the history grows. You can “remind” it about things to keep them relevant, and I think a cool product built like OP’s project would take that into consideration for long running tasks.

Yes, there’s a limit. Right now it’s 4000 tokens (gpt4 has a 32k model but i don’t think it’s available yet, at least with chatgpt). Once you near the limit, chatgpt will start dropping previous messages to stay under the limit. I don’t know what their algorithm is for deciding what to drop. It could be as simple as dropping the oldest stuff. Or maybe taking a long message and replacing it with a summary. But at a certain point, the conversation becomes “lossy”.

If it periodically outputs the latest state of the todo list, it won't have the problem of forgetting things from a long time ago.

FWIW that's exactly what it does (updates the current state of the whole todo list each time). After two weeks of use I've never had it make a mistake in keeping track of my total list of tasks. By default (I mean I didn't ask it to do this) it outputs the total estimated time for each category of tasks, and that summation has been wrong (which wasn't that surprising to me, as "LLMs aren't great at math" is a known issue), but even then I just tell it "Can you double check the category totals?" and it fixes them.

They might make it summarize intermittently behind the scenes when the context grows too large. Which would be how it has a gist of the stuff that was mentioned way earlier even thought it doesn't know the specifics.

And what happens when your window runs out? It prunes the oldest convo which you will need to keep track. Likely it could be fine, but if there was a task from a long time ago, that could get wiped out

I'm assuming at this point you'd need a means of pruning expired information from your list, and then importing that shorter list into a new session.

I guess the question is, can we get ChatGPT to make that new list reliably?

You could store whatever personal data you want in embeddings and use the api to refer to those. Can't do it on vanilla ChatGPT with the website though, I don't think.

Am I missing something or is that the opposite of how embeddings work?

https://platform.openai.com/docs/guides/chat - this API endpoint for completions doesn't take embeddings, just messages.

Their API docs for embeddings also don't talk about using them to get outside of the context size limit; instead, the way I've used it and seen others is "create embeddings from documents to enable fast search for relevant documents to populate in context" which still requires a separate data store.

Every time I add or complete a task, ChatGPT outputs the latest complete copy of my task list (I only had to ask it to do this the first time, then it just did that automatically). So, in other words, it always updates the latest "live state" in its most recent message. If I hit the limit on context window it would just take me ~30 seconds to open up a new window with my original prompt and the latest iteration of my task list if I so desired.

Here's where I posted a snippet of this convo a couple weeks ago: https://news.ycombinator.com/item?id=35390644

This looks way more like "improved chat interface to a Reminders app compared to Siri" or such than "prioritize and remember things with human intuition".

ChatGPT can keep some state for you, but there is a limit to the amount of tokens you can keep going in an instance.

It’s enough to keep a todo list going, it’s not enough to make it your friend / coworker

If you built what you were describing right now, either the flight questions would push out your todo list, or you would need to build something to keep state yourself.

good point, I'm doing this with chatGPT, I use a long conversation with 3.5 to help me write prompts for 4, it's fun, when I hit the rate cap on 4 I'll go back to 3.5 with what 4 came up with, then converse on the topic until my 4 cap lifts, combine that with being able to ask bing for things like links and current information and dalle for image based visualisations makes for an intriguing combination, bard gets a look in to but so far seems a little shy compared to 4 or bing.

Reply on news.ycombinator.com