Hacker News Reader

> Some practical notes from digging around in their documentation: In order to get access to this, you need to be on their tier 5 level, which requires $1,000 total paid and 30+ days since first successful payment.

Tier 5 level required for _API access_. ChatGPT Plus users, for example, also have access to the o1 models.

We just receivied this email:

Hi there,

I’m x, PM for the OpenAI API. I’m pleased to share with you our new series of models, OpenAI o1. We’ve developed these models to spend more time thinking before they respond. They can reason through complex tasks and solve harder problems than previous models in science, coding, and math.

As a trusted developer on usage tier 5, you’re invited to get started with the o1 beta today. Read the docs You have access to two models:

    Our larger model, o1-preview, which has strong reasoning capabilities and broad world knowledge. 
    Our smaller model, o1-mini, which is 80% cheaper than o1-preview.

Try both models! You may find one better than the other for your specific use case. Both currently have a rate limit of 20 RPM during the beta. But keep in mind o1-mini is faster, cheaper, and competitive with o1-preview at coding tasks (you can see how it performs here). We’ve also written up more about these models in our blog post.

I’m curious to hear what you think. If you’re on X, I’d love to see what you build—just reply to our post.

Best, OpenAI API

Reasoning tokens are indeed billed as output tokens.

> While reasoning tokens are not visible via the API, they still occupy space in the model's context window and are billed as output tokens.

From here: https://platform.openai.com/docs/guides/reasoning

This is concerning - how do you know you aren’t being fleeced out of your money here…? You’ll get your results, but did you really use that much?

obfuscated billing has long been a staple of all great cloud products. AWS innovated in the space and now many have followed in their footsteps

Also, now we're paying for output tokens that aren't even output, with no good explanation for why these tokens should be hidden from the person who paid for them.

If you read the link they have a section specifically explaining why it is hidden.

I read it. It's a bad explanation.

The only bit about it that feels at all truthful is this bit, which is glossed over but likely the only real factor in the decision:

> after weighing multiple factors including ... competitive advantage ... we have decided not to show the raw chains of thought to users.

Good catch. That indicates that chains of thought are a straightforward approach to make LLMs better at reasoning if you could copy it just by seeing the steps.

Bad, in your opinion.

Also seems very impractical to embed this into a deployed product. How can you possibly hope to control and estimate costs? I guess this is strictly meant for R&D purposes.

You can specify the max length of the response, which presumably includes the hidden tokens.

I don't see why this is qualitatively different from a cost perspective than using CoT prompting on existing models.

You can’t verify that you’re paying what you should be if you can’t see the hidden tokens.

With the conventional models you don't get the activations or the logits even though those would be useful.

Ultimately if the output of the model is not worth what you end up paying for it then great, I don't see why it really matters to you whether OpenAI is lying about token counts or not.

As a single user, it doesn’t really, but as a SaaS operator I want tractable, hopefully predictable pricing.

I wouldn’t just implicitly trust a vendor when they say “yeah we’re just going to charge you for what we feel like when we feel like. You can trust us.”

For one, you don't get to see any output at all if you run out of tokens during thinking.

If you set a limit, once it's hit you just get a failed request with no introspection on where and why CoT went off the rails

Why would I pay for zero output? That’s essentially throwing money down the drain.

They are currently trying to raise money (talk of new $150B valuation), so that may have something to do with it

In the UI the reasoning is visible. The API can probably return it too, just check the code

What's shown in the UI is a summary of the reasoning

OAI doesn't show the actual COT, on the grounds that it's potentially unsafe output and also to prevent competitors training on it. You only see a sanitized summary.

I think it's fantastic that now, for very little money, everyone gets to share a narrow but stressful subset of what it feels like to employ other people.

Really, I recommend reading this part of the thread while thinking about the analogy. It's great.

Your idea is really a brilliant insight. Revealing.

Any respectable employer/employee relationship transacts on results rather than time anyway. Not sure the analogy is very applicable in that light.

It is!

> Any respectable employer/employee relationship transacts on results rather than time anyway.

No. This may be common in freelance contracts, but is almost never the case in employment contracts, which specify a time-based compensation (usually either per hour or per month).

Yea, I said ‘respectable’.

That's just not how employment laws are written.

Employment law actually permits per-piece payments too, albeit that type of pay scale is rare.

I believe parent's point was that if ones management is clueless as to how to measure output and compensation/continued employment is unlinked from same... one is probably working for a bad company.

It’s nice on the outside, but employees are actually all different people and this here is one company’s blob of numbers with not much incentive to optimize your cost.

Competition fixes some of this, I hope Anthropic and Mistral are not far behind.

> […] with not much incentive to optimize your cost. Competition fixes some of this […]

Just like employing other people!

On the contrary. It will be the world's most scrutinized employee. Thousands of people, amongst them important people with big levers, will be screaming in their ear on my behalf constantly, and my — our collective — employee gets better without me having to do anything. It's fantastic!

I love this so much haha.

"I can only ask my employee 20 smart things this week for $20?! And they get dumber (gpt-4o) after that? Not worth it!"

No access to reasoning output seems totally bonkers. All of the real cost is in inference, assembling an HTTP request to deliver that result seems trivial?

Some of the queries run for multiple minutes. 40 tokens/sec is too slow for CoT.

I hope OpenAI is investing in low-latency like Groq's tech that can reach 1k tokens/sec.

It's slow and expensive if you compare it with other LLMs.

It's lightning fast and dirt cheap if you compare it to consulting with a human expert, which it appears to be competitive with.

I would say consulting with a human. Any expert who has a conversation with chatGPT about their field will verify that it is very far from expert

According to the data provided by OpenAI, that isn't true anymore. And I trust data more than anecdotal claims made by people whose job is being threatened by systems like these.

>According to the data provided by OpenAI, that isn't true anymore

OpenAI main job is to sell that their models are better than human. I still remember when they're marketing their gpt-2 weights as too dangerous to release.

I remember that too, it's when I started following the space (shout out computerphile/robert miles) and iirc the reason they gave was not "it's too dangerous cause it's so badass" they basically were correct in that it can produce sufficiently "human" output as to break typical bot detectors on social media which is a legitimate problem - whether the repercussions of that failure to detect botting is meaningful enough to be considered "dangerous" is up to the reader to decide

also worth noting I don't agree with the comment you're replying to - but did want to add context to the situation of gpt-2

> For each problem, our system sampled many candidate submissions and submitted 50 of them based on a test-time selection strategy. Submissions were selected based on performance on the IOI public test cases, model-generated test cases, and a learned scoring function. If we had instead submitted at random, we would have only scored 156 points on average, suggesting that this strategy was worth nearly 60 points under competition constraints.

Did you read the post? OpenAI clearly states that the results are cherry-picked. Just a random query will have far worse results. To get equal results you need to ask the same query dozens of time and then have enough expertise to pick the best one, which might be quite hard for a problem that you have little idea about.

Combine this with the fact that this blog post is a sales pitch with the very best test results out of probably many more benchmarks we will never see and it seems obvious that human experts are still several order of magnitudes ahead.

When I read that line too I was very confused lol. I interpreted it as them saying they basically took other contestant submissions and allowing the model to see these "solutions" as part of context? and then having the model generate its own "solution" to be used for the benchmark. I fail to see how this is "solving" a ioi level question.

What is interesting is the following paragraph in the post " With a relaxed submission constraint, we found that model performance improved significantly. When allowed 10,000 submissions per problem, the model achieved a score of 362.14 – above the gold medal threshold – even without any test-time selection strategy. " So they didn't allow sampling from other contest solutions here? If that is the case quite interesting, since the model is effectively imo able to brute force questions. Provided you have some form of a validator able to tell it to halt.

I came across one of the ioi questions this year that I had trouble solving (I am pretty noob tho) which made me curious about how these reported results were reflected. The question at hand being https://github.com/ioi-2024/tasks/blob/main/day2/hieroglyphs... Apparently, the model was able to get it partially correct. https://x.com/markchen90/status/1834358725676572777

What? Surely you have some area of your life you are above-average knowledgable about. Have a conversation with chatGPT about it, with whatever model, and you can see for yourself it is far from expert level.

You are not "trusting data more than anecdotal claims", you are trusting marketing over reality.

Benchmarks can be gamed. Statistics can be manipulated. Demonstrations can be cherry picked.

PS: I stand to gain heavily if AI systems could perform at an expert level, this is not a claim from someone 'whose job is being threatened'.

So, basically, it's chain of thought as a service?

Not a model, per se, but a service that chains multiple model requests behind the scene?

Who knows? Certainly not the public.

It might be a finetuned model that works better in such a setting.

The linked blog posts explains that it is fine-tuned on some reinforcement learning process. It doesn’t go into details but they do claim it’s not just the base model with chain of thought, there’s some fine-tuning going on.

Unless this is specifically relating to API access, I don’t think it’s correct. I’ve been paying for ChatGPT via the App Store IAP for around a year or less, and I’ve already got both o1-preview and o1-mini available in-app.

Yes, I was referring to API access specifically. Nothing in the blog post or the documentation mentions access to these new models on ChatGPT, and even as a paid user I’m not seeing them on there (Edit: I am seeing it now in the app). But looks like a bunch of other people in this discussion do have it on ChatGPT, so that’s exciting to hear.

I'm a bit late to the show, but it would seem the API calls for these new models don't support system messages (where role is system) or the tool list for function calls.

I have access to this and there is no way I spend more than 50$ on OpenAI api. I have ChatGPT + since day q though (240$ probably in total)

You missed your raise key on "day q"

Raise it up just one

The CoT is billed as output tokens. Mentioned in the docs where it talks about reasoning

I am an ordinary plus user (since it was released more or less) and have access.

I am a Plus user and pay $20 per month. I have access to the o1 models.

A bit out of context.

Am curious if at some point length of context window stops playing any material difference in the output and it just stops making any economical sense as law of marginal diminishing utility kicks in.

you need to be on their tier 5 level, which requires $1,000 total paid and [...]

Good opening for OpenAI's competitors to run a 'we're not snobs' promotion.

How so? I think most of the competition does this. Early partners/heavy users get access first which 1) hopefully provides feedback on the product and 2) provides a mechanism to stagger the release.

Marketing is about feelings, not facts.

We could tell it impacted your feelings but most business don’t run off of feelings. There is sometimes alignment on morals/being a good business partner but before that it’s the quality of the product and the cost.