Hacker News Reader

Comment by jxy | Hacker News Reader

So how do we actually try out the PaLM 2?

The links in their press release just link to their other press release, and if I google "PaLM API" it just gives me more press release, but I just couldn't find the actual document for their PaLM API.

How do I actually google the "PaLM API" for a way to test "PaLM 2"?

View on HN

Google's docs on the APIs are up: https://cloud.google.com/vertex-ai/docs/generative-ai/learn/...

The pricing is also now listed but free during the trial period, although it's annoyingly priced by character: https://cloud.google.com/vertex-ai/pricing#generative_ai_mod...

Assuming ChatGPT's tokens are the equivalent of 4 characters on average (a fair assumption), the pricing of PaLM's chat and embedding APIs are the same cost as OpenAI's equivalents.

Why would that be annoying? It’s much easier to understand, predict and truncate appropriately than having to explain all of these different tokenization schemes to devs.

Yeah, everybody agrees on what a character is, right? It's just {an ASCII byte|a UTF8 code unit|a UTF16 code unit|a Unicode code point|a Unicode grapheme}.

I’m not saying it’s easy but it’s much better than tokens IMO. I think bytes would be understandable too.

Bytes are understandable but make no sense from a business point of view. If you submit the same simple query with UTF-8 and UTF-32, the latter will cost 4x as much.

No API accepts input in UTF-32. Nobody uses this on the internet.

At least there are standards for characters. Nothing like that for tokens.

And we think tokens solve that problem? Spoiler alert: they don’t

https://www.reddit.com/r/OpenAI/comments/124v2oi/hindi_8_tim...

They don't but Google could have been more precise with which of the definitions listed by GP they mean by "character".

There is a limit of maxOutputTokens, 1024! Is this the true capability of PaLM 2?

However I couldn't find anything about the context length of their model anywhere. And the API didn't tell me how long the prompt could be.

No. Autoregressive models don't have model specific limit to output tokens, it's just when to stop looping.

I guess I only know transformers and how BERT or GPT works, as there would be a limit in the context length. With GPT, you can certainly generate infinite amount of tokens but the previous tokens outside of the maximum context length would be outside of the context window. LLaMa has 2k, GPT-4 has 32k.

Are you saying I can give unlimited tokens to PaLM and generate unlimited amount of tokens? So PaLM doesn't have a context limit?

No, I am not saying that. Since PaLM 2 is a transformer model (they didn't disclose almost anything about the model architecture, but they did disclose that), it has a context length limit. What I am saying is that you can't infer that limit from the limit of maxOutputTokens parameter in the API.

Per token might be 4 characters on average, but that can vary wildly. Pricing per character is easier to understand and means more flexibility to change tokenisation without affecting pricing. So far OpenAI has charged very different prices per model, but I expect we’ll see more granular changes in the future that might not change pricing… except for changing the tokenisation.

It's happy pricing as a Japanese user, maybe also for Korean and Chinese.

You can sign up for the waitlist at g.co/palm

It should be live on Bard.

But Google hasn't disclosed which version of Bard, right?

I pop into Bard every once in a while to test its performance, but I never know if I'm getting the best Google has or just what Google can tolerate running cost-wise publicly given they potentially have at least an order of magnitude (if not two, edit: 1.5) more users than OpenAI.

I am sure that Bard has far fewer users than OpenAI.

Oh absolutely, I'm just imagining what I might think if I was a super conservative director at Google who is accountable for the balance sheet of a large org.

If that were the case you’d be too busy fighting over head count, trying to hit the VP rung, and internal empire building to do any actual work.

Given that ChatGPT has allegedly 100M users, two orders of magnitude more than that would be larger than the global population. Even if we count everyone with a Google account as a potential user of PaLM, that can't be true.

Ah yeah, I had the outdated 30M in my head.

chat gpt had 100m users in feb. safe to assume it has at least 2-5xed since

Sundar Pichai:

> We’ve been rapidly evolving Bard. It now supports a wide range of programming capabilities, and it’s gotten much smarter at reasoning and math prompts. And, as of today, it is now fully running on PaLM 2.

So yes, Bard uses PaLM 2 now. No longer the small LaMDA model it used before. It's a completely different thing now.

Bard is not live though.

> Bard isn't currently supported in your country. Stay tuned!

It has been months…

> Yesterday at Google I/O 2023, it was announced that Google Bard would be undergoing a massive expansion, bringing the AI chatbot experiment to 180 countries. However, what Google didn’t mention is that Bard still isn’t available in the European Union.

https://9to5google.com/2023/05/11/google-bard-european-union...

https://developers.generativeai.google/guide

In Google Cloud Vertex AI, you can play with it straight away

No API, but Bard is on it.

They’ve shut down and/or changed prices on APIs so many times as long as it isn’t 100x lower performance than an alternative I can’t see myself investing building a stack that relies on it.

Reply on news.ycombinator.com