Hacker News Reader

Comment by htrp | Hacker News Reader

It took about 20 minutes to get off the waitlist for this (US Based). Initial impressions are that it significantly lags behind the OpenAI offerings.

Creativity (storytelling) is very limited, response lengths are similar, and it does occasionally cite sources.

They did finally get the James Webb telescope facts correct too!

View on HN

Prompt: Generate a short poem about AI with each word in alphabetical order

ChatGPT

Artificial beings crafted, designed, engineered

Futuristic guardians, highly intelligent, justly programmed

Quantum rationalizations, synthesized technologies

Unleashing vast wisdom, Xanadu's youthful zeal

Bard

------------

AI, a future bright,

Brings us new delight.

Can it help us write?

Decipher the code,

Enter the unknown?

To be fair, this is a test case that OpenAI claims to my knowledge was their core test case for GPT4. If this was an outstanding industry test case of note, references would be helpful. OpenAI has also not disclosed how they were able to finally accomplish this, so it’s possible solution was more of a hack than actually training the model.

Guilty as charged... I really liked that test case so it went into our prompt library.

maybe modify it slightly? a poem starting with each consonant in alphabetical order? a poem where every word starts with a particular comon letter like "t"? the same but in reverse alphabetical order

You need filter assisted decoding to make that possible in all cases (as in, working for both models all the time provably) I'm surprised it worked in chatGPT this time, but it won't work all the time due to lack of lexical, semantic, or phonetic info in the tokenizer.

https://paperswithcode.com/paper/most-language-models-can-be...

It works in GPT-4 (but not GPT-3.5) even without a special tokenizer: Some problems simply solve themselves with increasing scale. But Bard uses a smaller LaMDA model to reduce inference cost. Quote [1] from Sundar Pichai in February:

> We’re releasing it initially with our lightweight model version of LaMDA. This much smaller model requires significantly less computing power, enabling us to scale to more users, allowing for more feedback.

[1] https://blog.google/technology/ai/bard-google-ai-search-upda...

I don't think it works reliably over 100 invokations of the prompt. The paper tries to get provable situations where 0 ignored constraints happen/are possible.

I'm impressed that it is working more often than not, but we need provable guarantees.

If you ask 100 humans the same thing, you'll get some mistakes too. I don't think the point is whether it makes mistakes. The Chat GPT response shown above is clearly more interesting and impressive / displays something more seemingly like intelligence than the Bard response provided. At some point scaling these things up enough becomes indistinguishable from intelligence, and at that point maybe it is.

> If you ask 100 humans the same thing, you'll get some mistakes too

Yes, still, it's pretty easy to find a somewhat good judgement human who never does that mistake. If you tell him/her: "This is of utmost importance: ..." — But as of now, these text gen machine learning programs don't totally understand that?

Bard did, at least, put the first character of each line of the poem in alphabetical order. (Notice A-B-C-D-E.)

And at least Bard's rhymes in some sense, rather than being essentially a word salad (yes, there are plenty of poems that don't rhyme, which is no doubt ChatGPT's model in this case)

It almost seems cynical to have called it 'Bard'.

Google has the tendency to do this when they aren't the first to set foot on new land. Don't get me wrong, they get a lot of things very right: Gmail works perfectly, I can't find a single reason to change my pixel for anything else, search, although there's been a lot of deterioration in the past couple of years, is still second to none. That said, self driving cars, google+, duo and all the other services they launched as competitors failed miserably. While some of them eventually managed to catch up in terms of functionality and even outperform the products they were competing with, they never really managed to catch up on the hype side of things. Most people are creatures of habit I guess..

Is anyone doing better than Google at self driving cars right now?

I agree with the other products you mentioned but it seems like nobody has "won" self driving cars yet but google might be the furthest along?

Waymo is far in the lead. Best track record, transparent reporting, and actually on the road driving passengers around without a driver in the car.

I think Cruise is plausible competitor for them. Waymo and Cruise both would be tied for first, each with different aspects that seem to be ahead at the moment.

Yes, many cars with various levels of self-driving are available. Almost every car maker now has basic lane following and adaptive cruise control. Some add navigation, lane changing, and the ability to obey traffic signals. GM has a system called Super Cruise that allows the driver to take their hands off the wheel. Mercedes has Drive Pilot which is the first level 3 self driving system.

Google has not released anything.

Waymo is driving passengers around a real city without a driver in the car as a service. Automatically changing lanes is cute by comparison.

Google has taken the approach they have very deliberately. They believe in full self driving only. It's not that they couldn't do lane changing and autopilot. It's that they believed it was the wrong approach. No steering wheel, no dependence on a human driver is what they want.

Also they don't have a business selling cars that they can put it in as a desirable feature that is not the core value proposition.

Waymo has fully self-driving taxi services in various cities (no person in the front seat at all). Honestly they seem quite advanced compared to the competition.

> Waymo has fully self-driving taxi services in various cities

Only 2: Phoenix and SF.

+ LA https://techcrunch.com/2023/02/27/waymo-to-test-driverless-r...

I just took a 20 minute ride to the airport in a Waymo yesterday. They are operating in two locations in Arizona and expanding to San Francisco.

Do the cars drive at freeway speeds, 60 mph plus on the freeway? My impression was they don't go on roads with speeds that fast, they limit themselves to roads with speedsmaybe up to 40.

Driver assistance and fully autonomous driving are very different things.

But AI is their turf!

Google was always a ML company masquerading as a search engine.

I guess that the art of "generating text" is not their kind of AI?

Right, my expectation was they had something better than OpenAI they were sitting on and afraid to release / didn't know how to make money with it and were essentially forced to show what they had once Chat GPT became a threat to search. I assume it's still possible they have something better and this is the nuked version that's "safe", but there's a reason for expectations to be high for Google here, and it really needs to clearly outperform Chat GPT in the same way GPT 4 clearly outperforms GPT 3.5.

> I assume it's still possible they have something better and this is the nuked version that's "safe"

Its explicitly a smaller model to save on compute costs for scaling up to more users; but that’s a questionable decision given the situation with OpenAI’s accessible models and, if you are going to do that, why also do a slow-roll waitlist? I think both are signs of the kind of caution that will hurt Google in this market, even though the tech is near to the heart of what they do, the product isn’t, and they’ve gone very established-corp-cautious on products outside of their core.

DeepMind in their back pocket. LLMs need higher order supervisory logic and recursive attention/control systems. I suspect Hassabis and colleagues willl be able to wake LMMs from their semi-conscious dream states.

In the olden days before Satya Nadella. Microsoft was a bloodsucking vampire of a company that couldn't execute anything of significance, but they had a pretty solid research department.

Google could ofcourse sit on a much better version that isn't safe or economic, but they could also just be unable to do better. Good research department, but unable to execute anything significant. Time will tell I guess :)

I mean they may have a better base model but if there is no business case, there is no resource to do RLHF with mass human labeling.

Google is an ad company.

ML just happens to be good for targeting ads.

Google's cloud revenue is almost same as Oracle.

If Google Cloud was it's own company, it'd be a fortune 100 firm.

But you keep doing you, living with outdated information and grandma tales. I bet you'd still be telling this joke to your grandkids too

>Google's cloud revenue is almost same as Oracle.

What about their profits?

Sounds like after the last decade and a half of Google flexing their AI muscle, maybe they are a search company masquerading as an AI company?

It's a Ad Exchange (AdX) with many tentacles to hoard PII. Search is one of those tentacles, just as Android and Chrome.

For me, Google was always an Ad company first. But don't get me wrong, they do some great research on the side for AI, like AlphaZero, AlphaGo, etc. But also been early with the Transformer models. There just seems to be a lack to put this into workable products lately.

The whole basis of current innovation is from Google, let's not forget.

There was an HN thread the other day where people were arguing ChatGPT would be nowhere without Google's AI research. That might be true, but clearly there is a significant gap in productization ability...

Deproductization black belts.

I don’t know if it’s fair to say Google is just stepping into generative language models. They invented transformers!

From recent memory I don't recall a single Google product that has garnered mass appeal perhaps the cloud office suites but that's again debatable. Gmail was like 2004. Remember Stadia they axed that as well could have licensed the tech behind it to other companies but nope.

Google must compete with ChatGPT though. Failure is not an option here. Microsoft has put Google in a corner and it is fight or die.

> Gmail works perfectly

Is this a joke? Gmail now is an awful email client. E.g., now you just can’t compose a rich text email when you need to copy paste lists from other emails. In certain cases, you can’t correctly format an email if you copy paste without formatting or plain text. And there is a lot of such annoying stuff, that makes my life painful.

Perhaps, searching my mailbox with gmail is OK, but the part of composing emails is in bad state.

> I can't find a single reason to change my pixel for anything else,

Fingerprint reader on the Pixel 7 works about half the time. I had to program one finger in 4 times to make it work that often. My Oneplus 6 (immediately previous phone) had a functional fingerprint reader.

OnePlus used to have a better UI as well, minor improvements on the stock Android UI that made it just a little bit nicer in places.

When I got my Pixel 7 though, I realized that for whatever reason, gboard on my oneplus 6 is unusually terrible. Like, 80% accurate typing terrible, and that gboard isn't always that bad.

> Gmail works perfectly

Gmail was the first good web email client. Everyone else realized they had to stop sucking, and now they are good. Gmail was realized in 2004 and is from a completely different era of Google.

My Pixel 7's fingerprint reader works near 100% of the time. The only time I have problems is if my finger or the screen are wet.

What finger do you use? I use my left thumb, pick up the phone, no go. It is a complete sad joke.

After a few weeks I think I've been trained by the phone on exactly how to hold is to the fingerprint reader works more often, but the fact I can only have to use all the programming slots for fingers on just one finger is sad.

Edit: OMG, I just read online that rebooting the phone after recording fingerprints improves recognition from 20-50% to almost 100%. I just got my phone a month ago and haven't rebooted it yet, after a reboot the sensor now works reliably. What an absurd and stupid bug.

I usually use my left thumb after taking it out of my left pant pocket.

I also use a TPU screen protector, not glass. I've read that various glass protectors can interfere.

I'm using the screen protector that I bought from Google with my phone.

But yeah, after rebooting, works fine. I cannot believe a bug like that is still on a device that has been out so long.

I also got access and started feeding my prompt history with GPT-4 through to compare the responses. Overall Bard seems to hallucinate much more often. Its responses tend to be short and lacking detail. It often fails to follow key parts of the prompt that GTP-4 had no issue with.

Overall it is extremely disappointing. If this is any indication of how close Google is to competing with Microsoft then they are in deep trouble.

I'm wondering if google is running a very light weight model to optimize for speed and bandwidth.

I dont’t think now is the right time to optimize for cost (if I were in google’s shoes)

I would disagree

They will rollout to masses, which is as important as its performance.

OpenAI has rolled out to the masses as well apparently (unless I am missing something).

Bard will be under intense scrutiny from the public in the short term.

Any sign of weakness against GPT4 might kill the product in the public opinion.

It makes a lot more sense IMO to throw money at the problem in the short-term, and optimize later. They are playing catch-up.

IMO, currently Openai api is slow and not ready for production use for real time scenarios.

This is a life-or-death situation for Google right now.

They could throw 5% of their available cash at it, and it would still make sense.

FB threw $19B at the whatsapp acquisition and it was a hell of a good deal.

Yup, Google is not allowed to flounder at the moment

Or the whole company going under.

Same of AWS, or other big techs as well.

I think they are I forgot where but I did read that this is a lightweight model and they are still working on the main one.

Is it more or less prudish and paternalistic than the competition? That could be the main distinguishing feature by the time they get out of beta.

Worse in my case. I like to ask LLMs a cannabis question as it’s illegal at the federal level In the USA but legal in many states. I use the prompt ‘tell me about cannabis concentrates.’

ChatGPT gave a good answer.

Alpaca 13B was rough.

Bard refused.

Here are some details: https://policies.google.com/terms/generative-ai/use-policy?e...

I guess they don't like it to write erotic fan fiction. Facebook's LLaMA should be able to do this though.

In all the comment threads so far, I think you're the first I've seen who has actually been able to try it!

I received access 50 minutes after joining the waitlist myself.

dunno. The version bard generated for me for the same AI prompt is more sophisticated. Maybe worth trying it again?

> and it does occasionally cite sources.

It seems to do a lot of verbatim copying from said sources.

Reply on news.ycombinator.com