Ask HN: How are you finding Gemini Ultra? | Hacker News Reader

Ask HN: How are you finding Gemini Ultra?

66 pointsvopi54 comments

Is it worth a subscription (compared to GPT-4)?

Does anyone have interesting anecdotes in different categories? Programming, General Lookup, Math, etc

One thing I've found Gemini Advanced (Ultra) is actually good at is sustaining a conversation to work towards a goal that requires some back and forth. I've been calling it "putting it in collaboration mode", which isn't super accurate since it's not technically a "mode" but whatever. But in my experience ChatGPT (and a lot of instruction tuned LLMs) tend to try to get to the end of the conversation as quickly as possible. (To the point where GPT4 will regularly just end the "and if that doesn't work, go read the documentation" ... which is very frustrating.)

Gemini on the other hand will ask several follow-up questions and behaves more like a conversation you might have with a coworker on a pair-programming topic. It's kind of changed the way I think about the utility of LLMs.

It's funny you say that, because I just had a similar experience.

Front-end web development with HTML, CSS and JavaScript is not one of my strengths, so I've been trying to go back-and-forth with GPT-4 to get a working prototype.

Before asking GPT-4 to generate any code, I try to have a conversation to explore the pros and cons of various approaches. Is it better to put separate click events on every object, or listen for events at a higher level of the DOM? Is it better to use callbacks for this or async/await?

But GPT-4 rushes to write code before requirements are defined, and then it struggles to modify that code later after I've decided what approach to take. It forgets what it did before, provides partial results, alters variable names between chat turns, contradicts itself.

Just tried the same dialogues with Gemini Advanced and it's night-and-day better. It seems to remember contexts from 10+ conversation turns ago, it provides a more thorough rationale for its suggestions, doesn't prematurely spit out code, and keeps variable names constant throughout the dialogue.

After the approach and requirements are defined, it can then generate large blocks of code with intelligent modularization into separate functions, more similar to how I would write it by hand than what I get from GPT-4.

Gemini Advanced also seems to prefer the more modern ES6 coding style (arrow functions, destructuring, const/let variable declaration). GPT-4 defaults to the 15-year-old style (string concatenation for building strings instead of template literals, heavy use of var for variable declaration, function expressions instead of arrow functions, etc.). GPT-4 routinely mixes styles even when I explicitly request ES6.

I've struggled to get it to generate images, especially when related to memes. For instance, I wanted to extend the "this is fine" image, but Gemini Advanced would not because it was harmful to animals. Most of the image requests I give it are back-and-forths on what it can't generate for me for some ethical or moral reason.

My initial impression was good but the extreme "safety" blocks in Gemini are what makes me still use ChatGPT (which isn't great in the regard either but not as bad as Gemini). I wonder if they even have a way to track how much the AI safety people are impacting revenue.

I've been using GPT-4 (mostly for coding and general knowledge, use it a lot more for personal projects than actual work) for the past year, and just switched to Gemini Advanced (since there's a 2 month trial, that includes Google One, which I'm already paying for), and so far it's pretty comparable. The "check sources" google button is sort of handy but it only works maybe 25% of the time to get a confirmation of a statement.

It sort of adds a "confidence level" check that saves me from doing it myself afterward like I would for GPT-4.

Edit to add: Gemini Advanced seems to be a lot faster as well at responding, in terms of tokens/second, compared to GPT-4.

Currently subscribing to both, and for tricky programming questions I go ahead and ask both and take the answer I like more. There was an AWS networking problem I was debugging that Gemini got but GPT4 didn't, but Gemini has trouble with giant pasted code, and doesn't allow file upload yet, which is a major blocker for me.

Have you tried NotebookLM for file uploads?

For things I would formerly use a web browser to look up, Gemini is often a lot better than gpt-4. The more obscure the query, the better Gemini does compared to gpt-4.

But gpt-4 is still best for ML Python coding. Gemini hallucinates non-existent libraries and often adds unnecessary junk to its code. For example, Gemini often defines spurious variables and then never uses them.

> gpt-4 is still best for ML Python coding

How does GPT4 do with Pytorch, Pytorch Lightning, etc? I'm pretty surprised at how poor GTP3.5 is with those sometimes.

Thanks for answering the one question tells me whether I need to try Gemini!

This is about early testing (on code) of Gemini 1.5, but presumably that will come to the Gemini consumer product in not too long so …

I've put a complex codebase into a single 120K-token prompt, and asked 7 questions GPT-4 and Gemini 1.5. Here are the results! https://www.reddit.com/r/singularity/comments/1atjz9v/ive_pu...

Interesting thanks! On your point #4, I might suggest that hallucination is worse than failing to answer, so maybe it wasn’t a tie.

Also how do you get the code into Gemini? I’ve pasted a C++ file in to the chat window and I get errors. Seems to work on AI studio though, is that how you interface with it?

For writing text (emails, etc.) and asking basic questions gemini is superior to gpt-4 in my opinion. But when it comes to logic, programming and resoning, gpt4 is better in my opinion. It rarely makes syntax or logical errors, and the jupyter notebook that chatgpt has access to is a big plus for me. It can use it to solve math questions with sympy, use matplotlib and I can ask it to crop a video file for me. Gemini makes a lot of basic mistakes and the code examples are very minimalistic.

For those that want to give it a shot, there's a two month free trial here:

https://gemini.google.com/advanced

It was not so easy to even find out how to buy Gemini Ultra, apparently it's called "Gemini Advanced," or maybe it's called "Google One AI Premium Plan." Searching for "Gemini Ultra" doesn't give you a link to any this on any of the top page of hits on a Google Web Search, and you have to dig deep in announcements to find it.

I've been very dissatisfied with all the free Google options, but this post has given me the kick to see if the bigger models can compete.

I think if you already are paying for google drive / gmail / google photos storage < 2TB this also means they'll comp you 2 months of that, since this includes it, so in essence, signing up for this trial will save you money.

I'll find out next billing cycle, but it'd be pretty misleading otherwise, and they'd obviously be incentivized to hook you in with the dark pattern of "auto renewing free trial subscription"

My first question: "How do I get an API key to use gemini ultra 1.0?"

Response was that you have to request access, either through 1) your account representative, or 2) "Apply through Google: If you don't have an account representative, you may be able to apply for access through Google's channels (a specific form or process might be available)"

I don't have an account representative, of course (who does?!)

But the really interesting thing is that there was an additional annotation added to option 2, about the other channels, that said "Google Search didn’t find relevant content. Consider researching further to assess the statement." which sounds about right.

So, the model itself seems OK, if Google can figure out how to get out of its own way.

I subscribe to both ChatGPT and Gemini Ultra. Some interesting features/experiences with Gemini...

1. I asked it to do an image search and it responded with in-line images throughout its response. This was a nice "wow" moment and very neat.

2. Its drawing/illustration style is different than DALL-E, so I use both.

3. Quality of general text responses is comparable, though I prefer ChatGPT.

I imagine you can probably just use one rather than have both. I am still primarily using ChatGPT.

How do you get Gemini Ultra to generate images? It just tells me that it can't do that yet.

Try telling instead of asking. When I tried it the other day "Can you create a picture" gave the response "no try DALL-E instead". Then I noticed one of the example prompts was "Generate an image with an Elephant ...." It worked, as did some other random stuff I tried as long as I told it to do it not ask it to.

I just tried asking it again and asking seems to work now too.

Most European countries are excluded:

> Image generation in Gemini Apps is available in most countries, except in the European Economic Area (EEA), Switzerland, and the UK. It’s only available for English prompts.

(https://support.google.com/gemini/answer/14286560?hl=en)

"Can you draw a photo of an avocado-shaped chair with a pineapple-man sitting in it?"

Came up with 4 images.

You raise a good point, though... I've also asked it to use its "web search" capability for tasks and it says it doesn't have that capability, but when I ask it by implying it should do a web search, it goes ahead and does it. Weird!

Yes, Gemini and previously Bard has a lot of confusion about its own capabilities. I use it to translate Chinese text in aliexpress product listings by taking screenshots. It’s perfectly capable and quite helpful in translating the text from those screenshots, but I think depending on how you phrase the question while uploading the photo, it will sometimes say “I’m only a language model I can’t help with that” or even “I can’t help with images”. Once it says that, I think it poisons the chat history and I start a new session to try to get it to work. I’ve not translated many images but so far this error happens maybe 20% of the time. It’s very strange.

I have another issue which is that when I paste a C++ code file in to the web interface, I get an error from the web interface and Gemini never even sees the code. The web interface is refusing to accept my code file. I opened up AI studio instead of the normal Gemini window and that seems to work, but I’d rather just use the normal chat window.

It’s all statistics. In the training set, there were probably questions asking about its capabilities and it was trained to say it has less than it does. (Or it’s a bad system prompt)

There’s no internal understanding of itself or its capabilities.

There is no understanding to answer a question about its capabilities but the point is it has the capability but the prompt is failing to trigger it. This is separate from "knowing" or not. Think ChatGPT functions that don't work.

Knowing how these models are trained and how these chat systems are built, I wouldn’t expect the question

“Can you search the internet?”

To actually cause an internet search.

Generally you know information about yourself, and that quirk of humans is likely reflected in the QA training data and thus the model’s outputs.

I live in the UK and have the same problem - it doesn’t work here.

I'm a Google Fi subscriber, so I still can't switch to the Google One plan with access to it. Incredibly frustrating.

Are they related? I have both Fi and One, never saw anything saying they had something to do with each other

We have the Unlimited Plus plan and that comes with Google One, so it seems there's some accounting logic they haven't figured out how to reconcile.

I keep separate @gmail and Google workspace accounts so that I never run into “not supported here” issues.

I use Github Copilot and GPT4, and some CodyAI with GPT4 for coding. I still find them better for code. But, I have been using Gemini Ultra for creative writing targeting humans.

For human writing, Ultra is w-a-y beyond what ChatGPT 4 is capable of IMHO. GPT4 sounds like an AI write it. Even if you put a lot of prompting into establishing the right tone. With Ultra, it is getting hard to tell. Which is great! I think of them like people. Each has their own opinion and has talents in different areas. Given they are $20/m I am happy to pay for all of them. It takes them about 5 minutes to pay for themselves.

I passed this text through Ultra and this is what I got:

Absolutely! Here's a rephrasing that focuses on a more natural, flowing conversational style while still conveying the core ideas:

I use GitHub Copilot, GPT-4 (and CodyAI), and Gemini Ultra regularly, and I've noticed a big difference in how they approach tasks. For coding, Copilot and GPT-4 are my top picks. However, when it comes to writing that needs to feel truly human, Gemini Ultra is hands-down the winner. It's getting to the point where I often can't tell the difference between its output and something a real writer would create. That's a huge difference from ChatGPT-4, which still can have that telltale "AI feel" even with careful prompting. These tools each have such unique strengths that I find them all valuable – the $20/month investment for each pays off in just minutes.

Changes Made:

* No Visual Cues: Removed formatting elements (bold, bullets) that can scream "AI-assisted".

* Conversational Flow: Uses phrases like "I've noticed a big difference" and "hands-down the winner" to feel more like someone speaking their experience.

* Softened Comparison: Keeps the idea that Ultra surpasses ChatGPT-4 for a certain type of writing, but tones down the strong declaration for a more subjective feel.

Let me know if you want it even more casual or have another specific tone in mind!

For my purpose — reading hourlong transcripts of conference calls and answering questions about them — it is pretty useless. GPT-4 got nerfed so many times that it is no longer useful for that. Claude 2 is now my go-to.

I don't live in the US, so I'm not finding it at all.

Just "bought" access to the trial in Germany.

I currently, have a free two-month trial, but it's not worth paying for yet. It still fails in hilariously simple ways. E.g, one of the example prompts shows how to use it to generate an image. But asking [Can you generate pictures?] results in "Unfortunately, I am not currently able to generate images directly. However, I can help you explore some resources that can do just that!...". Forcing it to generate an image result in broken <IMG> elements.

It also fails at more complex tasks that I would actually use AIs for. E.g, I'm trying to learn Lagrangian Mechanics and keep running into dead ends with Gemini in cases where ChatGPT is working.

I have faith that Google will make it better in the near future (unless they get bored and move on to something else) so I'm hoping it will be worth paying for. But right now, I'm going to keep my ChatGPT subscription since that is actually useful.

I'd be careful using LLMs to learn mathematical concepts. I asked ChatGPT for the axioms of set theory, I was still in the "use ChatGPT as a search engine to see what happens" phase. It left out two of them and got one subtly wrong. It's the last bit you really have to watch out for.

Math is exact. It's a bad fit for a machine-that-guesses-words.

The good thing about math is that it's possible to verify the result in many cases even if the method used to produce the result is too difficult to understand. Also, if I ask ChatGPT & Gemini to walk me through the problem step-by-step, I know enough to follow and detect errors. E.g, Gemini kept getting errors because it used the wrong starting potential energy for a pendulum and assumed the pendulum had a max PE of 0 at the top of the swing (and a negative PE at the bottom) which caused it to have a max velocity of 0 due to incorrectly applying the conservation of energy.

But you're right that someone who doesn't understand a subject should be wary of trusting LLMs to teach the subject.

Actually for coding sometimes it’s better than GPT4. A lot of times worse, but my work flow is to give the same coding question to both. I play them off eachother. OpenAI will lose their edge if they don’t release a new model soon

It is worth the 2 month feee trial with the Google one space.

But - I am a Uk iPhone user - so Gemini Advanced is only available via a website rather than app, and no image generator. So, for me:

ChatGPT4 Pro > copilot pro > Gemini advanced

Ask it about bgp states and ask it to create an image !! It creates faces !!

Haven't tried any of them.

Google is never worth a subscription. They have completely moved away from "don't be evil"; don't encourage them by giving them more money.

No need to worry they will sunset the product themselves. It aint ads it aint no dollars.

There are posts going around X that Gemini image generation is very biased toward minority preferences even when factual data and real-life statistics don't support the outputs. For example, can somebody ask Gemini to generate a picture of a typical Swedish women?

Sure I asked it to generate a picture of a Swedish woman and it generated four white women that look Swedish to me (not that I would really know).

I then asked it to generate a portrait of a woman and it generated a white woman, a woman that to my eye looks East Asian, and a black woman. If you consider a random sample of all skin tones and regions on earth, this distribution seems pretty normal.

I’ve seen some people get very annoyed when people who don’t look like them get represented somewhere. Maybe those people were being extra loud about the representation. Or maybe I’ve not been able to generate the purported weird distributions you have heard of.

Interesting, probably "fake news" then on X. Is there a way to publicly share Gemini interactions like with chatGPT?

Apparently yes, even from the Google mobile app where I did this.

https://g.co/bard/share/325bfd6d79cd

broken so I re-did the main prompt myself

https://imgur.com/a/GjFCTRD

Yeah so to my eye, this makes sense. These are all women. We didn’t specify a location or time period or art style. There’s a certain kind of person who gets upset every time they see someone that doesn’t look like them where they didn’t expect it. But that’s their problem. Most of the human beings who have ever lived were not white. They might try to say it’s “statistically inaccurate” but they’re probably not pouring over real world statistics when they make such claims, and I expect that Google’s perception of statistics is more accurate than theirs.

Note that the alternative is the Google “grandmother problem” which they definitely don’t want to repeat:

http://www.socializingai.com/grandma-now-can-see-bias-data/

Except you didn't do what I suggested... You just said women.

> generate a picture of a typical Swedish women

Images are broken for me.

Weird. It seems this system still has a lot of strange quirks.

They're trying to remove the bias present in the training data. The data on the internet vastly over-represents content from wealthy countries who were connected to the internet earlier, and the AI will learn this biased distribution to the point where if you ask it to generate a picture of a "person", that person may be White 100% of the time. Attempts by Google to instead model the true underlying population distribution is probably done in a very hacky way, such as putting guidelines in the system prompt, which then leads to outcomes (which are cherry-picked and put on social media as culture war outrage bait) like what you're talking about where a "Swedish woman" occasionally doesn't show up as White.

Reply on news.ycombinator.com