Hacker News Reader

Comment by leumon | Hacker News Reader

It fails the simple clock test for me which Qwen3.8 27B got (nearly) right. given an image of a clock https://files.catbox.moe/kgwa5e.png

I asked it "what time does the clock show?" (both on reasoning: high)

DS answered: The clock shows *5:10* (and 45 seconds). Here is the breakdown: * *Hour hand (red, shortest):* Pointing at the *5*. * *Minute hand (green, longest):* Pointing at the *2*, which represents 10 minutes. * *Second hand (blue, medium):* Pointing at the *9*, which represents 45 seconds.

Qwen answered: The clock shows *8:10* (with the red second hand on the 5, i.e. *8:10:25*).

- *Hour hand* (short, blue) → 8 - *Minute hand* (long, green) → 2 (10 minutes) - *Second hand* (thin, red) → 5 (25 seconds)

Correct answer is 08:09:25.

View on HN

I’ll keep that in mind next time I need to tell what time it is by asking an llm to read an analog clock.

Snark aside, I’m not sure that these gotcha tests are any more useful than asking politicians gotcha questions. Sure, the model can’t tell me what time it is, but it can code the Wang algorithm for noisy audio matching in one shot. Maybe this is just me being an optimist, but this is my hiring philosophy and I guess maybe now my llm philosophy: I’m not interested in seeing how dumb I can make you look, I’m more interested in how smart you can be.

Reading any analog clock at any time level (edit: and a non-noisy vector rendered image at that) is absolutely table stakes for an allegedly frontier flagship vision model. As much as 1:1 OCR. If the model can't do that, there's something wrong. Doesn't matter if it's memorized some random thing you think is esoteric but is in all the training data and benchmarks.

The whole point of LLM/FMs vs good old fashioned ML is generalization to unknown domains, not just unknown tasks. The hunt for "gotchas" is the hunt for "not in your training data".

Is this an “alleged frontier flagship vision model”?

This is described as a brand new flash model - still experimental - from a lab that is a side project for an investment firm that has never had a vision model before. That doesn’t scream flagship or frontier to me.

I disagree. It's not even that useful to train LLMs to read an ancient analog clock.

Unless we're talking about AGI, I couldn't care less if an LLM is bad at things they won't be doing anyway.

I'd rather focus training data on more useful tasks.

It's not that useful for a person to be able to read an analog clock either, but if a supposed-genius came to me and confidently gave the wrong answer, it would say something about their strengths/weaknesses in general.

The whole point of these models is they're meant to be able to generalize fairly well, not just answer questions the got trained on.

Funny, I just ran the test myself and DS4 got it right first try.

Then I ran it 9 more times and it got right 9 out of 10 times.

Lesson learned to double check what I read on the internet.

It's because the messaging for what the point of these things is supposed to be is all over the place. Ask 10 different people and you'll get 10 different answers:

- A superintelligence that will usher in an age of human enlightenment

- A superintelligence that will usher in an age of human enslavement

- A really cool way to rake in trillion of rich VC/investor money by promising you're building a superintelligence that will usher in an age of human en[slave/lighten]ment

- A transformer model for predicting output tokens given a series of input tokens, informed primarily by reddit, stack overflow, and 6000 years of classical literature.

- A replacement for white collar labor. Start now or join the permanent underclass.

- A convenient fuzzy-find tool also capable of some probably-correct code generation.

- The ultimate customizable text RPG experience (you can pick if G stand for game or...)

And so on.

So, some people see a new model and check for how close humanity is to enslavement. Some people check to see if it got better at fixing broken unit tests.

What's amazing is that all of these are true at once. If you allow for some significant slack in what "superintelligence" means.

Where did deepseek claim any of that?

I find collecting “gotcha” questions helpful because when a new model can answer them, it shows improvement, and if it doesn’t, it’s a reality check that, despite the model being helpful for many tasks, there are other things it can’t do yet. It’s a demonstration of “jagged intelligence.”

It doesn’t have to be a negative thing! Simon’s pelican on a bicycle prompt is an example of a “gotcha” question.

How is this a "gotcha" question? "gotcha" implies there's some sort of trick. This is just... a question.

Knowing where it fails is just as important as knowing where is excels.

It is like asking a politician how much a coffee costs, to show how disconnected they are from common people. Super intelligence not being able to read a simple analog clock does the same.

> Sure, the model can’t tell me what time it is, but it can code the Wang algorithm for noisy audio matching in one shot

This is about a _vision_ model.

An _experimental_ _flash_ _vision_ variant of a model optimized for speed and cost.

The problem with judging a politician for not knowing what a cup of coffee costs is that it doesn’t tell you anything about his capabilities or why he doesn’t know. Maybe the politician is incredibly competent, but doesn’t drink coffee. Maybe he gets up at 5 to start working and makes a pot of coffee himself before the coffee shops open so knows what a bag of coffee costs but not a cup.

The issue with a gotcha question is that it is a shortcut to confirmation bias. There is no single correct answer to what a cup of coffee costs, and not knowing the answer doesn’t tell you as much as people want it to. I can go to Starbucks and get a cup of “coffee” that costs $10, or I can go to the corner store and get one for $.99. I can make one at home for $.50 or less. There’s no right answer.

Scroll through the comments and there is someone that tried the same clock test on this model and it answered correctly 9/10 times. That’s why catching someone or something in a gotcha once is kinda bad signal.

Can this model tell time? Yes, and also no. Does a cup of coffee cost .99? Yes and also no.

It's still useful to find things it can't do if anything so we can tell when it starts being able to do them.

I was literally working on an educational game for my kids last week where one of the activities is clock reading, and I ask codex to QA its Godot program via screenshots, so literally this exact scenario is something I was doing in a software engineering context. It can of course write code to figure out the angles to rotate by just fine, but it also needs to be able to figure out whether the whole picture comes together, whether the hand sprites are anchored on the clock face correctly with the right pivot, etc.

Is being asked to read a clock really a gotcha?

Not if you are aiming at a general intelligence but it’s worth considering that this is a tool that may not be able to count the number of strawberries in the letter R but can still center a div.

It is, apocryphally, a gotcha test for Gen-alpha. As is being able to return the correct change or make a phone call to a human being. Reminds me of the iRobot line: "Can you create a symphony?", "No. Can you?".

If I had to hire an engineer and there was one that could one shot the wang algorithm, but couldn’t read an analog clock, I would have no problem hiring them.

Also worth noting that both models got it wrong. Qwen made a mistake that humans very good at reading clocks would make. Deepseek made a mistake that a human who had just learned to read clocks would make.

It would be different if AI was known to be reliable but it isn’t, so this is less of a random failure and more a symptom of jagged intelligence.

And with every one of these there’s always an attempt to minimize the problem by saying it’s just one silly failure.

Well it succeeded on first try for me:

https://i.imgur.com/gljOYr9.png

    [Image 1] what time does the clock show?
    
    + Thought: 368ms
    
    The clock shows 8:10.
    
    - Blue hour hand: just past 8
    - Green minute hand: pointing at 2 (10 minutes)
    - Red second hand: pointing at 5 (25 seconds)
    ▣  Plan · DeepSeek V4 Flash Vision Exp · 4.4s

edit: I now ran the test 10 times total, fresh session every time, and DS4 got it right 9 out of 10 times. Lesson learned to double check what I read on the internet.

I'll leave a copy of the clock image for posterity here in case anyone wants to test it themselves: https://i.imgur.com/BQsfa3R.png

Try it on openrouter directly if you don't believe me that i got this result. Maybe image inputs behave differently in opencode, or it could even be that this agent harness uses different models for different tasks, not sure. Or maybe the reasoning (high) makes the difference.

I don't doubt you. I just think it's good practice to run more than n=1 before conclusions are made. We're all engineers here.

fwiw the clock reads 8:09:25 not 8:10:25 although it would be passable to read it as 8:10 if there were no second hand

This is not a normal looking clock - most clocks have either one color for all hands (second hand is thinnest and maybe also longest) or one color for hour/minute and one for second. I know that the hand lengths and thicknesses on this image are correct but for some reason I, a totally human person who grew up when analog clocks were still common, see this and think the hand on the 5 is the minute hand. How does the AI do if you just make all the hands black?

then deepseek answers: "The clock shows 8:25. The short hour hand is pointing to the 8, and the long minute hand is pointing to the 5, which represents 25 minutes."

and qwen still answers: "The clock shows *8:10* (with the second hand on the 5, i.e., 25 seconds). - *Hour hand* points to the 8 - *Minute hand* points to the 2 (= 10 minutes) - *Second hand* points to the 5 (= 25 seconds) So the time is *8:10:25*, or simply *8:10*."

Qwen still got the wrong answer, though.

Are we more forgiving because it’s the same type of mistake a human would make?

A good share of humanity would have also gotten this question wrong!

It's been four years that we are looping those

"The professional failed its task!" // "Laymen would have failed it too".

Which makes no sense.

Yeah, I heard most kids these days can't read analog clocks either.

I can't actually remember where I learned to read a clock, it might have actually been in school. I guess that means they don't teach it anymore. (Everyone's phone shows the time anyway...)

I would say, try without thinking on. I find reasoning on any rag type request seems to increase hallucinations, probably due to the thinking tokens taking attention away from the, in this case, vision tokens.

I'd recommend non-thinking for any non-prompt input, and leave the thinking where it has to actually reason.

Gemini 3.7 Flash and 5.6-Sol (on all reasoning levels) also answer 8:10:25. The new "stealth" Ox Alpha also replies with the same. Opus 5 replies with 8:10 (no seconds). Not sure why this is so hard for them; Gemini is especially good at vision and I would have expected better from it.

I was wasting hours yesterday trying to get DeepSeek V4 Flash (with Qwen 3.8 27b as the vision agent, actually) to read sheet music to pass a Terminal Bench 3 benchmark and none of it was working... nothing... I changed models to gemma 31b, I tried OCR models... nothing could get it...

And then I realized, wait a second... you're testing the harness not only against a difficult benchmarking problem, but it's one you're literally never going to use the coding harness for either, lol. I don't write programs that read or interact with sheet music and I never will.

tl;dr Being frustrated that a "state of the art" vision model doesn't have perfect vision is a fools errand.

It can read and extract information from screenshots and PDFs just fine (my setup). No need to worry about edge cases.

most likely a preview. they often release the preview via API, get more training data, post train some more then release the weight. i would expect to see it perform better in a few weeks or a month.

welp, damning indictment. not sure if that means DS is super crap, or qwen is super good

Neither. Performance of all models is incredibly spikey.

Reply on news.ycombinator.com