Introducing deep research
openai.com
openai.com
It stated that I had 47,000 reputation points on Stack Overflow -- quite a surprise to me, given my minimal activity on Stack Overflow over the years. I popped over to the link it had cited (my profile on Stack Overflow) and it seems it confused my number of people reached (47k) with my reputation, a sadly paltry 525.
Then it cited an answer I gave on Stack Overflow on the topic of monkey-patching in PHP, using this as evidence for my technical expertise. Turns out that about 15 years ago, I _asked_ a question on this topic, but the answer was submitted by someone else. Looks like I don't have much expertise, after all.
Finally, it found a gem of a quote from an interview I gave. Or wait, that was my brother! Confusingly, we founded a company together, and we were both mentioned in the same article, but he was the interviewee, not I.
I would say it's decent enough for a springboard, but you should definitely treat the output with caution and follow the links provided to make sure everything is accurate.
This is why I don't use AI for anything that requires a "correct" answer. I use it to re-write paragraphs or sentences to improve readability etc, but I stop short of trusting any piece of info that comes out from AI.
Artificial dementia...
Some parties are releasing products much earlier than the ability to ship well working products (I am not sure that their legal cover will be so solid), but database aided outputs should and could become a strong limit to that phenomenon of remembering badly. Very linearly, like humans: get an idea, then compare it to the data - it is due diligence and part of the verification process in reasoning. It is as if some moves outside linear pure product progress reasoning are swaying the RnD towards directions outside the primary concerns. It's a form of procrastination.
You might find it amusing to compare it to: https://hn-wrapped.kadoa.com/timabdulla
It's a very interesting usecase though, filter through billions of comments and give everyone a score on which real life person they probably are. I wonder if say, Ted Cruz hides behind a username somewhere.
not just for when discussion of the content not the personality behind it is important.
That would be exactly my verdict of any product based on LLMs in the past few years.
We'd never hire someone who just makes stuff up (or at least keep them employed for long). Why are we okay with calling "AI" tools like this anything other than curious research projects?
Can't we just send LLMs back to the drawing board until they have some semblance of reliability?
Well at this point they've certainly proven a net gain for everyone regardless of the occasional nonsense they spew.
It has helped me learn stuff incredibly faster. Especially I find them useful for filling the gaps of knowledge and exploring new topics in my own way and language, without needing to wait an answer from a human (that could also be wrong).
Why does it feel, that "we are entirely inside the bubble" for you?
and then you verify every single fact it tells you via traditional methods by confirming them in human-written documents, right?
Otherwise, how do you use the LLM for learning? If you don't know the answer to what you're asking, you can't tell if it's lying. It also can't tell if it's lying, so you can't ask it.
If you have to look up every fact it outputs after it does, using traditional methods, why not skip to just looking things up the old fashioned way and save time?
Occasionally an LLM helps me surface unknown keywords that make traditional searches easier, but they can't teach anything because they don't know anything. They can imagine things you might be able to learn from a real authority, but that's it. That can be useful! But it's not useful for learning alone.
And if you're not verifying literally everything an LLM tells you.. are you sure you're learning anything real?
I can remember numerous things I was told while growing up, that aren't actually true. Either by plain lies and rumours or because of the long list of our cognitive biases.
> If you have to look up every fact it outputs after it does, using traditional methods, why not skip to just looking things up the old fashioned way and save time?
What is the old fashioned way? I mean people learn "truths" these days from Tiktok and Youtube. Some of the stuff is actually very good, you just have to distill it based on the stuff I was being taught at school. Nonody has yet declared LLMs as a subtitute for schools, maybe they soon will, but neither "guarantees" us anything. We could as well be taught political agendas.
I could order a book about construction, but I wouldn't build a house without asking a "verified" expert. Some people build anyway and we get some catastrofic results.
Levels of trust, it's all games and play until it gets serious, like what to eat or doing something that involves life threatening physics. I take it as playing with a toy. Surely something great have come up from only a few piece of legos?
> And if you're not verifying literally everything an LLM tells you.. are you sure you're learning anything real?
I guess you shouldn't do it that way. But really, so far the topics I've rigorously explored with ChatGPT for example, have been better than your average journalism. What is real?
Looking in a resource written by someone with sufficient ethos that they can be considered trustworthy .
> What is real?
I'm not arguing ontology about systems that can't do arithmetic. you're not arguing in good faith at all
The difference is you have some reassurances that the human is not wrong - their expertise and experience.
The problem with LLMs, as demonstrated by the top-level comment here, is that they constantly make stuff up. While you may think you're learning things quickly, how do you know you're learning them "correctly", for lack of a better word?
Until an LLM can say "I don't know", I really don't think people should be relying on them as a first-class method of learning.
In the early days of ChatGPT where it seemed like this fun new thing, I used it to "learn" C. I don't remember anything it told me, and none of the answers it gave me were anything that I couldn't find elsewhere in different forms - heck I could have flipped open Kernighan & Ritchie to the right page and got the answer.
I had a conversation with an AI/Bitcoin enthusiast recently. Maybe that already tells you everything you need to know about this person, but to the hammer the point home, they made a claim to similar to you: "I learn much more and much better with AI". They also said they "fact check" things it "tells" them. Some moments later they told me "Bitcoin has its roots in Occupy Wall Street".
A simple web search tells you that Bitcoin is conceived a full 2 years before Occupy. How can they be related?
It's a simple error that can be fact checked simply. It's a pretty innocuous falsity in this particular case - but how many more falsehoods have they collected? How do those falsehoods influence them on a day-by-day basis?
How many falsehoods influence you?
A very well meaning activist posted a "comprehensive" list of all the programs that were to be halted by the grants and loans freezes last week. Some of the entries on the list weren't real, or not related to the freeze. They revealed they used ChatGPT to help compile the list and then went down one-by-one to verify each one.
With such meticulous attention to detail, incorrect information still filtered through.
Are you sure you are learning?
Maybe we're already at AGI and just don't know it because we overestimate the capabilities of most humans.
If AI is teaching you, you are going to collect a thousand papercuts of lies.
Does the food taste good? Oops, there's a bit too much vegetables here, they are never gonna fit in this pan of mine. Not a big deal, next time I'll be wiser.
AI is like a hypothesis machine. You're gonna have to figure out if the output is true. Few years ago, just testing any machine's "intelligence" was pretty quickly done and machine failed miserably. Now, the accuracy is astounishing in comparison.
> How many falsehoods influence you?
That is a great question. The answer is definitely not zero. I try to live by with a hacker mentality and I'm an engineer by trade. I read news and comments, which I'm not sure is good for me. But you also need some compassion towards oneself. It's not like ripping everything open will lead to salvation. I believe the truth does set you free, eventually. But all in one's time...
Anyway, AI is a tool like any other. Someone will hammer their fingers with it. I just don't understand the hate. It's not like we're drinking any AI koolaids here. It's just like it was 30 years ago (in my personal journey), you had a keyboard and a machine, you asked it things and got gibberish. Now the conversation with it just started to get interesting. Peace.
Problem is - LLMs pull answers from their behind, just like a lazy student on the exam. "Halucinations" is the word people use to describe this.
Those are extremely hard to spot - unless you happen to know the right answer already, at which point - why ask? And those are everywhere.
One example - recently there was quite a discussion about llm being able to understand (and answer) base16 (aka "hex") encoding on the fly, so I went on to try base64, gzipped base64, zstd-compressed base64, etc...
To my surprise, LLM got most of those encoding/compressions right, decoded/uncompressed the question, and answered it flawlessly.
But with few encodings, LLM detected base64 correctly, got compression algorithm correctly, and then... instead of decompressing, made up a completely different payload, and proceeded to answer that. Without any hint of anything sinister going.
We really need LLMs to reliably calculate and express confidence. Otherwise they will remain mere toys.
We've tried for decades to turn people into reliable robots, now many companies are running to replace people robots with (maybe less reliable?) robot-robots. What could go wrong? What are the escalation paths going to be? Who's going to be watching them?
This is contrary to my experience.
Because they are a way to launder liability while reducing costs to produce a service.
Look at the AI-based startups y-combinator has been funding. They match that description.
It's given you some information and now you have to seek out a source to verify that it's correct.
Finding information is hard work. It's why librarian is a valuable skilled profession. What you've done by suggesting that I should "verify" or "proofread" what a glorified, water-wasting Markov chain has given me now entails me looking up that information to verify that it's correct. That's...not quite doubling the work involved but it's adding an unnecessary step.
I could have searched for the source in the first instance. I could have gone to the library and asked for help.
We spent time coming up with a question ("prompt engineering"! hah!), we used up a bunch of electricity for an answer to be generated and now you...want me to search up that answer to find the source? Why did we do the first step?
People got undergraduate degrees - hell, even PhDs - before generative AI.
Look up the tweet from someone who said "Sometimes when coming up with a good prompt for ChatGPT, I sometimes come up with the answer myself without needing to submit".
An entire day of generating responses with ChatGPT uses less water and energy than your morning shower. You seem terribly concerned about signaling the virtues of abstaining from technology use on behalf of purported resource misuse, yet you're sitting at a computer typing away.
You're not a serious person, and you're wasting everyone's time. Please leave the internet and go play with rocks in a cave.
Do take care.
For other things, it depends, but if I'm asking it to do a survey I can look at its results and see if they fit what I'm looking for, check the sources it gives me, etc. People pay analysts/paralegals/assistants to do exactly this kind of work all the time expecting that they will need to check it over. I don't see how this is any different.
I don't think the library/electricity responses are serious but to move on to the point about degrees... people also got those degrees before calculators, before computers, before air travel, before video calls, before the internet, before electricity, yet all of those things assist in creating knowledge. I think its perfectly reasonable to look at these LLMs/chat assistants in the same light: as a tool that can augment human productivity in its own way.
We do all the time - of course we do, all the time.
Some people might find $500 worth of value, in their specific use case, in those "great" and "ok" categories, where they get more value than "lies" out of it.
A few verifiable lies, vs hours of time, could be worth it for some people, with use cases outside of your perspective.
I wonder if it’s carried over too much of that ‘helpful’ DNA from 4o’s RLHF. In that case, maybe asking for 500 words was the difficult part — it just didn’t have enough to say based on one SO post and one article, but the overall directives assume there is, and so the model is put into a place where it must publish..
Put another way, it seems this model faithfully replicates the incentives most academics have — publish a positive result, or get dinged. :)
Did it pick up your HN comments? Kadua claims that’s more than enough to roast me, … and it’s not wrong. It seems like there’s enough detail about you (or me) there to do a better job summarizing.
It didn't pick up my HN comments, probably because my first and last name are not in my profile, though obviously that is my handle in a smooshed-together form.
A report full of factual errors that a careful intern wouldn't make is worse than useless (yes, yes, I've mentored interns).
If the hard part is the language, then do the research yourself, write an outline, and have the LLM turn it into complete sentences. That would at least be faster.
Here's the thing, though: If you do that, you're effectively proving that prose style is the low-value part of the work, and may be unnecessary. Which, as much as it pains to me say as a former English major, is largely true.
Although I think people are the same, too big problem and you are getting lost unless taking it in bites, so seems like OpenAI implementation is just bad because o3 hallucination benchmark shouldn’t lead to such poor performance
Hypothetically speaking, if the time you saved is now spent verifying the statements of your AI researcher, then did you really save any time at all?
If the answers aren't important enough to verify, then was it ever even important enough to actually research to begin with?
Most people I talk to are at the point now where getting completely incorrect answers 10% of the time — either obviously wrong from common sense, or because the answers are self contradictory — undermines a lot of trust in any kind of interaction. Other than double checking something you already know, language models aren’t large enough to actually know everything. They can only sound like they do.
What I’m looking for is therefore not just the correct answer, but the correct answer in an amount of time that’s faster than it would take me to research the answer myself, and also faster than it takes me to verify the answer given by the machine.
It’s one thing to ask a pupil to answer an exam paper to which you know the answers. It’s a whole next level to have it answer questions to which you don’t know the answers, and on whose answers you are relying to be correct.
"Limitations Deep research unlocks significant new capabilities, but it’s still early and has limitations. It can sometimes hallucinate facts in responses or make incorrect inferences"
How do I know which parts are false? It will take as long to verify as to research!
Still useful for the odd task here and there, but not as useful as all the money being invested in this (except for the companies getting that money, that is).
edit: actual example of something I'd expect a real AI to be able to solve by itself, but currently LLMs fail miserably https://x.com/RadishHarmers/status/1885884032220643587
1) Paramount task: searching in naturally structured language, as opposed to keywords. Odd tasks: oh yes, several tasks of fuzzy sophisticated text processing previously unsolved.
2) They translate NN encodings in natural language! The issue remains about the quality of /what/ they translate in natural language, but one important chunk of the problem* is in a way solved...
Now, I've been probably one of the most vocal here, shouting "That's the opposite of intelligence!" - even in the past 24 hours -, but be objective: there are also progresses ...
(* Around five years ago we were still stuck with Hinton's problem of interpreting pronouns as pointers in "the item won't fit in the case: it's too big" vs "the item won't fit in the case: it's too small" - look at it now...)
edit: furthermore, LLMs probably tackle very little "real state" in the "make machines THINK" land. But a crucial piece on the overall puzzle.
I always thought there was a kind of NP-flavor to the problems for which LLMs-like AI are helpful in practice, in the sense that solving the problem may be hard but checking the solution must be fast.
Unless the domain can accomodate errors/hallucination, checking the solution (by a human) should be exponentially faster than finding it (by some AI) otherwise there's little practical gain.
A year back that number was 30%, and a couple of years back it was 60%. There will be a point where it'll be good enough. There are also better and better ways to verify answers these days.
It'll never be a solution for everything, but that's similar to many engineering problems we have: for example, ORMs aren't great for all types of queries, but they're sufficient for a good part of them.
The meaningful discussion is about how they perform NOW and the edge cases that have persisted since GPT-2 which no one has yet found a good solution for.
I disagree though, it is useful as this problem has been whittled down and I think there is expectation that there will be continued effort. Its of course worth discussing but I find that for my workflows, I rarely encounter issues with hallucinations, they certainly exist but its gotten to a point that I don't have major issue with it.
This is why I haven't found AI tools very useful. I find my self spending more time verifying and fixing it's answers than I would have just doing or learning the darn thing myself.
The challenge with something like this for research, in its current state, is you’ll need to go double check it because you don’t trust it and it will end up effectively being a list of links.
It’s progress though and evidently good enough to find a sweet NSX in Japan, which is all some really need.
More importantly, I think, is that they are incapable of not doing so. Have we figured out how to make an LLM realize and answer that it doesn't know an answer?
I'd say they don't know anything.
An LLM base model, before it is post-trained with RL, just has access to a sliced and diced corpus of human output. Take the contents of 4chan and WikiPedia, put in blender and mix and chop into "training sample" sized bites, then learn the statistical regularities of this blended mess. It is what it is - not exactly what I'd call a knowledge base, even though there are bits of knowledge in there.
When you add RL-based post-training for reasoning, all you are doing is trying get the model to be more selective when you are sampling from it - encouraging it to suppress some statistics, and emphazise others, such that when you sample from it the output looks more like valid reasoning steps and/or conclusions, per the verified reasoning examples you train it on.
I'm well aware of how useful RL-tuned models (whatever the goal) can be, but at the end of the day all they are doing is taking a statistical babbler and saying "try to output patterns more like this". It's not exactly a recipe for factuality or rationality - we've just gone from hallucination-prone base models, to gaslighting-prone RL-tuned "reasoning" models that output stuff that sounds like coherent reasoning.
What missing from all of this - what makes it different from how animals learn - it that the model has no experience of it's own, no autonomy or motivation to explore, learn and verify, and hence no episodic memories of how it learnt something (tried it and ran controlled experiments, or just overheard it on the bus), and what that implies about it's trustworthiness.
It's amazing that LLMs work as well as they do, a reflection of how much of what we do can be accomplished by reactive pattern matching, but if you want to go beyond that to something that can learn and discern the truth for itself, this seems the wrong paradigm altogether.
I've seen this thinking to be especially prevalent among tech types (especially among many comments on this site), and more so than among average non-tech people I know.
It seems to be a reflection of a certain forced, almost ideological techno-reductionist thinking against the honestly complex and largely mysterious nature of consciousness. Many non-tech people on the other hand accept this mystery of consciousness and paradoxically are thus less likely to consider an LLM to be anything deeper than the clever but mindless pattern-matching trick that it is
Management Consulting was bad enough before the ability to mass produce these graphs and stats on a whim. At least there was some understanding behind the scenes of where the numbers came from, and sources would/could be provided.
The more powerful these tools become, the more prevelant this effect of seepage will become.
Oh Sweet summer child.
------
Hey, I've noticed a few things in your style that are both strengths and opportunities for improvement:
Strengths:
- You clearly have deep knowledge and back up your points with solid data and examples.
- Your confidence and detailed analysis make your arguments compelling.
Opportunities:
- At times, your tone can feel a bit combative, which might shut down conversation.
- Focusing on critiquing ideas rather than questioning someone's honesty can help keep the discussion constructive.
- A clearer structure in longer posts could make your points even more accessible.
Overall, your passion and expertise shine through—tweaking the tone a bit might help foster even more productive debates.
------
Just reply here if you want the full 500+ words analysis that goes into more detail.
I think this is a real problem for these AI tools. If you can’t separate the signal from the noise, it doesn’t provide any real value, like an out of range FM radio station.
Things aren't quite so black and white in reality.
Catching errors/validating work is obviously a different process when they're coming from an AI vs a human, but I don't see how it's fundamentally that different here. If the outputs are heavily cited then that might go someway into being able to more easily catch and correct slip-ups
Previously there was alot of stress/pressure which might or might not have led to sloppy work (some consultants are of a high quality). With this, there will be no stress which will (always?) lead to sloppy work. Perhaps there's an argument for the high quality consultants using the tools to produce accurate and high quality work. There will obviously be a sliding scale here. Time will tell.
I'd wager the end result will be sloppy work, at scale :-)
Same problem I have with code models, honestly. We already have way too much boilerplate and bad code; machines to generate more boilerplate and bad code aren't going to help.
Consultants aren't the ones doing the fact-checking, that falls to the client, who ironically tend to assume the consultants did it.
Lots of kids don’t want to do homework. That said, previously many would because there wasn’t another choice. But now they can just ask ChatGPT for the answers they’ll write that down verbatim with zero learning taking place.
Caring isn’t a binary thing or works in isolation.
Sure, but if you're a professional you have to care about your reputation. Presenting hallucinated cases from ChatGPT didn't go very well for that lawyer: https://www.nytimes.com/2023/05/27/nyregion/avianca-airline-...
Doing a proper fact check is as much work as doing the entire research by hand, and therefore, this system is useless to anyone who cares about the result being correct.
> I don't see why this changes.
And because of the above this system should not exist.
Same thing as misinformation. A sufficient quantitative difference becomes a qualitative difference at some point.
The majority of human written consultant reports are already complete rubbish. Low accuracy, low signal-to-noise, generic platitudes in a quantity-over-quality format.
LLMs are innoculating people to this kind of low information value content.
People who produce LLM quality output, are now being accused of using LLMs, and can no longer pretend to be adding value.
The result of this is going to be higher quality expectations from consultants and a shaking out of people who produce word vommit rather than accurate, insightful, contextually relevent information.
I suspect this can be done simply by poisoning a query with supplemental suggestions of sources to use in a RAG, many of which don't even have to be publicly available but are made accessible to the LLM (perhaps by submitting hidden URLs that mislead the summary along with the query).
But even after such a practice is uncovered and roundly maligned, that won't stop the infinite supply of net con men from continuing to inject their poisons into the background that drives deep research, so long as the LLM maker doesn't actively oppose this practice actively and publicly -- which none of them have been willing to do with any other LLM operational details so far.
In fact, I predict that if a LLM summary like DR's does NOT soon provide references to the sources of the facts it relies on, in no time users will disregard such summaries to be yet more uselessly unreliable pfaff from yet another net disreputable -- as we do with search engine summaries now.
Meta question: what's with all of the naming overlap in the AI world? Triton (Nvidia, OpenAI) and Gro{k,q} (X.ai, groq, OpenAI) all come to mind
Now with LLMs it is what human evaluators feel about the LLM output?
I think you have to prefix the query with "@Meta AI", hope this helps
"PS: The name Triton was coined in mid-2019 when I released my PhD paper on the subject. I chose not to rename the project when the "TensorRT Inference Server" was rebranded as "Triton Inference Server" a year later since it's the only thing that ties my helpful PhD advisors to the project."
They seem to be ok with outsourcing any and all creativity to a language model, so it’s not surprising that they can’t come up with unique names themselves.
We found in our experience it is pretty hard to force LLM to do something in proper depth, and OAI's deep research definitely feels like one of the first examples from big labs on how this can be done. What we typically see is that it is not even the "agent" part that is hard to do, but how to force model to not "forget" to go deep...
Would have loved to try it when they released it, but I'm apparently in the wrong country. I think it's not available outside the US (?). OpenAI and DeepSeek have no such issues. It's a bummer really, I'm happy paying for this but they don't want me to.
Effectiveness in this task environment is well beyond the specific model involved, no? Plus they'd be fools (IMHO) to only use one size of model for each step in a research task -- sure, o3 might be an advantage when synthesizing a final answer or choosing between conflicting sources, but there are many, many steps required to get to that point.
I wonder how much of an impact our being still so early in the productization phase of this all is. Like it takes a ton of work and training and coordination to get multiple models synced up into an offering and I think the companies are still optimizing for getting new ideas out there rather truly optimizing them.
OpenAI is very much in an existential crisis and their poor execution is not helping their cause. Operator or “deep research” should be able to assume the role of a Pro user, run a quick test, and reliably report on whether this is working before the press release right?
It's the only tool/system (I won't call it an LLM) in their released benchmarks that has access to tools and the web. So, I'd wager the performance gains are strictly due to that.
If an LLM (o3) is too expensive to be released to the public, why would you use it in a tool that has to make hundreds of inference calls to it to answer a single question? You'd use a much cheaper model. Most likely o3-mini or o1-mini combined with o4-mini for some tasks.
The same reason a lot of people switched to GPT-4 when it came out even though it was much more expensive than 3 - doesn't matter how cheap it is if it isn't good enough/much worse.
What makes you believe that?
> Powered by a version of the upcoming OpenAI o3 model that’s optimized for web browsing and data analysis, it leverages reasoning to search, interpret, and analyze massive amounts of text, images, and PDFs on the internet, pivoting as needed in reaction to information it encounters.
If that's what you're referring to, then it doesn't seem that "explicit" to me. For example, how do we know that it doesn't use less thinking than o3-mini? Google's version of deep research uses their "not cutting edge version" 1.5 model, after all. Are you referring to something else?
Can someone test it with this prompt.
"As a research assistant with comprehensive knowledge of particle physics, please provide a detailed analysis of next-generation particle collider projects currently under consideration by the international physics community.
The analysis should encompass the major proposed projects, including the Future Circular Collider (FCC) at CERN, International Linear Collider (ILC), Compact Linear Collider (CLIC), various Muon Collider proposals, and any other significant projects as of 2024.
For each proposal, examine the planned energy ranges and collision types, estimated timeline for construction and operation, technical advantages and challenges, approximate costs, and key physics goals. Include information about current technical design reports, feasibility studies, and the level of international support and collaboration.
Present a thorough comparative analysis that addresses technical feasibility, cost-benefit considerations, scientific potential for new physics discoveries, timeline to first data collection, infrastructure requirements, and environmental impact. The projects should be compared in terms of their relative strengths, weaknesses, and potential contributions to advancing our understanding of fundamental physics.
Please format the response as a structured technical summary suitable for presentation at a topical meeting of particle physicists. Where appropriate, incorporate relevant figures and tables to facilitate clear comparisons between proposals. Base your analysis on information from peer-reviewed sources and official design reports, focusing on the most current available data and design specifications.
Consider the long-term implications of each proposal, including potential upgrade paths, flexibility for future modifications, and integration with existing research infrastructure."
Hopefully helpful, happy to test others for you :)
-- Raw output --
Understood. I will conduct a detailed technical analysis of next-generation particle collider projects, focusing on the Future Circular Collider (FCC), International Linear Collider (ILC), Compact Linear Collider (CLIC), Muon Collider proposals, and other significant initiatives.
The report will include: - Planned energy ranges and collision types - Estimated construction and operation timelines - Technical advantages and challenges - Approximate costs - Key physics goals - Review of technical design reports and feasibility studies - International collaboration and support - Comparative analysis on feasibility, cost-benefit, scientific potential, infrastructure needs, and environmental impact - Long-term considerations, including upgrade paths and integration with existing facilities
The analysis will be formatted as a structured technical summary, suitable for presentation at a topical meeting of particle physicists, with comparative tables and figures where relevant.
I will update you when the report is ready.
# Next-Generation Particle Collider Projects: A Comparative Technical Analysis
## Introduction High-energy physics is charting plans for new colliders to extend beyond the LHC and its High-Luminosity upgrade (HL-LHC). The discovery of the 125 GeV Higgs boson has motivated proposals for a dedicated “Higgs factory” lepton collider to study its properties with precision, as well as future energy-frontier machines to search for new physics. Several next-generation collider initiatives are under consideration internationally as of 2024. These include CERN’s proposed *Future Circular Collider (FCC)*, Japan’s *International Linear Collider (ILC)*, CERN’s *Compact Linear Collider (CLIC)*, various designs for a *Muon Collider*, China’s *Circular Electron-Positron Collider (CEPC)* and its successor *Super Proton-Proton Collider (SppC)*, among others. Each proposal differs in collision type (electron-positron, proton-proton, muon-muon, etc.), energy scale, technology, timeline, cost, and physics focus. This summary reviews each project’s key parameters – *planned energy ranges, collision types, timeline, technical advantages/challenges, cost, and physics goals* – based on technical design reports and feasibility studies. A comparative analysis then contrasts their *technical feasibility, cost-benefit, scientific potential for discoveries, timeline to first data, infrastructure needs, and environmental impact*, highlighting the relative strengths and weaknesses of each approach. We also discuss long-term implications such as upgrade paths, flexibility for future modifications, and integration with existing infrastructure.
(Citations refer to official reports and peer-reviewed sources using the format 【source†lines】.)
## Future Circular Collider (FCC) – CERN - *Type and Energy:* The FCC is a *proposed 100 km circular collider* at CERN that would be realized in stages. The first stage, *FCC-ee*, is an electron-positron ($e^+e^-$) collider with center-of-mass energy tunable from ~90 GeV up to 350–365 GeV, covering the Z boson pole, WW threshold, Higgs production (240 GeV), and top-quark pair threshold (~350 GeV). A second stage, *FCC-hh*, would use the same tunnel for a proton-proton collider at up to *100 TeV* center-of-mass energy (an order of magnitude above the LHC’s 14 TeV). Heavy-ion collisions (e.g. Pb–Pb) are also envisioned. An *FCC-eh* option (electron-hadron collisions) is considered by adding a high-energy electron injector to collide with the proton beam. This integrated FCC program thus spans both *precision lepton* collisions and *energy-frontier hadron* collisions.
- *Timeline:* The conceptual schedule foresees *FCC-ee construction in the 2030s* and a start of operations by around *2040* (as the LHC/HL-LHC program winds down). According to the FCC Conceptual Design Report, an $e^+e^-$ Higgs factory could begin delivering physics in ~2040, running for 15–20 years. The *hadron collider FCC-hh* would be constructed subsequently (using the same tunnel and upgraded infrastructure), aiming for *first proton-proton collisions in the late 2050s】. This staged approach (lepton collider first, hadron later) mirrors the successful *LEP–LHC sequence*, leveraging the $e^+e^-$ machine to produce great precision data (and to build infrastructure) before pushing to the highest energies with the hadron machine. ...
(Too long for HN to write more)
What I like is that it asked you before clarifying questions before but I wonder if it just generic. Because the prompt mentioned that this would be for "presentation at a topical meeting of particle physicists" but still asked its last question about
> Intended Audience: Should the analysis assume a general physics audience or a more specialized group of particle physicists?
Also probably expected but it didn't include or reference graphs/plots.
Reasoning, problem solving, research validation - at the fundamental outset it is all refinement thinking.
Research is one of those areas where I remain skeptical it is that important because the only valid proof is in the execution outcome, not the compiled answer.
For instance you can research all you want about the best vacuum on the internet but until you try it out yourself you are going to be caught in between marketing, fake reviews, influencers, etc. maybe the science fields are shielded from this (by being boring) but imagine medical pharmas realizing that they can get whatever paper to say whatever by flooding the internet with their curated blog articles containing advanced medical “research findings”. At some point you cannot trust the internet at all and I imagine that might be soon.
I worry especially with the rapidly changing landscape of the amount of generated text in the internet that research will lose a lot of value due to massive amounts of information garbage.
It will be a thing we used to do when the internet was still “real”.
Moravec detailed the idea of a landscape of human capabilities slowly being submerged by AI capabilities, and the point at which AI can do anything a human can, in practice or in principle, we'll know for certain we've reached truly general AI. This idea includes things like feeling pain and pleasure, planning, complex social, oral, and ethical dynamics, and anything else you can possibly think of as relevant to human intelligence. Deep Research is just another island being slowly submerged by the relentless and relentlessly accelerating flood.
can machine feel? without that there is no AGI according to definition above.
and the second question: are animals "GI"? they don't have language and don't solve math problems, never heard of np-complete.
Feelings are nothing very special you know...
That depends entirely on how you choose to define "AGI".
So you wouldn't use this tool for those types of use cases.
But still, a valid point. I recall I once wanted to compare Hydroflask, Klean Kanteen and Thermos to see how they perform for hot/cold drinks. I was looking specifically for articles/posts where people had performed actual measurements. But those were very hard to find, with almost all Google hits being generic comparisons with no hard data. That didn't stop them from ranking ("Hydroflask is better for warm drinks!")
Would I be able to get this to ignore all of those and use only ones where actual experiments were performed. And moreover, filter out duplicates (e.g. one guy does an experiment, and several other bloggers link to his post and repeat his findings in their own posts - it's one experiment but with many search results).
> In Greek mythology, who was Jason's maternal great-grandfather?
https://www.google.com/search?q=In+Greek+mythology%2C+who+wa...
ideally a model would be able to answer this accurately and completely.
For example, Jason's mother was Philonis, daughter of Mestra, daughter of Daedalion, son of Hesporos. So Jason's maternal great-grandfather was Hesporos.
pass rate really only matters in context of the difficulty of the tasks
I mean I too can complain that my iPhone doesn’t automatically screen out spammers and send my mom flowers on Mother’s Day.
Pixel phone launched in 2016.
And eyeballing the benchmarks, it'll probably reach a >50% rate per query by the end of the year. Seems to double every model or two.
The set of natural transformations between two functors F,G :C→DF,G:C→D can be expressed as the end Nat(F,G)≅∫AHomD(F(A),G(A)). Nat(F,G)≅∫A HomD (F(A),G(A)).
Define set of natural cotransformations from FF to GG to be the coend CoNat(F,G)≅∫AHomD(F(A),G(A)). CoNat(F,G)≅∫AHomD (F(A),G(A)).
Let: - F=B∙(Σ4)∗/F=B∙ (Σ4 )∗/ be the under ∞∞-category of the nerve of the delooping of the symmetric group Σ4Σ4 on 4 letters under the unique 00-simplex ∗∗ of B∙Σ4B∙ Σ4 . - G=B∙(Σ7)∗/G=B∙ (Σ7 )∗/ be the under ∞∞-category nerve of the delooping of the symmetric group Σ7Σ7 on 7 letters under the unique 00-simplex ∗∗ of B∙Σ7B∙ Σ7 .
How many natural cotransformations are there between FF and GG?
Also I'm curious as to what percentage of the questions in this benchmark are of this type / difficulty, vs the seemingly much easier example of "In Greek mythology, who was Jason's maternal great-grandfather?".
I'd imagine the latter is much easier for an LLM, and almost trivial for any LLM with access to external sources (such as deep research).
could also be aggressive caching from cloudflare. could be they're just trying to announce more stuff to maintain cachet and can't yet support all users forking over 200/month.
OpenAI is very much in an existential crisis and their poor execution is not helping their cause. Operator or “deep research” should be able to assume the role of a Pro user, run a quick test, and reliably report on whether this is working before the press release right?
They do say "Currently, deep research can access the open web...", so maybe "open" there implies something significant. Like, "websites that have agreements with OpenAI and/or do not enforce norobot policies".
I wouldn’t even be surprised if a law is passed requiring sites to provide equal access to humans whether accessed directly or via these models.
It’s too important an innovation to stall, especially considering the US’s competitors (China) won’t respect robots.txt either.
Amazon listings are blocked from google shopping and other price comparison sites.
I see Amazon results there all the time. 3 of the visible 8 sponsored results are Amazon, in the non-sponsored results an Amazon listing is either first or second in every category.
How would you know its a crawler?
Not sure if I'm too tired and can't see it but the lack of images/examples of the resulting report in this announcement doesn't inspire a lot of confidence just yet.
Still I think there is a big market for this sort of „go away for 30 mins and figure this out“ style agent
All the AI commercials are designed to appeal to people that don’t produce any actual value but haven’t been detected by the system yet.
Need to send email to boss? Press magic button! Job well done, idiot.
Someone send you big scary email? Press magic button! Good job dummy!
Someone wants to go eat some Italian with you, push magic button for totally not-ad result. Enjoy your Olive Garden, moron.
https://blog.google/products/gemini/google-gemini-deep-resea...
Oh and "Deep research" is available to people on the $200 per month plan? Lol - cool. I've been using DeepSeek a lot more recently and it's so incredibly good even with all the scaling issues.
Though, the jump for Gaia relative to SOTA is relatively not that high. Especially given that this is o3
The examples they have on the page work well on chat.deepseek.com with r1 and search options both enabled.
Do I blindly trust the accuracy of either though? Absolutely not. I'm pretty concerned about these models falling into gaming SEO and finding inaccurate facts and presenting them as fact. (How easy is it to fool / prompt inject these models?)
But has utility if held right.
AFAIK OpenAI's current offering uses 4o, and it does a web search and then pipes it into 4o. I'm guessing adding CoT + other R1/o3 like stuff is one of the key effective differences. But time will tell how different it is. Maybe it's a dramatic improvement.
I've used Code Interpreter to compile and run C code - https://simonwillison.net/2024/Mar/23/building-c-extensions-... - and I've managed to get it to run JavaScript (by uploading a Deno binary) and even Lua and PHP in the past as well: https://til.simonwillison.net/llms/code-interpreter-expansio...
Aren't there more than one articles that did not mention plasmons or plasmonics in Scientific Reports in 2012?
Also, did they pay for access to all journal contents? that would be useful
HN has a huge cultural problem that makes this website almost irrelevant. All the interesting takes have moved to X/twitter
Maybe you can define what "AGI" really means and what the end-game and the economic implications are when 'AGI" is some-what achieved? OpenAI somehow believes that they haven't achieved "AGI" yet, which they continue to do this on purpose for obvious reasons.
The first hint I will give you is that it certainly won't be a utopia.
More seriously, it’s unclear why one should be excited by the prospect of AGI, especially when instrumentalized by corporations and authoritarian governments.
Man, I've got a great deal on some oceanfront property in Wyoming for you.
I don't feel like OpenAI does a good job of getting me excited either.
Find the perfect snowboard? How can that idea get pitched and make the final cut for a $200 a month service? The NFL kicker example is also completely ridiculous.
The business and UX example seems interesting. Would love to see more.
https://chatgpt.com/share/67a0d59b-d020-8001-bb88-dc9869d52b...
I do use these systems from time to time, but it just never renders any specific information that would make it great research.
These systems serve best at augmenting information discovery. When I'm tackling a new area or looking for the right terminology, these models provide a quick shortcut because they have good probabilistic "understanding" of my naive, jargon-free description. This allows me to pull in all of the jargon for the area of research I'm interested in, and move on to actually useful resources, whether that be journal articles, textbooks, or - rarely - online posts/blogs/videos.
the current "meta" is probably something like Elicit + notebookLM + Claude for accelerating understanding of complex topics and extracting useful parts. But, again, each step requires that I am closely involved, from selecting the "correct" papers, to carefully aggregating and grooming the information pulled in from notebookLM, to judging the the usefulness of Claude's attempts to extract what I have asked for
I thought funding was the biggest barrier to great research
Synthesize? Seems like the wrong word -- I think they would want to say something like, "analyze, and synthesize useful outputs from hundreds of online sources"..
> combine (a number of things) into a coherent whole: pupils should synthesize the data they have gathered | Darwinian theory has been synthesized with modern genetics.In fact, it's been three month since the release and I wouldn't even be surprised if OpenAI took inspiration from it.
What I did was to ask Bing ai about that topic and it returned information aswell as sources to where it found those, so I picked up all those links and researched them myself.
Bing ai was a great resource for finding relevant links, this was until I found out about perplexity, my life haven't been the same since.
Granted, these strike me as difficult tasks and I’d likely ask it to do far simpler things, but I’m not really sure what to expect from looking at these graphs.
Ah, but the fact that it bothers to cite its sources is a huge plus. Between that and its search abilities it sounds valuable to me
AskPandi has a similar feature called "Super Search" that essentially checks more sources and self validates it's own answers.
iT's AgEnTic.
The answers are easier to digest, if you search for products, you'll get a list of products with images, prices and retailers.
To be clear I'm constantly impressed with what they have and what I get as a customer, but the delivery since 4 hasn't exactly been in line with Altman's Musk-tier vapoware promises...
No one if ever going to trust the output this generates and then needs to spend so much time fact checking that they might as well do the entire research from scratch themselves...
I highly doubt that the concerns of internal political commissars were holding up this particular openai release.
> when Google released Gemini to have a product in the space.
Bard preceded Gemini.
That’s probably a feature not a bug (from OpenAI’s perspective…).
Does this mean they skipped supervised fine tuning like DeepSeek did with R1?
This plateau behavior reflects something deeper about our current approach to AI. We've built transformer architectures partly inspired by simplified observations of human cognition - particularly how our brains use attention mechanisms to filter and process information. And like human attention, these models have inherent constraints: each attention layer normalizes scores to sum to 1, creating a fixed "attention budget" that must be distributed across all inputs.
A recent paper (https://arxiv.org/abs/2501.19399) explores this limitation, showing how standard attention becomes increasingly diffuse with longer contexts. Their proposed "Scalable-Softmax" helps maintain focused attention at longer ranges, but still shows diminishing returns - pushing the ceiling higher rather than eliminating it.
But here's the deeper question: As we push toward AGI and potentially superintelligent systems, should we remain bound by architectures modeled on our current understanding of human cognition? The human brain's limited attention mechanism evolved under specific constraints and for specific purposes. While it's remarkably effective for human-level intelligence, it might be fundamentally limiting for artificial systems that could theoretically process information in radically different ways.
Looking at the Deep Research results through this lens, the plateau might not just be a technical limitation to overcome, but a sign that we need to fundamentally rethink how artificial systems could process and integrate information. Instead of trying to stretch the capabilities of attention-based architectures, perhaps we need to explore entirely different paradigms of information processing that aren't constrained by biological analogues.
This isn't to dismiss the remarkable achievements of transformer architectures, but rather to suggest that the path to AGI might require breaking free from some of our biologically-inspired assumptions. What would an architecture that processes information in ways fundamentally different from human cognition look like? How might it integrate and reason about information without the constraints of normalized attention?
Would love to hear thoughts from others working on these problems, particularly around novel approaches that move beyond our current biological inspirations.
Nonetheless, I don't think this is even something that can easily be benchmarked. I'd recommend you take a look at aider [1], and consider how I drew similarities between it and what's presented here.
Has ClosedAI presented any benchmarks / evaluation protocols?
So far I tried it on one problem and it seems limited by the "front end" being 4o-mini. It ignored most of my initial prompt and also ignored the previous research it asked for which I provided. The final output was high quality and definitely was enriched by the web searching it did, but it left out a crucially important dimension of the problem because it was unable to ingest the background info I provided adequately.
I'd like to see a version of it where the front end model is o1-pro
Different pre-cooked prompts and filters don’t really amount to new products either, despite them being marketed as such. It’s like adobe treating each tool in photoshop as its own product.
Agents that can search the internet exist for a while now and have been essentially solved and happily used in platforms like Perplexity.
It's really "meh", very far from revolutionary.
Keep in mind this company is trying to convince everybody they need 500B USD now (through the Stargate project).
To say this is trivial is like saying the one shot ai prompted twitter clone is the same thing as twitter.
Peak HN indeed.
Without internet: 10%
With internet: 23%
In addition:
> We found that the ground-truth answers for one dataset were widely leaked online
in very small letters, and they blocked these URLs at runtime but not training time.
It's not bad, but not revolutionary at all compared to the leap that was GPT-2 from GPT-3, or GPT-4o to DeepSeek-R1
Again: the assertion was yours, so let us know the results of your own work.
Few people care or even know about various advances in various specialized fields. It's enough if AI simply seeps into various applications in boring and non-flashy ways for it to have significant effects that will affect a wider range of people, whether they get hyped by the news announcements or not. Jobs etc.
An analogy: the Internet as such is not very exciting nowadays, certainly not in the way it was exciting in the 90s with all the news segments about surfing the information superhighway or whatever. There was a lot of buzz around the web, but then it got normalized. It didn't disappear, it just got taken for granted. No average person got excited around HTML5 or IPv6. It just chugs along in the background. AI will similarly simply build into the fabric of how things get done. Sometimes visibly to the average person, sometimes just behind the scenes.
Within minutes, it generated a detailed, well-cited research report that significantly expanded my original analysis, covering: * Legal precedents & case law interpretations (including a nuanced breakdown of UETA Section 10). * Comparative international frameworks (EU, UK, Canada). * Real-world technical implementations (Stripe’s AI-driven transaction handling). * Industry perspectives & business impact (trust, risk allocation, compliance). * Emerging regulatory standards (EU AI Act, FTC oversight, ISO/NIST AI governance).
What stood out most was its ability to: - Synthesize complex legal, business, and technical concepts into clear, actionable insights. - Connect legal frameworks, industry trends, and real-world case studies. - Maintain a business-first focus, emphasizing practical benefits. - Integrate 2024 developments with historical context for a deeper analysis.
The depth and coherence of the output were comparable to what I would expect from a team of domain experts—but delivered in a fraction of the time.
From the announcement: Deep Research leverages OpenAI’s next-generation model, optimized for multi-step research, reasoning, and synthesis. It has already set new performance benchmarks, achieving 26.6% accuracy on Humanity’s Last Exam (the highest of any OpenAI model) and a 72.57% average accuracy on the GAIA Benchmark, demonstrating advanced reasoning and research capabilities.
Currently available to Pro users (with up to 100 queries per month), it will soon expand to Plus and Team users. While OpenAI acknowledges limitations—such as occasional hallucinations and challenges in source verification—its iterative deployment strategy and continuous refinement approach are promising.
My key takeaway: This LLM agent-based tool has the potential to save hours of manual research while delivering high-quality, well-documented outputs. Automating tasks that traditionally require expert-level investigation, it can complete complex research in 5–30 minutes (just 6 minutes for my task), with citations and structured reasoning.
I don’t see any other comments yet from people who have actually used it, but it’s only been a few hours.I’d love to hear how it’s performing for others. What use cases have you explored? How did it do?
(Note: This review is based on a single use case. I’ll provide further updates as I conduct broader testing.)
[1] https://www.dazzagreenwood.com/p/ueta-and-llm-agents-a-deep-...
The first was on a topic with which I am deeply familiar -- myself -- and it made three factual errors in a 500-word report: https://news.ycombinator.com/item?id=42916899
The second was a task to do an industry analysis on a space in which I worked for about ten years. I think its overall synthesis was good (it accorded with my understanding of the space), but there were a number of errors in the statistics and supporting evidence it compiled, based upon my random review of the source material.
I think the product is cool and will definitely be helpful, but I would still recommend verifying its outputs. I think the process of verification is less time-consuming than the process of researching and writing, so that is likely an acceptable compromise in many cases.
Or is the position of OpenAI that Wiles' proof is incomplete?
**with browsing + python tools
Maybe we have different definitions of scaling?
Seriously, for the past 20+ years it's hard to imagine doing research without Google platform namely Google Search, Scholar, Patent and Book, but now it seems agent AI based on LLM is the way to. In twenty years in the future it will be hard to imagine that doing research without them. But as many people already pointed out Google probably the best company by far to perform this emerging AI based research. In data eco-system terms (refer to any book on data engineering), Google has already perform has the most important data preparation and data engineering upstream activities including data ingestion and transformation. Now given their vast amount of processed data they can just serve it to downstream data analytics or AI for performing research with minimum error/hallucinations as possible. According to Google there is no moat for any companies against open source LLM, but if any company that can has the moat it will be Google itself.
AI’s triumph in the white collar workplace will be gradual, not instantaneous. And it will be grimly quiet, because no one likes white collar workers the way they like blue collar workers, for some odd reason, and there’s no tradition of solidarity among white collar workers. Everyone will just look up one day and find that the local Big Corp headquarters is…empty.
Anyone who's done any kind of substantial document research knows that it's a NIGHTMARE of chasing loose ends & citogenesis.
Trusting an LLM to critically evaluate every source and to be deeply suspect of any unproven claim is a ridiculous thing to do. These are not hard reasoning systems, they are probabilistic language models.
This is like arguing an Airbus cannot possibly fly because it is 165 tonnes of aluminum, steel and plastic.
The proof is in the fact that it flies, not what it is constructed from.
And LLMs do not.
> "But it looks like reasoning to me"
My condolences. You should go see a doctor about your inability to count the number of 'R's in a word.
CoT reasoning is reasoning, whether you like it or not. If you don't understand that, it means the models are already smarter than you.
Stealing from thieves is fine by me. Sama was the one claiming that all information could be used to train LLMs, without permisdion of the copyright holders.
Now the same is being done to openai. Well, too bad.
OpenAI and other LLMs scraping the internet is probably covered under fair use. DeepSeek’s violation of OpenAI’s terms is pretty clearly a violation of their terms and not legal.
Here is a new thing you learn today, ToS are not laws, you can ignore any ToS and at worst the company might close your account.
Meanwhile, their entire training corpus was the result of scraping the intellectual property and copyrighted materials of THE ENTIRE PUBLIC INTERNET.
Woe is them to be sure.
I want to understand does this mean it's ethical for anyone to create a research AI tool that will go through arXiv and related GitHub repo and use it to solve problems, implement ideas like cursor.
Seriously begging the mods to take a closer look, or at least PG to not abandon his curated internet space.
Yes, they can do some cool tricks, and tool calling is fun. No one should trust the output of these models, though. The hallucinations are bad, and my experience with the "reasoning" models is that as soon as they fuck up (they always do) they go off the rails worse than the base LLMs.
Now they are scrambling against open-source after their disastrous operator demonstration and using this deep research demo as cover. Nothing that Google or Perplexity could not already do themselves.
By the end of them month, this feature is going be added by a bunch of other open-source projects and this feature won't be as interesting very quickly.
Of course. It is in response to their disastrous operator demo which did not justify the $200 per month ChatGPT Pro subscription on top of the release of DeepSeek to make matters worse for them.
> This has nothing to do with open weight models like DeepSeek (note: DeepSeek, Llama, etc are NOT open source).
It obviously does. Even before they rushed this presentation, they made o3-mini available for ChatGPT free users so it in direct response to DeepSeek.
> This feature doesn’t just require the research on the model but also enormous compute. Plus anyone using such a feature for real work is not going to be using DeepSeek or whatever, but a product with trustworthy practices and guarantees.
Nothing that Perplexity + DeepSeek-R1 can already do.
So what is your point?
OpenAI is very much in an existential crisis and their poor execution is not helping their cause. Operator or “deep research” should be able to assume the role of a Pro user, run a quick test, and reliably report on whether this is working before the press release right?
This isn’t a single model. Almost the entire leadership team around sama has left and almost certainly agrees with me on this. OpenAI’s business model is not sustainable.
can sometimes hallucinate facts in responses or make incorrect inferences, though at a notably lower rate than existing ChatGPT models, according to internal evaluations. It may struggle with distinguishing authoritative information from rumors, and currently shows weakness in confidence calibration, often failing to convey uncertainty accurately
These tools are just good at creating pollution. I don't see the point of delegating a (not just) research where 1% blatant mistakes are acceptable. These need much better grounding before handing out to masses.
I can not take any output by these tools (google summaries, comment summaries by amazon, youtube summaries etc etc) while knowing for a fact some of that is a total lie. I can not tell which part is a lie. e.g. If LLM says that in any given text the sentiment is divided, it could be just one person with an opposing view.
If same task was given to a person, I could reason with that person on any conclusion. These tools will reason on their hallucinations.