Graph-based AI model maps the future of innovation
news.mit.edu
news.mit.edu
The thing about Beethoven's 9th and biological materials which is mentioned in the OP is just that, out of a very large knowledge graph, they found small subgraph isomorphic to a subgraph created from a text about the symphony. But they seem not to cover the fact that a sufficiently large graph with some high-level statistical properties would have small subgraphs isomorphic to a 'query' graph. Is this one good or meaningful in some way, or is it just an inevitable outcome of having produced such a large knowledge graph at the start? The reader can't really tell, because figure 8 which presents the two graphs has such a poor resolution that one cannot read any of the labels. We're just expected to see "oh the nodes and their degrees match so it has the right shape", but that doesn't really tell us that their system had any insight through this isomorphism-based mining process.
For the stuff about linking art (e.g. a Kandinsky painting) with material design ... they used an LLM to generate a description of a material for DALL-E where the prompt includes information about the painting, and then they show the resulting image and the painting. But there's no measure of what a "good" material description is, and there certainly is no evaluation of the contribution of the graph-based "reasoning". In particular an obvious comparison would be to "Describe this painting." -> "Construct a prompt for DALL-E to portray a material whose structure has properties informed by this description of a painting ..." -> render.
It really seems like the author threw a bunch of stuff against the wall and didn't even look particularly closely to see if it stuck.
Also, the only equation in the paper is the author giving the definition of cosine similarity, before 2 paragraphs justifying its use in constructing their graph. Like, who is the intended audience?
https://iopscience.iop.org/article/10.1088/2632-2153/ad7228#...
- real output here is text, using a finetuned Mixtral provided leading Qs
- the initial "graph" with the silly beethoven-inspired material is probably hand constructed, they don't describe its creation process at all
- later, they're constructing graphs with GPT-3.5 (!?) (they say rate limits, but somethings weird with the whole thing, they're talking about GPT-4 vision preview etc., which was roughly a year before the paper was released)
- Whole thing reads like someone had a long leash to spend a year or two exploring basic consumer LLMs, finetune one LLM, and sorta just published whatever they got 6 months to a year later.
Publish and perish...
I wrote more below with a quote, but re: "who's the intended audience?" I think the answer is the same kind of people Gary Marcus writes for: other academic leaders, private investors, and general technologists. Definitely not engineers looking to apply their work immediately, nor the vast majority of scientists that are doing the long, boring legwork of establishing facts.
In that context, I would defend the paper as evocative and creative, even though your criticisms all ring true. Like, take a look at their (his?) HuggingFace repo: https://huggingface.co/lamm-mit It seems clear that they're doing serious work with real LLMs, even if it's scattershot.
Honestly, if I was a prestigious department head with millions at my disposal in an engineering field, I'm not sure I would act any differently!
ETA: Plus, I'll defend him purely on the basis of having a gorgeous, well-documented Git repo for the project: https://github.com/lamm-mit/GraphReasoning?tab=readme-ov-fil... Does this constitute scientific value on its own? Not really. Does it immediately bias me in his favor? Absolutely!
Looking forward to a more serious effort.
Likely not the only visitor with that experience. Just a heads up.
This is not serious.
The article itself seems generated.
More likely, this author read a bit too much Deleuze and is echoing that language to make the discovery feel more important than incidental.
https://app.gitsense.com/?doc=4715cf6d95689&other-models=Cla...
Note, sentences highlighted in yellow means one or more models disagree.
The sentence that makes me think this might not be AI generated is
"Researchers can use this framework to answer complex questions, find gaps in current knowledge, suggest new designs for materials, and predict how materials might behave, and link concepts that had never been connected before."
The use of "and" before "predict how materials" was obviously unnecessary and got caught by both gpt-4o and claude 3.5 sonnet and when I questioned Llama 3.5 about it, it also agreed.
For AI generated, it seems like there are too many imperfections, which makes me believe it might well be written by a human.
Run your papers through AI and have them identify simple corrections. It's like having an endlessly patient English Literature major at your beck and call.
https://app.gitsense.com/?doc=696357b733b
which contains enough grammatical errors that I'm pretty sure it was not generated by AI.
https://app.gitsense.com/?doc=381752be7fd0&prompt=Is+AI+Gene...
Edit:
I created another prompt that tries to better analyze things and they (models) all agree that it is most likely AI (+60%). The highest was gpt-4o-mini at 83%.
https://app.gitsense.com/?doc=381752be7fd0537&prompt=Is+AI+G...
The simpler explanation makes more sense: knowledge graphs naturally show certain structural properties, and these properties appear across domains due to basic mathematical constraints, common organizational principles, and human cognitive patterns reflected in data. Sure, LLMs trained on human knowledge can identify these patterns, generate plausible narratives, and create appealing connections - but this doesn't necessarily indicate novel scientific insights, predictive power, or practical utility.
If you find yourself going down a rabbit hole like this (and trust me, we've all been there), my advice is to ask "is there a simpler explanation that I'm missing?" Then start from square one: specific testable hypotheses, rigorous controls, clear success metrics, practical demonstrations, and independent validation. And maybe add a "complexity budget" - if your explanation requires three layers of recursive AI analysis to make sense, you're probably way too deep in the sauce.
He said they were producing too many engineers and not enough scholars. When alumni offered to endow the program (in case it was a funding issue) he refused our donations.
Which makes this “scholarship” of chaining together some GPT prompts especially insulting.
Also there are also certain aspects of physical geometric relationships and even sound relationships that would not be able to be conveyed to an AI by any other means than thru art and music. So definitely using art to inspire science is a good approach.
Even the great Physicists throughout history have often appreciated how there is indeed beauty in the mathematical symmetries and relationships exhibited in the mathematics of nature, and so there is definitely a connection even if not quite tangible nor describable by man.
> The application could lead to the development of innovative sustainable building materials, biodegradable alternatives to plastics, wearable technology, and even biomedical devices.
That a transform from materials to a 19th century Russian painter somehow is applicable to what just so happens to be the zeitgeist of materials science beggars belief.
Markus J. Buehler is the McAfee Professor of Engineering and former Head of the MIT Department of Civil and Environmental Engineering at the Massachusetts Institute of Technology. He directs the Laboratory for Atomistic and Molecular Mechanics (LAMM), leads the MIT-Germany program, and is Principal Investigator on numerous national and international research program... [he] is a founder of the emerging research area of materiomics. He has appeared on numerous TV and radio outlets to explain the impact of his research to broad audiences.