Argonne National Lab is attempting to replicate LK-99
science.org
science.org
But as an experimental chemist who got a PhD and works at a national lab, if the twitch/Twitter streamers don't produce a semiconductor, it says nothing
Experiments and processes are hard and particular. You're expecting to produce a nanoscale material through macroacale processes. Chemistry can be like producing a microchip with a ballpeen hammer. "I hit it seventy times in thirty seconds in 3.8mL acetone and it just forms a single crystal structure. Do not hit it seventy-one or sixty-nine times."
Chemists are experts at clearly explaining their procedure while leaving out the meaningless keystone detail that will only take a year or two to suss out if you're a clever experimentalist.
And this isn't a clearly outlined procedure.
But, again, I think the best case scenario is to watch an ambitious amateur do materials synthesis and see it work less perfectly than he intended. Maybe he and they stick around for the other 5 or 10 or 150 weeks as he does the same thing again to make sure the problem is nature rather than technique.
If bet-placing is the fashion, shall we place bets about tenacity and thoroughness?
As someone who sucks at languages, I can appreciate that learning new languages does not come easy to everybody, or even to most people, especially in older age. If it seems like all reputable scientists have some English proficiency, that's a selection effect--within the English-speaking world in particular you're not going to bump into or even hear much about those who cannot speak or write English. AFAIU there's a substantial amount of high quality research in, e.g., Japan that is invisible to non-Japanese speakers. Japan's scientific establishment long predates English as a global language, unlike China; and unlike many European countries but similar to the United States there was never a widespread norm of learning foreign languages even among the professional classes, AFAIU. Though Korea is a younger industrialized country, I wouldn't take for granted that the principal discoverers have any English proficiency, especially considering that they would be late middle-age or older if they were working in the lab in 1999.
I've known so many researchers who apologize every other sentence because they say they've only read English and I would never had known otherwise.
Which is to say - it'd be wierd to say "well I don't know English and I don't know anybody who does oh well". But, anyway, the papers were published on aeXive in...
Plenty of people would learn Latin to consume some content, but doesn't mean they are proficient in it itself, would be another example.
Having said that, I think that in the research field that merely ends up with "very confusingly written papers with mistakes large enough to hide process errors" more than "cannot communicate any info in English". But there are plenty of people who get their doctorate and their only English-language output is their final papers in themselves.
The paper is just a guide. Who would replicate likely needs to contact the authors to understand the implicit details of the recipe. That is process knowledge. There's a good reason world-leading lithography happens in that region. With luck, researchers and engineers will be able to refine and industrialize the process these coming years.
Japanese and Korean professionals speak English. There will be one researcher for whom explaining the language is a minor dalliance.
I work in a very minor polymer materials field. There are two Japanese researchers who cheerfully banter. If somebody had hidden some transformative work behind their language, even as a student, they would have talked about it straightaway, if only for the conversation and a bit of competitive intellectual football. Can the foolish American grad student reason around it? Let's see!
The twitch channel was packed with interested Koreans. If there had been the tiniest bit of confusion there would have been 200 translations from which to utilize the Translation Of The Crowd.
Of course the Japanese scientists you've met speak English. People who don't speak English well enough are not generally going to participate in events or situations where they would need to rely on English-language speaking skills. That someone apologies for their bad English is, of course, irrelevant; if it were truly bad you wouldn't have encountered them. Likewise, Japan has many international science programs, forums, and conferences; and they'll be filled with proficient Japanese English speakers, but Japan is a huge nation so it says little necessarily about broader patterns.
I don't doubt that a majority of professional research scientists in Japan have some proficiency in English. Especially the top scientists; even in Japan "impact" would undoubtedly be heavily biased toward publication in prestigious international journals. But there's plenty of literature on the internet that describes the relatively poor English-language proficiency (especially conversationally or with verbiage outside that in their technical field) and poor English-speaking research community integration of the broader Japanese scientific establishment. See, e.g., "Japanese materials scientists' experiences with English for research publication purposes".
I first encountered this cultural dynamic when researching the tilting of the Millennium Tower. While friction piles weren't first used in Japan, Japan was where friction piles were first heavily researched and employed for large buildings, including skyscrapers. People kept saying friction piles were idiotic in earthquake-prone, sandy soils, but that's precisely the environment where they were first researched and from whence the global engineering community's confidence sprouted. However, AFAICT much of that research was published in Japanese, even through the late 20th century and (IIRC) even early 21st century. Was that Nobel-worthy science? No, but it was very useful applied science; and applied science which was highly consequential to global engineering and architecture, notwithstanding that much of the substance of it existed on the other side of a Japanese language barrier.
[1] See https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8024886/
[2] But see, e.g., China and Brazil where domestic publication dominates and even outpacing international publication: https://www.digital-science.com/blog/2018/11/japan-collabora...
[3] E.g. many of the publications by the The Japan Institute of Metals and Materials are only or predominately in Japanese. Some domestic publications require summaries and tables in English but permit the text to be in Japanese; e.g. the Journal of the Society of Materials Science.
Friction pilings have been in use in NL for 100's of years.
If it's been 1000 experiments, do another. I'm not convinced that 1000 experiments over 24 years is all that intense if you actually do the math.
I suspect if they really did 1,000 experiments then they often did 2/wk because I don't think they ran all of them continuously over 20+yrs.
Speaking of which, can we see their notes on the other 999 experiments that didn't work out? That would be very useful, especially if LK-99 turns out not to be awesome but is good enough to hint that this approach is worthy of more research -- there's 999 compounds not worth testing, right? (Well, maybe maybe. If the physics of LK-99 come to be understood, then maybe some of the other 999 experiments might need to be re-examined.)
They didn't have a theory to drive them to doggedly pursue this system as an SC. Was it just the blip 20 years ago? Why wouldn't you wonder if your machine was shitty?
If I was them, its decades past believing this might be an empty rabbit hole. You're committed. If it's real, it's heroic. If not? It's a failure of rationality. But it's both simultaneously.
If they're right, I'm sure every notebook will be published. If not? We'll never see their fury or process.
For a humble mechanical engineer its very helpful to make hammer analogies.
It's too complex to be perfectly predictable, like particle physics
But without instruments,you have to hope the nanoscale changes are slow and visible in solution at 1,000,000x larger scale
The fact that we can throw shit in a beaker and identify useful reactions is a hell of a thing. Our magic balls are the only thing that take it much beyond it's origins in alchemy.
One of his favorite movies was It Happens Every Spring (1949), about a chemist who produces a non-reproducible compound.
A huge part of science is being a knowledgeable and rigorous experimentalist. You can do every great for years, but you still have to do things right 25 years in or you'll be unable to sleep
If you're referring to Andrew McCalip, he's been very clear that he is not claiming to be doing a serious replication attempt, but he also does zero-g/orbital chemical synthesis for a living so its not quite fair to characterize him as a random twitch streamer.
Arxiv/preprints are peer review
You can upload whatever trash methods you want, but a normal journal will have at least one guy who tells you to wipe your ass and make your bullshit presentable in public, if only because they're expected to be gatekeepers for a minimum standard of supposed reproducibility.
And it is still silly that people call it "peer review." Peer review is not 3-4 randos briefly glancing my work in an adversarial setting who say my work is not novel because it is the same as some unrelated work that they didn't read either; peer review is the grad student building on top of my work, peer review is lucidrains rewriting my work from scratch, it is Ross Wightman integrating it into timm and retraining, it is the forks that use my work for projects (hobby or professional). Peer review is peers looking and reviewing. More peer review happens on Twitter than these conferences. You can say these conferences and journals are a form of peer review, but we gotta stop saying that just because something is a preprint that it isn't peer reviewed. That's just incorrect. Peer review is when peers review.
I'm a chemist, who has considerable established competition. So our reviwers know the systems and studied the method in grad school and sweated over it as a postdoc and had to innovate for cash in a professional capacity while doing something other than what we were doing.
They'd excoriate for vague methods, or poor explanations, or general nonsense. You had to explain everything and make sure to refence the Big boys you were close to but definitely not stealing from.
I think ML is unformed enough that there isn't that sort of public stricture as a random field in chemistry. Feel lucky. We don't even get to hide behind beautiful LaTeX, because there's too much benchwork for anybody's boss to ever give a fuck about it.
That sounds problematic
> there isn't that sort of public stricture as a random field in chemistry. Feel lucky.
It's pure noise over here. I don't feel lucky, I feel frustrated. I get 3 reviewers and 3 completely different reasons to reject, often nonsensical (not joking, recently was rejected because a broken reference link to the appendix). Then submit to another conference and it is another 3 reviewers with 3 different reasons to reject that are completely different from the previous ones. Often asking me why I didn't cite arxiv papers that were released after the submission deadline. ACs don't care as long as reviewers agree because they gotta keep that acceptance rate low. It's playing the fucking lottery except your graduation and career depends on it despite people frequently saying conferences don't matter they'll still critique on lack of top tier publications.
Your system doesn't sound great and also sounds frustrating, but I'll trade you. Regardless, I don't have faith in either system to significantly determine if a work is good (I do believe they can identify bad works, just not good works).
Could one problem with these kinds of reviews be that the reviewers have absolutely zero skin in the game? Like, you can write complete nonsense and nobody will blame you personally for it, you have no stake in the outcome of the submission/paper what so ever, and you're much more incentivized to come up with reasons to reject than to say "good enough". I realize the latter part can be good in theory, as it sets a high bar, but it also often feels like this goes awry when not kept in check somehow.
Actually, it is worse. They probably have skin in the game, and incentives to reject! Most CS fields publish through a conference system. This is a zero or one shot system (if you get a rebuttal phase, _if_ you want to count that). These conferences have acceptance rates that they "need to maintain" to keep their rankings. In other words, you are competing against all other papers being submitted to the same conference, not just papers with similar topics. Even if it is only a little, rejecting a paper actually increases the odds that your submission makes it through.
But yeah, you're in the right ball park. This is also why you will quite frequently see conference review guidelines blatantly violated and why you will see area chairs and metareviewers just not care. They all have incentives to encourage rejection, let alone be impartial.
I think the system works when the community is small and there is accountability among peers. Accountability creates a dampening effect on bad behavior. But at scale, you only need a few bad actors to setup a feedback system and to not just spoil the entire barrel, but the entire shipment.
Edit: I should also add that there's an additional negative incentive. You are not judged by how often you review, how many reviews you perform, or how good your review is (hard to measure). So reviewing ends up taking away time from the very limited time you have to do work that you are actually evaluated on. This is likely why reviews are so rushed. There's a feedback loop too, since many will see that others rush when reviewing their work so they get tired and end up rushing when reviewing the work of others. "If they aren't going to give my work their time, why should I have their work my time?" thinking grows.
The people who review your papers in conferences and ask you why you didn't cite future arxiv papers are the same people who put their work on arxiv and cite each other's preprints. You can't rely on the process of "peer-review" on arxiv any more than you can rely on the conference peer-reviews because they're performed by the same people, and they're people who don't know what they're doing.
The sad truth is that the vast majority of the researchers in the machine learning community haven't got a clue what the hell they're doing, nor do they understand what anyone else is doing. The typical machine learning paper is poorly motivated, vaguely written, and makes no claims, nor presents any results, other than "our system beats some other systems". As to reproducibility, hell if we know whether any of that work is really reproducible. Everybody who references it ends up doing something completely different anyway and they just cite prior work as an excuse to avoid doing their job and properly motivating their work. The people who write those papers eventually get to be reviewers (by sheer luck), or sub-reviewers. They have no idea how to write a good paper, so they have no idea how to write a good review, either. And they couldn't recognise a good paper if it jumped up and bit them in the cojones.
I love to cite Geoff Hinton on this one:
GH: One big challenge the community faces is that if you want to get a paper
published in machine learning now it's got to have a table in it, with all
these different data sets across the top, and all these different methods
along the side, and your method has to look like the best one. If it doesn’t
look like that, it’s hard to get published. I don't think that's encouraging
people to think about radically new ideas.
Now if you send in a paper that has a radically new idea, there's no chance
in hell it will get accepted, because it's going to get some junior reviewer
who doesn't understand it. Or it’s going to get a senior reviewer who's
trying to review too many papers and doesn't understand it first time round
and assumes it must be nonsense. Anything that makes the brain hurt is not
going to get accepted. And I think that's really bad.
So the problem is not arxiv or not arxiv, the problem is that peers in peer review lack expertise and knowledge and they can't do their job well.
So I'd ask you do one of two things, either:
- Review a work with the diligence and care that you wish someone would give to you
or
- Don't review
I'd also appreciate it if you openly recognized how stochastic the system is and that if/when you become in a position where you need to evaluate someone, that you remember this and take it into consideration. It has a lot of value to you too, since if the metric is extremely noisy it doesn't provide you value to heavily rely upon that metric. Look for others.
But, people pay me to do a job. It's not in my contract in any clear terms, but to do my job well I need to do all those things; and I like doing my job well.
I don't have a problem citing arxiv works. I mean I have no faith in the official system, so why wouldn't I? But this is a clear indication of an impossible bar to pass and demonstrates the ridiculousness of the system. It isn't just that there's a malicious or dumb reviewer, it is that the other reviewers don't call them out, the AC doesn't call them out, and the metareviewer doesn't call them out. The best case was that broken link person and the AC made them update their response, but this was in the response to my 1 page rebuttal (for 4 reviewers) where they got to say whatever they wanted and I had no chance of response. (Their initial review was literally 2 lines) I don't know how anyone can see this and not think that the system absolutely failed at every level. You're probably unsurprised to hear that this is a common occurrence rather than uncommon.
> the vast majority of the researchers in the machine learning community haven't got a clue what the hell they're doing
I'm well aware. __I__ have no idea what I'm doing, but at least I can tell you the difference between probability and likelihood or that tuning on the test set is information leakage. Research often requires venturing into the unknown and unexplored. That's fine. I don't care if we're all stumbling around in the dark. I do care when people are not just unable to admit it, but unable to recognize this. But that's the classic "tell a lie enough times and you'll start to believe it" situation.
I am in full support of that Hinton quote. This is the first I've heard it (or recall at least), but I often say quite similar things (in fact, just did in another thread). I do mean it when I say that our current system harms us and I'm confident that we won't get to AGI with this system.
> the problem is that peers in peer review lack expertise and knowledge and they can't do their job well.
I won't disagree with this point, but I believe that this is a systematic problem rather than an individual. The system encourages this behavior rather than stamps it out. So in that sense, I think people are doing their job very well. It is just that I don't think their job actually aligns with the intent of the job. Classic case of irony, that the group of people that highly discusses alignment is one of the worst at this. But I guess we shouldn't be surprised given that lately we've seen how unethical we've seen people who write about ethics are.
I do want to add one thing though. A good paper is hard to recognize. A bad paper is easy, but a good paper may be indistinguishable from a bad paper. This is the "paradox" of research and something people need to take to heart. That is if we want to align our job descriptions with our actual jobs.
Oh, in that sense I'm clueless too. We all are. What I'm pointing out though is that many people in machine learning research don't really understand what's the point of research, or even what's the point of a research paper. They copy each other's writing style and produce the same kind of low-information-content paper.
There's been a huge influx of new people to machine learning research in recent ish years. I think it started around 2012 and the ImageNet results. I did my Master's in 2014-15 (part-time) and I was surprised by the number of people in the machine learning class I took, and how many of them had nothing to do with computer science or AI, and were instead coming from backgrounds in business or more rarely science. At least I'd expect the people with the science backgrounds to know how to write a paper, but it seems most people who come to machine learning from, say, physics or biology, bring with them a facility with calculus and continuous mathematics, but not much in terms of scientific methodology.
Hinton hints at that in the rest of his comment I didn't quote:
What we should be going for, particularly in the basic science conferences, is radically new ideas. Because we know a radically new idea in the long run is going to be much more influential than a tiny improvement. That's I think the main downside of the fact that we've got this inversion now, where you've got a few senior guys and a gazillion young guys.
Well I guess the "gazillion young guys" need a bit of time to figure out how to write good papers, and review them.
>> I do want to add one thing though. A good paper is hard to recognize. A bad paper is easy, but a good paper may be indistinguishable from a bad paper. This is the "paradox" of research and something people need to take to heart. That is if we want to align our job descriptions with our actual jobs.
That's true, and part of the reason why reviewing is hard work. I still find it very hard to reject a paper. What if I'm the one who doesn't get it?
> That sounds problematic
Bingo, it is problematic. It's part of how "the Big boys" get their academic and compensation (investment in their firms) rewards. You have to play the publish-or-perish, paper rank game to get ahead and stay ahead in academia, and this leads to all sorts of problems. Authors don't want to question "the Big boys" because that leads to their papers not getting published because the "the Big boys" and their bootlicker wannabes are the reviewers and they will exact their tribute. Make it to "the Big boys" club and now you're a gatekeeper and now you're also responsible for perpetuating this system.
It's why Nature-style peer review needs to become a thing of the past.
I'm not saying that "popular" (for a value of "popular" that involves peers at large, not the public at large) peer review is / will be without problems. But it seems to me that it will -at least for a while- be less corruptible.
The other hill I will die on is that we shouldn't refer to journal/conference publishing as "peer review." This is one form of peer review, but there are MANY more. And as far as I'm concerned, 3 randos that briefly skim my paper in an adversarial setting (zero sum) looking to reject works barely constitutes peer review. Peer review is what happens when your peers read your work, test it, build upon it, replicate it, etc. We need to stop this language because it helps no one.
I would add: when your peers' review commentary is public.
If journals posted first-submission comments for published papers I think there's be a radical change in comments. For good or ill, I have no idea.
Much as giving credit is important, this is unfortunately also how you end up with intro sections littered with references to the same ten papers everyone's read already just to back up some extremely vague/general statement.
No, this isn't a survey paper. So it is serving no purpose other than greasing their peers.
The history of science is a sequence of profound platitudes, each time refining the prior ones. It is a part of awakening to pick the lowest hanging fruits first, just to get a foot in the door. While its not theoretically impossible to go straight from ignorance to -say- the postulates of quantum field theory, its incredibly unlikely to scale such a huge step at once, both for scientists at the frontier as well as each generation of students catching up. In this sense the profound platitudes are quasi necessary to build up an understanding of the world around us. Individually each outdated platitude may appear so very wrong when looking back with 20 / 20 hindsight, and most of these individual platitudes could plausibly have been skipped by scaling 2 or 3 steps at a time, but no one could scale all of them at once.
Literally respect and review mean the same thing. To view or observe anew (re-).
One may objectively claim that each step or correction of a prior platitude was an improvement of our worldview, even though in hindsight each step of correction also contributed the next erroneous stumbling block to be scaled.
Throughout history policy decisions (from household decisions, to court decisions, to budget decisions, to research direction decisions, etc...) humans sought ways to settle matters.
The desire for Finality is universal and justified. However the expectation that this universe came with a manual is not.
A core tenet of the scientific mindset is the recognition that this finality is a temporary illusion.
Selling this illusion of a recipe for finality is profitable and what journals and gatekeepers have gravitated toward, especially since information became dirt cheap to duplicate.
I have no problem citing, and actively encourage it. I do have a problem with someone rejecting my work because I failed to cite a work which would have been impossible for me to cite. The key is that it was released (as a preprint) AFTER my paper. Not only is it in bad taste to make concurrent works a requisite (happy to update btw), but it is ludicrous to require that I have a time machine.
This isn't about respect and me complaining about pomp and circumstance. This is me complaining about the frequency that I have been criticized for not having a time machine.
The point is that a system that allows people to reject works due to lack of access to a functioning time machine is not a system we should support.
Perhaps you confused me for xorbax, or you were still in the stream of consciousness of discussing with him: if you double-check which of your comments I replied to I think that you'll find it is not the one where you mention the demand for a citation that was simply non-existent.
It appears you believed my comment to be a reply to: https://news.ycombinator.com/item?id=36917506
while in fact I replied to: https://news.ycombinator.com/item?id=36917126
I would appreciate if you re-read your own comment at the last link, and observe the second paragraph you wrote about peer review, and then re-read my correctly placed comment. I think you'll find my comment to make more sense, and in my opinion ties in nicely with your second paragraph. I was not tryihg to summarize you, merely appending my gripes with the supposed finality and atomicity of the Journals redefinition of "peer review".
In reality the real peer review is in the back and forth discussions within the community, spread out over many papers, and sometimes even over generations.
As an oversimplified example consider relativistic mechanics correcting Newtonian mechanics: if we anachronistically pretend Newton published his Principia in some Respectable Journal under the current implementation of "peer review", then it would have been accepted with apparent Finality under peer review, even though it would eventually be repudiated by relativity generations later; this artificial example should make anyone pause about the insinuated Finality of "peer review", effectively forcing one to recognize that the current concept of artificially muzzled "peer review" through the arbitrary selection of the reviewers by journal editors does not constitute true peer review. I'm not saying it's useless to have an initial screening, but then it needs a better name, perhaps "Quality Control" or "Publication Criteria" or "Discipline Specific Methods and Publication Criteria" etc... Before the modern incantation where reviewing by peers is made artificially atomic and pseudo-final, anyone in the field could provoke questions about specific claims. The illusion of Finality works discouraging: where-as in the past you would feel compelled to respond to any criticism, the current concept of "peer review" implies "it already happened since its published" and nothing forces an author to come up with a rebuttal for a criticism. While this may help prop up appearances of the scientific field with an aura of Finality of conclusions made in its domain, its really damaging to the scientific process.
Back to the comment confusion:
I've seen this happen to others when a sub-thread gets a lot of attention, resulting in a high rate of replies, where one of the authors gets confused which exact comment was being replied to. I would be lying if I were to claim it never happened to me ;)
> In reality the real peer review is in the back and forth discussions within the community, spread out over many papers, and sometimes even over generations.
This is 100% what I believe and I'm surprised others don't. Like how arxiv exists because people were already sharing papers outside the walls of journals. For some reason people are acting like this sharing means these are drafts rather than the actual publications themselves. As if arxiv and what's published are different versions. Maybe a lot of non researchers are participating in these discussions and assuming things?
I'm sure you know this, but others might not. Newton is the perfect example here because many of the works at these times were "published" years or even decades after the original work was done. Which is the whole Leibniz and Newton debate thing. I'd also highly recommend the podcast "Opinionated History of Mathematics". It's hilarious and well done. By a mathematician but for the general nerdy public
https://intellectualmathematics.com/opinionated-history-of-m...
Triple integrals and sigmas over sets, instead of clearly stated limits, and that are not analytically solvable, is not "presentable". Scientists do this stuff on purpose to make their papers harder to read.
> expected to be gatekeepers for a minimum standard of supposed reproducibility
They don't enforce this at all. Otherwise they would demand code and training data be released for ML papers.
Not just that, but if your paper has math that the reviewer doesn't understand they're more likely to think the work is good and rigorous. It's not like they read it anyways.
> and training data be released for ML papers.
Other than checkpoints and hyper-parameters, what do you want? The wandb logs? I do try to encourage people to save all relevant training parameters in checkpoints (I personally do). This even includes seeds.
Hyperparameters yes, but also the data used for training. I should be able to reproduce the checkpoint bit-for-bit by training from scratch. If their training process is not deterministic, also release the random seed used.
All ML training is non-deterministic. That's kinda the point. But yeah, people should include seeds AND random states. People forget the latter. I also don't know why people just don't throw args (including current iteration and important metrics) into their checkpoints. We share this frustration.
There are exceptions [1] but most journals don't expect the reviewers to even attempt to reproduce results, which makes sense given how specialized and expensive scientific experiments often are. As a reviewer on open code papers I would usually try to run the provided code, it didn't always work and that wasn't always addressed before publication. (I was also usually the only one who even tried.)
Usually peer review is more about making sure the work is novel and interesting, fits the journal's audience and doesn't have any glaring flaws. Not entirely unlike code review: if it builds, merge it, and we can address problems in a future PR. Those are basically the reviewer instructions you get from most journals IME.
[1] OrgSyn famously requires a reproduction from one of its editors lab before it accepts any paper,
http://www.orgsyn.org/about.aspx
It has a very high reputation amongst chemists, even if it's "impact rating" is low. High impact journals are not usually considered the most accurate.
You don't get to handwave away the instructions of your experiment in my mind. Maybe other fields are fine without that, but I would never write a paper that doesn't clearly explain how I made samples or reference a paper which does. To do otherwise is bad science.
It's not about a reviewer replicating it, it's about anybody replicating it in a year or 30 years.
You're only one of two people I've ever heard make this claim. Which I'm sure you're aware, but many people probably aren't. Fwiw, I'm often called diligent because I read the code (looking at main method and anything critical or suspicious. Might run if suspicious). Even reading supplementary materials will earn you that title (which is inane). According to this informal survey, ~45% of neurips read the supplementary material <13% of the time and less than a third always read it[0] (I'm in that third, and presumably xmcqdpt2).
> Usually peer review is more about making sure the work is novel and interesting
This is why I find p̶e̶e̶r̶ ̶r̶e̶v̶i̶e̶w̶[1] journal/conference reviewing highly problematic and why this system is at the root of our current existential crisis: the reproduction crisis. Reproduction is the cornerstone of science. And many MANY good works are not novel in the slightest. See the work of Ross Wightman (timm) or Phil Wang (lucidrains). These people are doing critical work in the area of ML but they aren't really going to get "published" for these efforts. Many others do similar work, but just not at the same scale and so you'll likely not hear of them, but they are still critical to the ecosystem.
But with your next point: if it builds, merge it; I'm all for. The system should be about checking technical soundness and accuracy, NOT about novelty and how interesting it is. Of course we shouldn't allow plagiarism (claiming works/ideas that aren't your own), but we should allow: replications, revisiting (e.g. old methods, current frameworks (see ResNet strikes back)), surveys, technical studies, and all that. Novelty is a sham. Almost all work is incremental and thus we get highly subjective criteria for passing the bar.
Which is probably why high impact journals are not considered the most accurate. Because they don't encourage science so much as they encourage paper milling, rushing, and good writing.
[0] https://twitter.com/sarahookr/status/1660250223745314819
[1] We need to stop calling journal/conference reviewing "peer reviewing." Peer review is when your peers review. Full stop. This can come in many forms. Similarly publishing is when you publish a paper. Many important works come through open publishing.
There are decades of beta-amyloid papers that show that this is false. Like any for-profit entity, premium journals' primary purpose is to satiate shareholders - it is their feduciary responsibility to get away with as much as possible in the name of profit.
That's not to say that open access doesn't have equally as concerning issues: which of the two is better or worse is an extremely difficult call to make.
It just means it's less stinky/noticable.
I think I did take this analogy too far though.
The 'republic of letters' was much more similar to the journal-style approach than to modern arxiv, IMO.
I don't give this very low signal any meaning because the truth is that good works filter to the top regardless of the journals. They rise because peers share them, not because they got the stamp mark from a journal.
I’m not sure about the pull request workflow though, how would that work for review? Is your concept more for collaboration?
* It's owned by a (single) commercial corporation, Microsoft.
* There is censorship both by content and in some respects by country of origin.
* The code is closed.
but otherwise it's an interesting idea.
The C++ standardization committee uses GitHub to track papers submitted to them, see:
Like I'm not sure what you expect, just dumb people to read arxiv? If that were the case then no one would be submitting works to arxiv. The reason works go there is because researchers are reading them. They then talk about the works to their peers, post on twitter, blog, or whatever. And guess what, that's also exactly how works that are published in journals and conferences get passed around because there's a million of them too and no way for people to sort them. No one is sitting around ranking papers. That process happens by citations.
Who do you think the end user is? Laymen?
There's a reason most papers that end up in journals or conferences don't get a revision, or if they do only get minor ones. In general it is the same exact work which got sent to a journal or conference. So that is either the final draft or near. I do not want people to confuse this with "early" or "first" draft.
The history of arxiv is a few physicists getting annoyed that, like many of their peers, that sending per-prints (papers awaiting review by a journal) by email was burdensome. This was about just making access easier. It was about literally making peer review easier. Because let's be real, your colleagues reading your work closely is better review by peers than a few random people who are annoyed about having to review and know it is easy to just dismiss and reject.
[1] https://blog.arxiv.org/2019/08/29/our-moderation-process/
· By one side the Journal reclaim a high amount of money for being able TO TRY to publish on them (APCs fee), more if one want to be published for to allow free access, as to read the publication in such Journals require subscription, are pay walled.
· By other side, the reviewers of such papers, academics and researchers from other universities, don't receive economical compensation for doing the review, while the taken APCs fee its supposed for doing it? only in few cases the review is compensated by some programs of the reviewer's university or other 3rd parties. Then, what is for all those pay-walls and publish-fees?
Sounds like a big filter... for what? for who? Because can be seen it's not working just as a mere "quality" filter.
Being a paper published in one of those popular Journals, or being done directly in medias like arXiv, both sources needs the same amount of "grain of salt" about what is being read, until others replicate experiments, or contrast theories.
So IMHO, outsider point of view, from time to time I have to use Sci-Hub in fact, I understand if they decide to don't publish in such popular Journals. I consider absolutely legitimate when they do it in distribution channels like arXiv.
What I really would like it were exists an open research platform without pay walled papers, like happens in arXiv, and in addition, peer reviews through the platform were possible, as to read such reviews, and also, equally important, to have available debates about the paper, like if it were hacker news format.
To me it seems it keeps not only a wall around papers and journals. But also about whole academia which is probably an unwanted side effect since inside and outside of academia there seems to be consensus that knowledge transfer could be better. Also it seems everybody who can rather does research outside (AI/ML, Chemistry, Pharmacy).
Someone here mentioned:
Looks exactly like what is asked for.
Never heard about it though. Would need some more visibility (and word of mouth, I guess).
And was available only to rich bored people who could self teach themselves
I am a scientist and I'm all for "get rid of journals and conferences BECAUSE we already do figure it out." It isn't "somehow" we're already fucking doing it. We aren't going to journals and searching them, we go to arxiv. We read what our peers share with us. We read what comes out of other groups that we know that are doing similar work to us. Journals and conferences aren't helping us do science, they are helping us advance our careers because we live in a publish or perish ecosystem that quantifies our work based on the prestige of a conference who is prestigious because they reject 80% of works. Nowhere in here is a measure for the actual quality of a work. That is often a difficult task and can only be done by peers, which is already happening. Identifying bad works is easy but identifying good works is hard.
I'm sorry, but we already live in the system that you're suggesting is silly and it's been this way for centuries. Journals are just people capitalizing on our work and labor. Conferences at least put us in the same room, but they have no business claiming that they can accurately ascertain the quality of a work.
I do. Some of my peers do and some don't. It differs between fields and you're probably from CS (peer reviewed conferences) where progress is often faster than publication cycle, but it is not like that everywhere.
> they are helping us advance our careers
And it is important role which cannot be disregarded just because it doesn't match your idea of best possible promotion scheme.
> but we already live in the system that you're suggesting is silly and it's been this way for centuries.
Centuries? Can you elaborate? Peer review in journals as we know it is less than 100 years old.
I disagree. I think academia has fallen for Goodhart's law. While I would say that CS is probably on the worse side of things (especially ML), I don't think we're special in having critical problems. The publish or perish paradigm combined with the noisy signal of jorunaling is a big reason we are in the replication crisis. Because novelty is held in high regard while replication isn't of any concern. The irony being that novelty is quite rare and that the faster you publish the less novel works will be (in general. There are exceptions of course). You just can't do novel work fast.
> Centuries? Can you elaborate? Peer review in journals as we know it is less than 100 years old.
What I was referring to is that we've been just sending papers to one another for centuries. Uploading to arxiv is just an easier form of that (and was the explicit reason for its existence). That we scientists have been communicating with one another for centuries "figuring it out" without journals just fine. As far as I'm concerned, it is a failed experiment. It was okay in the beginning, had some good ideas, but then the rot grew and took over. It only works when everyone acts in good faith, and at scale only requires a few bad actors to spoil the barrel. Maybe I'm biased because of the problems of my field, but my friends in other areas still frequently hit problematic walls that just slow things down rather than improve them.
It may or may not, but my claim was that we need some mechanism to decide who gets a position and who doesn't, and it should be scalable for the current scale of science and beyond, i.e., at least hundreds/thousands of applications. Saying that journals are used for career decisions and this is not good (enough?) and thus doesn't need replacement mechanism is weird.
> a big reason we are in the replication crisis.
The stated reasons are the same for different branches of science, while replication crisis is far from that. Replication of 100-year-old experiments is also not always easy/successful. Publish or perish motivates people to cut corners, but I think it is far from being the main reason.
> What I was referring to is that we've been just sending papers to one another for centuries. Uploading to arxiv is just an easier form of that (and was the explicit reason for its existence).
(Almost) nobody reads arxiv (edit: I mean (relevant part of) arxiv as a whole, of course most people read specific papers on arxiv). In fact the snr of arxiv is so low, even compared to journals that just by filtering garbage you can get quarter a mil followers [1]. So instead of random experts of varying quality and motivation I'll get a single rando on twitter (no offense to A K, he does good job)? No thanks.
> It only works when everyone acts in good faith, and at scale only requires a few bad actors to spoil the barrel.
Citation needed? You could claim the same for science as whole, but self-correction turns to be extremely robust, even if slower than you may want.
It's a struggle to replicate 3 year old papers, forget 100 years.
I don't disagree. My claim is that the current mechanisms are nearly indistinguishable from noise but we pretend that they are very strong. I have also made the claim that Goodhart's Law is at play and made claims for the mechanisms and evidence of this. I'm not saying "trust me bro," I'm saying "here's my claim, here's the mechanism which I believe explains the claim, and here's my evidence." I'll admit a bit messy because a HN conversation, but this is here.
What I haven't stated explicitly is the alternative, so I will. The alternative is to do what we already do but just without journals and conferences. Peers still make decisions. You still look at citations, h-indices, and such, even though these are still noisy and problematic.
I really just have two claims here: 1) journals and conferences are high entropy signals. 2) If a signal has high entropy, it should not be the basis for important decisions.
If you want to see evidence for #1, then considering you publish in ML, I highly recommend reading several of the MANY papers and blog posts on the NeurIPS consistency experiments (there are 2!). The tldr is "reviewers are good at identifying bad papers, but not good at identifying good papers." In other words: high false negative rate or acceptance is noisy. This is because it is highly subjective and reviewers default to reject. I'd add that they have a incentive to too.
If you want evidence for #2, I suggest reading any book on signals, information theory, or statistics. It may go under many names, but the claim is not unique and generally uncontested.
> The stated reasons are the same for different branches of science, while replication crisis is far from that.
My claim about journals being at the heart of the replication crisis is that they chase novelty. That since we have a publish or perish paradigm (a coupled phenomena), that we're disincentivized from reproducing work. I'm sure you are fully aware that you get no credit for confirming work.
> (Almost) nobody reads arxiv
You claimed __aavaa__ was trolling, but are you sure you aren't? Your reference to Ashen is literally proof that people are reading arxiv. Clearly you're on twitter and you're in ML, so I don't know how you don't see that people aren't frequently posting their arxiv papers. My only guess is that you're green and when you started these conferences had a 1 year experiment in which they asked authors to not publically advertise their preprints. But before this (and starting again), this is how papers were communicated. And guess what, it still happened, just behind closed doors.
And given that you're in ML, I'm not sure how you're not aware that the innovation cycle is faster than the publication cycle. Waiting 4-8 months is too long. Look at CVPR: submission deadline in early November, final decision end of February, conference in late June. That's 4 months to just get Twitter notifications, by authors, about works to read. The listing didn't go live till April, so it is 6 months if you don't have peer networks. Then good luck sorting through this. That's just a list of papers, unsorted and untagged. On top of this, there's still a lot of garbage to sort through.
> the snr of arxiv is so low... that just by filtering garbage you can get quarter a mil followers
You and I see this very differently, allow me to explain. CVPR published 2359 papers for 2023 and 2066 for 2022. Numbers from here[0] but you can find similar numbers here[1] if you want to look at other conferences. Yes, conferences provide a curation, but so does Ashen and Twitter. The conferences cover many areas and don't tag or sort, and I know you aren't just going down that list and reading each paper title to determine if you should read it or not. I follow several accounts like Ashen. I also follow several researchers because I want to follow their works. This isn't happening because conferences are telling me who to listen to, this is because I've spent years wading through the noise and have peers that communicate with me what to read and not to read. Additionally I get recommendations from my advisor and his peers, from other members of my lab, from other people in my department, from friends outside my university, as well as emails from {Google,Semantic} Scholar. I don't think it isn't a well known problem that one of the most difficult things in a PhD is to get your network established and figure out how to wade through the noise to know what to read and not to. It's just a lot of fucking noise.
The SNR is high EVERYWHERE
And I'd like to add that I'm not necessarily unique in many of my points. Here's Bengio talking about how the pressures create incrementalism[2]. In this interview Hinton says "if you send in a paper that has a radically new idea, there's no chance in hell it will get accepted"[3]. Here's Peter Higgs saying he wouldn't cut it in modern academia[4]. The history of science is littered with people who got rocketed to fame because they spent a long time doing work and were willing to challenge the status quo. __Science necessitates challenging the status quo.__ Mathematics is quite famous for these dark horses actually (e.g. Yitang Zhang, Ramanujan, Galois, Sophie Germain, Grigori Perelman).
> Citation needed? You could claim the same for science as whole, but self-correction turns to be extremely robust, even if slower than you may want.
This entirely depends on the latter part. Yeah, things self-correct since eventually works get used in practice and thus held up to the flames. But if you're saying that given infinite time to retract a work it proof that the system is working then I don't think this is an argument that can be had in good faith. It is better to talk about fraud, dishonesty, plagiarism, how a system fails to accept good works, how it discourages innovation, and similar concepts. It is not that useful to discuss that over time a system resolves these problems because within that framework the exact discussion we are having would constitute part of that process, leading us into a self referential argument that is falsely constructed such that you will always be correct.
There are plenty of long term examples of major failures that took decades to correct, like Semmelweis and Boltzman (both who were driven insane and killed themselves). But if you're looking for short term examples and in the modern era, I'd say that being on HN this past week should have been evidence. In the last 8 days we've had: our second story on Gzip[5], "Fabricated data in research about honesty"[6] (which includes papers that were accepted for over a decade and widely cited), "A forthcoming collapse of the Atlantic meridional overturning circulation"[7] (where braaannigan had the top comment for awhile and many misinterpreted) and "I thought I wanted to be a professor, then I served on a hiring committee"[8] (where the article and comments discuss the metric hacking/Goodhart's Law). Or with more ML context, we can see "Please Commit More Blatant Academic Fraud"[9] where Jacob discusses explicit academic fraud and how prolific it is as well as referencing collusion rings. Or what about last year's CVPR E2V-SDE memes[10] about BLATANT plagiarism? The subsequent revelation of other blatant plagiarisms[11]. Or what about the collusion ring we learned about last year[12]. If you need more evidence, go see retractionwatch or I'll re-reference the NeurIPS consistency experiments (which we should know are not unique to NeurIPS). Also, just talk to senior graduate students and ask them why they are burned out. Especially ask both students who make it and those struggling, who will have very different stories. For even more, I ask that you got to CSRankings.org and create a regression plot for the rank of the universities against the number of faculty. If you even need any more, go read any work discussing improvements on the TPM or any of those discussing fairness in the review process. You need to be careful and ask if you're a beneficiary of the system, if you're a success of the system, a success despite the system, or where you are biased (as well as me). There's plenty out there and they go into far more detail than I can with a HN comment.
Is that enough citations?
[0] https://cvpr2023.thecvf.com/Conferences/2023/AcceptedPapers
[1] https://github.com/lixin4ever/Conference-Acceptance-Rate
[2] https://yoshuabengio.org/2020/02/26/time-to-rethink-the-publ...
[3] https://www.wired.com/story/googles-ai-guru-computers-think-...
[4] https://www.theguardian.com/science/2013/dec/06/peter-higgs-...
[5] https://news.ycombinator.com/item?id=36921552
[6] https://news.ycombinator.com/item?id=36907829
[7] https://news.ycombinator.com/item?id=36864319
[8] https://news.ycombinator.com/item?id=36825204
[9] https://jacobbuckman.com/2021-05-29-please-commit-more-blata...
[10] https://www.youtube.com/watch?v=UCmkpLduptU
[11] https://www.reddit.com/r/MachineLearning/comments/vjkssf/com...
[12] https://twitter.com/chriswolfvision/status/15452796423404011...
The journals take works others have done (work often paid for with public money) and get unpaid volunteers to peer-review it. Next they require the authors to sign away their copyright to it. And then they turn right back around and sell it to very same universities where it came from for thousands of dollars a year.
And for what? So they can have have profit margins rivaling big tech? (Their major expenses are lawyers and server costs for pdf documents…)
And why do researchers publish there? Because they need to in order to keep their jobs, or to get funding. You need those high-impact-factor and prestigious journals. And so you sign away your research.
Most 1st world researchers don’t care or don’t know about the costs. But most of the world is not so lucky. Yes you could go around begging the authors for a free copy (but definitely don’t go to scihub or library genesis and download them for free, that would be illegal…).
The “space for improvements” starts with burning down the journals and scattering their ashes among the fields to fertilize new science.
Oh yeah, destroy working system and then maybe figure it out, works like a charm every time.
The only currency that matters is if your results can be reproduced, as as far as I'm aware the overwhelming majority of journals make no attempt at reproducing the results before accepting it. Hell, it doesn't seem like they even make an attempt at make string there's enough information provided to even attempt to reproduce it.
When I come across a paper that does any sort of modelling, I simply assume I won't be able to reproduce it if the code isn't included. They all seem to leave out important information.
Again, the scale and accessibility of science is different now and then. We had commute before steam engine, it doesn't mean we can get rid of all engine-based transportation now without consequences.
> The only currency that matters is if your results can be reproduced
This is extreme reductionism. LHC results cannot be replicated unless you have multi billion particle accelerator in your backyard. JWST cannot be replicated. Many other large scale experiments are nearly impossible to replicate. In fact, these are often considered "important" ones. And this not necessarily due to price, but rather combination of price, unique expertise and willing to spend effort.
You know who will be able to tell you if the experiment description makes sense and has chances to be replicated if you ask them? Peers. Peer review at will never worked (and there were attempts) and never will because nobody likes to do peer review. I'm yet to see any proposal for peer review without editorial invitations which has at least theoretical chances to succeed.
> I won't be able to reproduce it if the code isn't included.
So you do mean "reproduction", not "replication". This is very low threshold and definitely not "only currency that matters". If I send you the code that alters the results somewhere deep inside, you'll be able to reproduce them.
Thankfully, the LHC is not what all of science looks like. There is a great deal of science that can be reproduced without needing billions of dollars.
> If I send you the code that alters the results somewhere deep inside, you'll be able to reproduce them.
But the current status quo for most papers is that you don't even have that. If you public put out code that manipulates data in this way then it can be scrutinized.
The alternative is to attempt to replicate the results, and waste months of time trying to guess what the authors did (since they won't tell you), and in the end never being able to reproduce it, not knowing if it was because key information was missing or because it was falsified.
That is, if you can spend the time attempt to reproduce the results, rather than publishing your own novel and revolution results. And if you'll find a journal who'll be interested in publishing your failure to replicate.
So what I'm saying is, I agree with your point here. The proof is in the pudding. Journals are convincing everyone that anything not in them is false and garbage but once garbage gets into them it creates and impossible bar to pass. Now repeat this for 50 or so years in a highly competitive space and what do you get? Well I think we all know the old saying: garbage in...
But for a lot of this, what happens is that they make the claim, they contact peers, send out the data, and have their peers confirm. You ever wonder why experiments like these have hundreds of authors[0,1]? That's because that shit is peer review. [0] has literally 8 pages of authors, which is in the format of first initial, last name. There's also 5 pages of university/lab affiliations.
Everything you're talking about here is done outside journals. And guess what, there are two versions of the arxiv work for [0], July 31st 2012 and August 31st 20122. Guess when it was "published". September 17th of that year. The announcement? July 4th! Btw, the journal is showing v2.
This is how big science is done. The replication and verification is happening intra-paper. You of course have to trust the machine, but that's true for everyone and a limit that can't be bypassed until a later date when a new one is built. Which frequently the first tasks are to confirm other observations, such as the JWST confirming exoplanets that were previously observed.
So to sum up: Replication happens, on big important experiments, outside journals
> [reproduction] is extreme reductionism
I'd still argue, with _aavaa_, that reproduction is the only currency that matters. Your experiment isn't worth shit if it can't be reproduced (i.e. it is indistinguishable from fraud)
> LHC results cannot be replicated
Most of the work is replicated, prior to publication.
> JWST cannot be replicated.
Observational data is confirmed. And JWST replicates its predecessors (an important part of its mission). (I'll even add LIGO in here and repeat the LHC and JWST comments)
> You know who will be able to tell you if the experiment description makes sense and has chances to be replicated if you ask them? Peers.
Everyone agrees with this. We are just saying the reviewers aren't necessarily your peers and the people you should be asking.
> because nobody likes to do peer review.
Lots of people like to do peer review and it happens all the time. If you're talking about reviewing for a journal, then yes I agree with you. But if you're talking about replicating work, building on, doing it to learn, or for fun, then this happens all the time. In fact, grad students do this professionally. How many works did you rebuild and try to replicate?
> I'm yet to see any proposal for peer review without editorial invitations which has at least theoretical chances to succeed.
You haven't dismissed __aavaa__ or my claims, only asserted they are wrong. If you would like to say why we are wrong, we're clearly open to hearing it and responding. But you've just asked us to trust you.
> So you do mean "reproduction", not "replication".
Reproduction is a form of replication. Replication is a spectrum. Reproduction is the minimum and with a lot between confirming experiments within a different context frame. But I don't think anyone is saying that just running your code constitutes reproduction. I'd say it is confirmation of your code. The source availability is about verification, as in I can probe it and look for errors, not so much reproduction.
[0] page 25 https://arxiv.org/abs/1207.7214
[1] appendix C page 41 https://arxiv.org/abs/2207.00043
You can find dozens of proofs on the arxiv that P=NP and that P!=NP. What does peer “review” look like here? Most people won’t bother to write an article specifically rebutting one random incorrect proof.
Now somebody tomorrow posts a proof that P!=NP. Will people pay attention? If it’s a “big name” or somebody at a university with a good PR office then yes; otherwise no. Doesn’t seem ideal to me.
doesn't upend your point, but just for the record, what we call science has only been around for about 500 years
Right up until Murray Gell-Mann and beyond, speaking outside the current limitations to knowledge was hard. I don't want this to descent into AGW and antivaxx denialism, this is inside classic science but considering radical theory/paradigm shifts which were still testable propositions. New models are hard on early stage career.
Blinded peer review was partly designed to help some of that. In a narrow enough field it's impossible for reviewer and submittor not to know each other. There may only be 3-4 people who understand your niche fully. Rueben Hersh discusses that a bit in "what is mathematics really" (I think, could be another book of his, "the mathematical experience")
Rhodes discusses Michael Polyani's theory of science as old fashioned apprenticeship. Journeyman scientists publish reproducible, testable work. Theoreticians.. harder to test sometimes.
Reviewers know who you are. This is quite shocking to discover if you come from Math or CS.
I've had pretty hard bounces which were deserved and I know how to get work over the threshold, but some review feedback has been petty, passive aggressive ignorance, and suspiciously similar stuff pops up from time to time in "3 papers and your phd is done" which makes me wonder if copycats are getting softballed through for career development.
the rules behind length, word count and mark-up are pretty silly too. I've seen some rather odd latex tricks to compensate for length inside word count because typographical checkers were bouncing.
I don't do this for a living, barely act as maybe helpful co author these days: it's hard work.
And while double blinding sounds nice in theory I'm not so convinced it is that useful in practice, because it requires the reviewer to play along and pretend they can't figure it out from the text alone: if they are able to do that reliably they can probably be trusted to keep an open mind anyway.
Reliable blinding of the author would mean having them consciously copy the style of others and avoiding citing their own previous work, which would be very hard in a small sub-field since they are by definition a sizeable fraction of it!
Double-blinding gives unknown groups and juniors the chance to publish at top venues, which are incredibly biased towards big-name universities and big-name groups.
In this case, the peer review is being conducted without the need for publishing venues. However, I don't think that is usually the case. Take a look at the countless number of ArXiv papers that lack any sort of peer review.
IIRC, the 1989 cold fusion fiasco did not involve scientific publishers as well.
Not saying this is a similar fiasco, just that 34 years ago it was done via a press conference and circulating drafts rather than arXiv.
Figure out how to send an independent lab's equipment and personal there or a sample of the substance to an independent lab that verify and if true. Also ensure that they can also do material analysis such as X-Ray spectroscopy & diffraction , and a battery of other stuff if a miracle did occur.
For all we know, this could be one of those accidents of sloppiness that introduces a particular containment that makes everything line up as a super conductor that nobody else will easily reproduce.
Apparently, people feel it's pretty easy to make, but in any case, if they have a sample they can see if it has the claimed properties and analyze what it's made of.
https://twitter.com/condensed_the/status/1684960318718406656
I get that terrible sense too - either the original studies were in error, or some one-off fluke makes it difficult to reproduce.
On the upside however, and I'd caveat this as not being fully in the know about the art of the possible in solid state labs, it seems that the material isn't that awfully difficult to produce given an appropriate lab and equipment. Hopefully we'll know one way or the other shortly.
I'm sure it will take a lot of time and money to run everything again. but all of earth seems willing to give them whatever resources they need.
The challenge is proving they have even that 1 sample.
If it was some kind of fluke and is not easily reproducible but the sample exists and is a room-temperature super-conductor then it becomes a reverse engineering problem.
Maybe we should call this the Hyde Phenomenon.
> Jekyll's involuntary transformations increased in frequency and required ever larger doses of the serum to reverse....Eventually, the supply of salt used in the serum ran low, and subsequent batches prepared from new stocks failed to work. Jekyll speculated that the original ingredient had some impurity that made it work.
The submitted title has been heavily editorialized. That’s the only relevant part of the article, and that’s far from implying that there’s a concerted effort at Argonne.
What's wrong with the submitted title (other than being very narrow)? It just says they're attempting to replicate, which they are.
If anything I'd say that "taking it seriously" is stronger language, and the submitted title is slightly underselling it.
The editorialised title here is misleading, as it's not what the article is really about, it's just mentioned in passing.
I agree that it was too narrow, but it wasn't exaggerating.
I objected to shusaku implying that this headline was exaggerating.
I don't know how to make this clearer.
> Do you think an equally good title might be "If you’ve ever had an MRI, you’ve lain inside a big electromagnet made of superconducting wire."?
Well, if you really want to get into this, even though I acknowledge it's cherry-picking:
If that fact is what the submitter cared about, and this was the only page on the internet talking about that fact... it wouldn't be a terrible idea.
The HN guidelines aren't great here. You can make your own blog post about something and link that, but it will probably get replaced to the "original" link even if the "original" link had a completely different focus.
Sometimes you have to pray that whoever wrote the original splits it up into different articles themselves. Or that they write on twitter, so you can link to a specific tweet.
Sometimes a newspaper will put three completely separate stories into the same article, and trying to link the second or third story on HN risks the title getting replaced with something utterly unrelated.
https://en.wikipedia.org/wiki/Holbrook_Superconductor_Projec...
>What’s more, lead and copper atoms have similar electronic structures, so substituting copper atoms for some of the lead atoms shouldn’t greatly affect the electrical properties of the material
The lead (II) ions claimed have an even number of electrons. Copper (II) has an odd number. Or if copper (I) is present, then the charge itself is different. Again, this is just a very confusing argument to hear from a physicist.
>First, the undoped material, lead apatite, isn’t a metal but rather a nonconducting mineral.
The cuprate and iron superconductors are not metals either. In fact some are Mott insulators (materials with unexpectedly high resistance) under normal conditions.
There is a case for skepticism about LK-99, but it isn't this one.
As a non physicist i wonder if that is useful in and of itself? Skimming wikipedia it doesn't seem like there are that many strongly diamagnetic materials. Would discovering a new one still be a big discovery (just not earth shattering)
A few people suggested they might have measured an insulator, the threshold where resistance changes being the breakdown where current starts flowing.
I'm skeptical either way, I think it's too early to say.
Even when the resistance is measured with a 4-point probe, the only conclusion that can be reached is that the resistance is smaller than the experimental error.
However it should be possible to determine that the resistance is zero through various indirect effects, for instance by establishing a closed current in the material and verifying that it does not decrease in time.
Even such experiments must be done carefully because some materials with unusual magnetic properties may behave apparently in a similar way to superconductors in some experiments, e.g. both a magnetized ferromagnetic material and a superconductor ring carrying a current will have a remanent magnetic field that is constant in time and in both cases the magnetic field will disappear when the temperature is raised above a threshold.
===========
Nadya Mason, a condensed matter physicist at the University of Illinois, Urbana-Champaign says, “I appreciate that the authors took appropriate data and were clear about their fabrication techniques.” Still, she cautions, “The data seems a bit sloppy.”
He's really laying on the sarcasm during the intro, I could totally see that getting misconstrued by someone not familiar with our flavour of sarcasm.
I get probably this is a popular video maker in whatever sphere he's in, but as an outsider, it's difficult to get into the video because of what I attribute as the presenter's YouTube personality.
(also it very much so bugs me to see people using Chrome and see all the nags for notifications, the random extensions/features, etc. And any browser without uBlock origin throws me as it takes me a minute to realize why all the websites look so bad, but that's just a personal issue :) )
He explains why the video is bunk using information provided by the lab, and basic physics. And then 'replicates' using his own sheet of metal and a magnet.
Also, he raises the fact that it's a commercial lab.
I've downgraded my level expectations a couple of orders of magnitude. But hey, let's hope for a miracle.
Thunderfoot comment though seems ridiculous. It would be irrelevant because it is a ceramic??? lol. They proposed a mechanism how it works! Obviously this would just be the first step. That's like saying the first transistor was irrelevant because we don't use that design anymore.
...and based on that demo, it really does seem like a dumb way to demonstrate your miracle material. Really doesn't inspire confidence.
Maybe we'll have maglev everything by the time I'm old and gray.
Hirsch is also apparently a bit of a shitposter at age 70 and was banned for flaming people on arxiv last year.
No, I don't know why it took over two decades for their discoverers to figure out it was a room-temperature superconductor.
I particularly love #3
At the extreme end of big news, stupid things can happen (I've worked in a lab: people suck at labelling things they make).
Yes. Some may remember the problems with cold fusion. I once went to a packed talk at Stanford where some physicists and chemists were trying to reproduce the effect. They mentioned that in the first attempts, they had radiation alarms set up in case the thing suddenly started emitting substantial amounts of radiation. After a while, it became clear that the effect, if it existed at all, was small, somewhere below twice background radiation, so it wasn't high risk. They discovered that people moving around the apparatus affected the results; humans have a lot of water and are neutron reflectors. Finally they did the experiment inside a "neutron cube", a box made of lead bricks, to eliminate stray neutrons. No neutron generation measured inside the cube.
They worked hard for months getting to a definitive "no".
Entertain our sudden fickle intellect! We're paying attention now, so you should all hop to, regardless of theory or evidence!
Exactly how much more proof than "Here, have a super conductor we made" do you want exactly?
The only thing required is a little patience.
Is it unstable?
Oh wow. If true, then that's that.
If they are stable, the original samples will validate, or they will not.
The researchers could wave their hands and claim the sample deteriorated, but I think that would still end optimism for replication.
- Lenz’s law applying to copper (Lenz's law - Wikipedia 7) - Copper phosphide (“Lanarkite and Cu3P were uniformly mixed in a molar ratio of 1:1 in an agate mortar with a pestle”). Pretty high copper content. -No Meissner effect just standard copper-related phenomena based on Lenz’s law.
Not claiming the researchers of LK99 are intentionally faking their results, by the way!
I replied, “It might be or it might not be…” and I looked at him and grinned and said “but you still have to clean your bedroom!”
“Oh dad!” He said as he scampered off.