Collusion rings threaten the integrity of computer science research
m-cacm.acm.org
m-cacm.acm.org
Part of the problem is that people seem to want an objective evaluation of a piece of research, and measure the value for taxpayer money. Well, you can’t get an objective evaluation. It’s all subjective. And “impact” as a quantifiable entity in general is nonsense, for one thing the timescales prohibit its measurement at the point where it may be useful.
The solution is to use management. Lots of people object here and say “but nepotism, favouritism” and yep that’s a problem, but it is less of a problem that the decline of western universities. You can circumvent it somewhat by rotation, by involving external figures, by a hierarchy that ensures people are not colluding, but ultimately you just have to trust people and accept some wastage.
People aren’t in academia for the money. It’s a vocation. You’re not going to have many people milking the system. Things went pretty well before the metric culture invaded the academy. They can go well again.
(Speaking about Spain).
The KPIs are there because OFFICIALLY (and this is strictly so) you are not even allowed to get a tenured position without an absurd number of (in Maths) JCR papers IN THE FIRST QUARTILE.
This is so stupid it is not even funny but how can you fight that when your PhD students depend on those metrics?
There was a time in history when tenure made sense, but today the tenure track process forces a lot of people to go after low-hanging fruit that has a high probability of being accepted for publication, instead of trying things that are meaningful to try but may fail.
You are constantly under evaluation whether you have tenure or not.
And what happens if you're poorly evaluated? They can't fire you anyway. Will they move you to a boiler room as a punishment?
In short, I think it is definitely clear neither the citation nor the journal brand is the best proxy for the worthiness but the system you are proposing is worse while still reliant on subjective judgement.
Before introducing the KPIs, a majority of polish science was basically people milking the system and doing barely any (valueable) research. It was seen as an easy, safe and ok paying job where the only major hassle is having to teach the students. You often needed connections to get in. It was partially like that because of the communist legacy, where playing ball with the communist party was the most important merit for promotion, which, over the course of 45 years (the span of communism in Poland), filled the academia management ranks with conformist mediocrities.
Now, after a series of major reforms, there's a ton of KPIs, and people are now doing plenty of makework research to collect the required points, but still little valueable work gets done. Also, people interested in doing genuine science who would be doing it under the old system are now discouraged from joining academia, because in the system they're expected to game the points system and not to do real work.
What is the lesson from this is? Creating institutionalized science is hard? It requires a long tradition and scientific cultural standards and can't be just wished into place by bureaucrats? Also, perhaps it's good to be doing the science for some purpose, which in the US case are often DoD grants, where the military expects some practical application. This application may be extremely distant, vague and uncertain (they fund pure math research!), but still, they're the client and they expect results. Whereas the (unstated) goal of science in Poland seems to be just to increase the prestige of Polish science and its Universities by getting papers into prestigious journals, whereas the actual science being done doesn't matter at all - basically state-level navel gazing.
Looking at other departments, say, sociology* it's a dumpster. Full of cronies, retired politicians and relatives.
*Just an example which doesn't require "hard" skills like math.
If you start cheating the metrics, or optimizing a lot towards them, it becomes counter-productive when they change. As such, the most efficient way forward would be to work without trying to optimize for a temporary metric. On the flip side, it would be troublesome to convince people to different, complex forms each time.
What first gave me the idea was the concept of "lubricating" headers (submitting some with random values) for future http protocols, to combat "ossification", where middle boxes start to meddle with them and become obsolete when they don't recognize the new fields, instead of transmitting them.
In that case, if you gamed the "wrong" metrics, you still come out ahead. Not as much as for gaming the right metrics, but still.
Relying on altruistic tendencies for people in academia is not adequate. Everyone starts out in academics as school children, and get filtered out or filter themselves out pursuing other things. Those who remain will be the ones who love to learn and teach, those who just cannot accept loss/failure, and, sadly, those who are afraid of change. The more competitive the field becomes, the harder it is to succeed, the more we select for the hyper-competitive or fearful over the altruistic.
The system I was presented with would have meant my supervisors got more say in what I spent my time on than I did. They heavily skewed to supporting existing psychological models of entrepreneurial failure which I wasn't interested in.
The bureaucracy, authoritarianism and endless hoop-jumping around it was a total red flag for me. I opted out.
My conclusions:
PhD grads aren't smarter than others. They're just more willing to put up with bullshit, conform to meaningless rules, and jump through hoops.
Academic research is rarely about the things it says that it's about, and seems to be more about maintaining/improving the career prospects of the academics.
Academic research is heavily affected by political spats with other academics that have nothing to do with the actual subject, but more to do with ego, pride and interpersonal dislike.
Humanity as a whole is losing out on some potentially amazing long term research and researchers because of this dynamic.
I have no idea how to fix it systematically.
PhD grads aren't smarter, they're playing a different game. There's bullshit everywhere, you just chose a different pile to call home -- probably because it smelled better to your reward/bullshit tradeoff.
Why would PhD's choose that pile of bullshit of yours? ...
As you say, "Academic research is rarely about the things it says its about" is a good observation. The reason is that most of the time grants are awarded to professors to try a method (their specialty) to a problem (the thing the grant is about). This looks like the professor is padding his career, but honestly, that's the point.
We would ideally have a huge set of professors with perfect specializations such that a combination of professors could solve any problem. Science funding is ensuring that deep, old expertise is preserved in case it is useful. Grants are a way to simultaneously test that usefulness for modern problems and expand it a little towards modern problems by producing new phd's with slightly mutated expertise. This is why PhDs endure their own type of bullshit, because they want to be part of this particular kind of knowledge legacy. There are other ways of doing this. A problem-first (vs solution-first) approach is kind of better for a different kind of venue, like business, NASA, etc.
Political spats? Absolutely. No contest there. Turf wars are a thing.
You're totally right that academia doesn't have a monopoly on this kind of bullshit. But few other places get away with it so much, and manage to maintain a (rapidly crumbling) reputation of not being 90% bullshit.
I would suggest that your own rapidly crumbling opinion of academia is probably not endemic. The top universities still produce top-tier R&D, and top-tier candidates that go on to do top-tier work for large and small companies.
It's definitely ok and understandable that some (or even many) people would feel differently, but my own opinion (coming from a mid-tier school) is that my training was absolutely of inestimable benefit to my job prospects, and was 100% enabling of my current career in robotics R&D. And I'm guessing that getting admitted to MIT would, without fail, make anyone happy in the entire world. (Though I didn't go to MIT obv)
You comment does not seem to contain any explanation as to why 'using management' would solve the problem you allude to. Can you elaborate?
There is only one reasonable way to evaluate research and researchers, that's to evaluate the content of their work and publications by external evaluation panels and tell these panels explicitly that they should not base their assessments solely on indicator counting, but on the overall merits, originality, and prospects of the research according to their subjective opinion. Metrics shouldn't even be used as a tie-breaker, they should only ever be used as weak indicators, and this must be explicitly mentioned in the guidelines.
In addition, you need a few other guidelines and laws. For example, it must be prohibited that someone becomes a postdoc at the same place where they obtained their Ph.D. We have people who study at university X, do their exams at university X, do their Ph.D. at university X under the same professors they always knew, then become postdocs working for their former supervisors and being exploited by them (teaching the same bad seminar their former supervisor teaches the past 20 years), and then get tenure in a rigged call. And the worst thing about it is that they feel entitled to all of this.
You've got to break this vicious cycle, but with your suggestion of using a methodology that worked in 1950s (with an order of magnitude less candidates) this could never be achieved.
But that means that a tenured professor will start acting in non-academically-independent ways at some point before their tenure is up to avoid messing up their re-applicaction.
It is also quite likely that events such as departmental re-orgs could be timed around tenure expiration to eliminate specific job descriptions in order to make re-applying more difficult.
You might be able to achieve the same goals though some combination of making tenure transferrable between cooperating institutions, mandatory sabbaticals, requiring review committees be partly or wholely staffed from outside the institution, etc.
Independent of every other point, this is in my point of view a real problem. It happens so often and I don't understand why there's often no law that at least a postdoc at another institution has to be done.
I think external evaluation panels (and in particular, from a third country) are the way to go. We already have good examples, for example, in ERC grant panels. The ERC uses exactly the strategy you mention and it has an impeccable reputation, I know many people who applied with or without success but I know no one who felt treated unfairly.
But I'm against blanket rules prohibiting postdocs or positions at the same place of the PhD, at least from my Southern European (Spanish) point of view. This is often touted in Southern European countries because in the US no one does that so it must be evil, so clearly we should ban it to be more like Americans and see if this makes our scientific productivity closer to theirs. But European (especially Southern European) culture is not US culture. People want to settle down, be near their loved ones, and there's nothing intrinsically bad about that that should be punished. Plus, the job market is much less dynamic so even for those who don't mind bumping around, it can be hard to reconcile with a possible spouse who has a career too. And finally, if you push people in this region to move, most of the time the result will be that they end up in a Northern European country (or the US, Canada, etc.) where they make 3x or 4x more, and never come back - once you have experienced a much better salary it's hard to justify returning, I have seen it plenty of times.
Bring on the external evaluation panels, and then there will be no need for any measure forcing people to move, which would reduce inclusivity and thus the talent pool.
That won't happen, mostly because employers have outsourced education and vetting (in the form of requiring bachelor/master degrees) to universities (and the associated costs to governments and/or students who pay tuitions) instead of the old style vocational training/apprenticeship system where the employers had to pay.
Want to restore academia to only those actually interested in science? Make employers pay a substantial tax on jobs requiring academic degrees.
Not sure what you mean by that, but KPIs are generally put in place by high-level management. Or do you want more micromanagement?
Either way, I think the solution is not more control, quite the opposite. I think the solution is just to remove the extrinsic incentives.
Some people say UBI will cause people to do nothing and that is probably true, but the flip side is that the output of the remaining people will likely be many times higher both in volume and quality (with the total volume much lower but higher quality). Not having their energy completely destroyed by all the busy work necessary to show they are working.
Well, science doesn't care that much for KPIs, per se. It's more that the managers want numbers to steer by.
In academia, getting promoted means more management tasks. So higher up academics have been indoctrinated to want numbers. Is the scourge of management.
As such: not a big fan of your solution.
I agree overall with what you wrote, but have to comment here, because I think this is already not the case in many settings. I can only speak for the US, but in my experience with some other places overseas similar issues are developing.
There are many legitimate hypotheses for why this is the case, but in general at many universities, as far as climbing the academic ladder is concerned, publication metrics are no longer relevant. That is, some baseline is required, but beyond that, most of the focus is on money and grant sizes. I've been in promotion meetings discussing junior faculty that are not publishing and this is brushed aside because they have large grants. I've also repeatedly heard sentiments to the effect of "papers are a dime a dozen, grant dollars distinguish good from bad research."
Again, there's lots of reasonable opinions about this, but I've come to a place where I've decided this is incentivizing corruption. Good research is only weakly correlated with its grant reimbursement, and regardless, it's lead to a focus on something only weakly associated with research quality. Discussions with university staff where you're openly pressured to artificially inflate costs to bring in more indirect funds should raise questions. Just as it's apparent that incentivizing (relatively) superficial bibliometric indices like publication count or h factors leads to superficial science, incentivizing research through grant money has the same effect, but differently.
So yes, going into academics is not the way to make money if that's what you want. However, I think nowadays in the US, it's very much all about the money for large segments, who are milking the system right now at this moment.
Also, in theory, yes, management is the solution, but really management is how we've gotten into this mess. Good management, yes, bad management no. But how do you insure the former?
Fixing this mess academics has slid down (in my perception, maybe everything really is fine) will require a lot of changes that will be controversial and painful to many, and I don't think there's a single magic bullet cure. Eliminating indirect funds is probably one thing, funding research through different mechanisms is another, maybe lotteries, probably opening up grant review processes to the general public. Maybe dissociating scientific research from the university system even more so than has been the case is also necessary. Maybe incentivizing a change in university administration structures. Probably all of the above, plus a lot else.
How to get things to go well again is achievable in theory but how to get there is less clear given the amount of change involved.
It's not really about judging people either. When it comes to choosing which people to reward with jobs, promotions, grants, and prizes, we already know the solution: expert panels that spend nontrivial time with each application. Sometimes there are political or administrative reasons that override academic excellence, but in general the academia has figured out how to evaluate shortlisted people.
The real problem is shortlisting people. For example, when a reputable university has an open faculty position, it typically gets tens of great applications. Because the people evaluating the applications are busy professors, they need a way of throwing away most of the applications with minimal effort. From the applicant's perspective, this means you only get a chance if the first impression makes you stand out among other great applicants. And that's why it matters that you publish in prestigious conferences/journals, come from a prestigious university, and have prestigious PhD/postdoc supervisors.
Once it's no longer about being in the esteemed and scarce "10%", they won't bother because they don't need to. Imagine a process where the only criteria are technical soundness and novelty, and as long as minimal standards are met, it's a "go". Call it the "ArXiv + quality check" model.
Neither formal acceptance to publish nor citation numbers truly mark scientific excellence; perhaps, winning a "test of time award" does, or appearing in a text book 10 years later.
I've been reviewing occasionally since ~1995, regularly since ~2004, and I've never heard of collusion rings happening in my sub-area of CS (ML, IR, NLP). I have caught people submitting to multiple conferences without disclosing it. Ignoring past work that is relevant is common, more often our of blissful ignorance, and occasionally likely with full intent. I'm not saying I doubt the report, but I suspect the bigger problem that CS has is a large percentage of poor-quality work that couldn't be replicated.
BTW, the most blantant thing I've heard of (from a contact complaining about it on LinkedIn) is someone had their very own core paper from their PhD thesis plagiarised - submitted to another conference (again) but with different author names on it... and they even cited the real author's PhD thesis!
One possible issue is that researchers usually need to justify their research to somebody who's not in their field. Conferences are one way to do this. So are citation counts. Both are highly imperfect, but outsiders typically want some signal that doesn't require being an expert in a person's chosen field. The "Arxiv + quality check" model doesn't seem to provide this.
> I suspect the bigger problem that CS has is a large percentage of poor-quality work that couldn't be replicated.
As a sort of ML researcher for several years, I agree.
I don't have data, but from subjective experience, 5-6 years ago most papers in major NLP conferences didn't have an associated code repository. Now, the overwhelming majority do.
There are still many other problems, for example a big one is reporting of spurious improvements that can vanish if you get a less lucky random seed. But at least including code is now common practice.
I suppose the situation regarding common datasets might vary between subfields and NLP tasks, so maybe I just saw a weird corner of it.
Of course the code was also nowhere to be seen.
Availability of code would of course be even more important, both because of replicability and general verifiability, and also because that would allow you to do a comparison with any number of datasets yourself.
Glad to hear that code availability has been improving.
> There are still many other problems, for example a big one is reporting of spurious improvements that can vanish if you get a less lucky random seed.
Considering that a lot of NLP is at least somewhat based on machine learning, don't people do cross-validation or something?
You do a paper showing that problem X can be solved slightly better by downloading and training on a billion tweets.
But you don’t have the copyright to those tweets, so you can’t share data.
> don't people do cross-validation or something
A lot of stable problems comes with a dataset already split into train and test.
That's true. Sometimes you might try to tweak the algorithm itself rather than the data, though, or experiment with different kinds of preprocessing or something, and in those cases it would be helpful to be able to do different experiments with shared datasets.
My limited experiences were from around the time deep learning was only about to become a big thing, so it might have been different then. Maybe you nowadays just throw more tweets and GPUs at the problem.
More info here: https://michaelnielsen.org/blog/three-myths-about-scientific...
I think the recent wave of low-impact submissions and co-authorship rings is the result of developing countries trying to simplify that process and tying hiring/promotion/pay directly to publication count and related easy metrics.
It may be true that the SNR in research output from developing countries is lower, but there is still lots of good science. But essentially no publishers with good reputation. So even with the same SNR, the increased pool of countries producing science would add to the publication pressure.
Also, a "best 10%" conference/journal is valuable -- I have only so much time in the day. There are a few conferences for which it is always a good use of my time to read all the abstracts, plus one or two of the papers that seem most interesting. I can't do that for every conference, or even most conferences, in my area.
So the "best 10%" conference/journal is valuable to the consumers, and the prestiege is valuable to the producers. Therefore I think such a thing would simply re-emerge if you somehow killed it.
[1] https://ieeexplore.ieee.org/abstract/document/6746236 [2] https://www.youtube.com/watch?v=MFNFScqN47o
I mean I could buy your paper but I would have to know it by heart and understand it in order to defend it.
At least that's how it went when I was studying applied physics (1974-1977).
If I had just bought a paper I would have had a really hard time in the viva voce.
Edit: by -> buy
So that even if people don't fail their defenses often, there might be many that didn't get theirs scheduled at all? This is just speculation on my part.
In my program, many of us would strategize for the 30-60 minutes of the closed door grilling. We sought to give our committee members obvious things to criticize with the PhD student having prepared arguments to defend against these criticisms. E.g., I ashamedly included quite a few spelling and grammar errors in the first few pages of the summary section of the thesis (the only part anyone would actually read) and we spent at least 15 minutes on my horrible writing ability.
In general, the main outcome of the closed door portion of the defense was requests for additional work. It was common for committee members to suggest additional things that could “improve” the thesis work. Not surprisingly, many of these suggestions involved applying a committee member's methods, even if not plausibly applicable, so that one would publish another paper citing the committee member’s work. Some students, including myself, would have a job lined up before the defense to timebox the amount of additional work that could be requested.
"I don't trust your maths" "I don't feel this analysis is right, but I can't describe in what way" "you are clearly not very knowledgeable" and many other similar things.
Asking me a question and then before I can open my mouth answering it yourself, and then insulting me for not answering it was the start of the viva defence and it set the tone for the rest.
I was also heavily criticised for not having cited a paper that came out in-between submitting my thesis and the defence, despite this being literally impossible to have done so, and, despite having already had gotten a job in that time, was given limited time to do additional experiments, write whole new chapters, new code, do new experiments, etc. Ended up adding 90 pages of material to the thesis.
In the end I had to quit the job I had just got, because It would have been impossible to not fail my PhD program and keep the job.
Afterword's, in behind closed doors discussions it was revealed that one guy had pushed for almost all of the required extra work deliberately to try to make me fail, because I had done something he could not.
I mean, personally, I don't really care about reputation of academia, but given that you described a horrible story where somebody basically tried to ruin your career, and given that you decided not to name specific people or institutions, it seems that the whole (UK) academia will have to take the reputation hit for the alleged scandal.
In the first half I was privately grilled by my committee. They wouldn't let the dissertation go to a public defense until they'd satisfied themselves that it was fine.
As far as I can tell the public defenses usually look ceremonial at GMU (at least in its engineering department), but they aren't actually ceremonial. Anyone from the public can ask questions at a public defense, so they aren't ceremonial. However, the goal of the (first) private half was to try to make sure that the defender is ready for arbitrary questions (because he understands the material). So it's unusual for the public to ask questions that the defender isn't able to answer. I got some questions I hadn't heard before in my public defense, but I was able to handle them.
You can see my public defense here: https://www.youtube.com/watch?v=QYH18NpsRu8
Moreover, the correlation with acceptance and impact in existing, but not that high: https://medium.com/ai2-blog/what-open-data-tells-us-about-re...
Of course, only corporate researchers can rely on not publishing in established journals - as their salary and position does not come from "publish or perish" metrics. Ironically, it means that there is more academic freedom in private companies than in academia.
I've been thinking on an off about review system that might improve on things. I'm imagining perhaps: reviewers and authors both get to see who they are, and conflicts of interest can be called out by other people after review and before publication; reviewers are chosen at random, not allowed to bid; reviewers are sent a series of pairs of papers, and asked to choose which one they'd rather see, scores and ultimately publication can be decided by rank choice vote rather than reviewer assignment; comments on paper improvement would be completely optional. Would this be better or worse than what we have? Would it deter explicit collusion?
In my corner of CS, there are plenty reviews of debatable use that are very negative. In fact, I have a pet hypothesis that the average score of the lower scoring but accepted papers is negative (scale from -3 to 3). And I wouldn't be surprised if the median paper's score was negative.
Positive uselessness doesn't seem like that much worse.
I’ve seen a lot of overly and unnecessarily harsh reviews. Anonymizing enables over-stating criticism, it happens routinely. I don’t think I agree that harsh critique is necessary for a healthy review system. It is already the case that good reviews are not extremely harsh, they focus on the facts and are willing to stand by their statements. I don’t personally know that many researchers who have trouble being direct in person and face to face, or of offering constructive criticism.
> As a reviewer you are only advising the editor
This completely depends on the journal or conference. Quite a few of them, especially the larger ones, do not override reviews casually nor often. And what I’m suggesting is a system where this idea can change, where editors can more easily trust the review results, and won’t need to override the decision.
> People are petty.
This might well be true. And so I’m not entirely understanding your argument. It seems to be simultaneously suggesting we have a problem, and defending the status quo as the way it needs to be. What would you suggest as a way to improve the review system so that pettiness has less influence than it does today?
You’re also misunderstanding the point of reviews. They also serve as comments to the authors to modify their manuscript and make it acceptable for publication.
I think this is a little more pessimistic than what the piece says. The NeurIPS (then-NIPS) experiment said that about 60% of papers accepted by one PC got rejected by the other. That doesn't actually mean "the quality of the work has little to do with its odds of acceptance". It may just be that there's a paper has to cross a quality threshold, and once it's past, then the outcome has a lot more variation.
My personal take on NeurIPS specifically is that there's a fraction of bad papers, maybe 40%, that probably shouldn't and won't get in. Then there's a minority of very nice papers that probably should and will get in, maybe 5-10%. And then there are a bunch of middling papers where a lot of it is luck and drawing friendly reviewers. But these aren't bad papers, and you can't really just churn them out, they're just not very good papers.
As you mention, the problem with having two sets of reviewers is that it's hard enough for conferences like NeurIPS to find one set of qualified reviewers. Usually at least 1/3 of reviewers on any given paper produce a poor review, either because they don't care or because they really lack expertise. Complaining about this is so widespread that it even doubles as a sort of icebreaker for researchers, but nobody has a good solution.
From memory, it was 25% rejected by both PCs, 15% accepted by both PCs, and the middle 60% random.
Essentially if you look at review scores for a conference which has say a 40% accept rate (and this is quite similar across fields I'd imagine), you find there's 10-20% (depending on conference) of papers that are clear reject for all reviewers, then there's probably around 10%-15% of papers which are very clear rejects now the rest of the papers are very similar in scores so the cut-off becomes quite arbitrary (and depends on luck as well). This is actually well known for grant applications and a sign that there is likely not enough money in the system.
Maybe it's time to move on from some of these conferences, and focus on interactions that maximize sharing research findings. I know that is unrealistic, but like every other metric, conference acceptance ceases to have value once the metric itself is what people care about.
And one of the suggested solutions (at ~22:00) seems that it would work if it can be adopted - essentially, have top university administrators evaluate top x papers for hiring/tenure decisions and ignore everything beyond that number. What you measure is what you get; if you measure count, you get a deluge of 'least publishable units', if you measure your top 3 papers, then everyone will focus on quality instead of quantity. A counterargument probably is that it's easier for the administors to measure quantity in a way that seems objective and resistant to any arguments or appeals decisions, and it's far harder to objectively compare quality especially if the candidates are from different subfields of research.
The REF is evaluated by humans. I take 2/3 weeks to read a paper. For good papers I might work on them for 6 weeks + to really get into the technique. How can a REF reviewer consume 5 papers to evaluate an academic? How can they consume 200 papers from 40 academics?
The right thing would be to have the department sumbit it's top 5 papers - 3 reviewers could really see what is going on is a department then.
So - how many IEEE conferences are there!? Also things change, it was much easier to get into Neurips 7 years ago... but it's not necessarily the case that the papers from this year will have as much impact or be as good as the papers from then. And as Neurips itself showed, the peer review process is somewhat random - papers are rejected by different panels meaning that getting in is likely an achievement but also possibly a bit lucky. I don't think that evaluating the papers based on where they were published is a good way to allocate public money.
As a strong concrete example, paper 13 from Ferguson's group at Imperial is probably one of the most important documents for the last 20 years (if you live in the UK or France), it was "published" on a website...
It's not even just about academia in general, it's a universal problem caused by lack of personal accountability through the globalization of talent. The exact same dynamic was shown multiple times in HBO's The Wire. Gaming the metrics. LPU's are no different than 10 minute YouTube videos.
Findings and papers from well known individuals (read: twitter accounts) do get far more attention , more citations. Of course, one can argue that, broadly, well known labs and individuals are wel known because of their tendency to do great work, write better papers. And that’s true. However, the above still holds, in my experience as PhD student in ML. Anecdotally, I have seen instances where a less interesting paper from a renowned lab got more attention and eventually more citations than a better paper accepted at the same venue by a less renowned lab on the same topic.
I would also argue that with the increased importance of the (mostly) commercial "high-impact" journals this has become worse. I know that some of the professional (non-expert) editors of these journals specifically look at the citation counts of authors before accepting to send them out to review, because their main aim is to get people reading the articles, not necessarily good science.
I believe journals need to adapt the openreview process. There are obvious parallels in the lifecycle of a journal submission (from submission to acceptance) and the openreview process, except that the latter is accelerated (for better or for worse).
I have to say that I find the review process of the copernicus journals very interesting. You can see a description here: [1]. Unfortunately I don't work in a related field otherwise I would have published there already.
[1] https://www.atmospheric-chemistry-and-physics.net/peer_revie...
> We find considerable evidence that, overall, article citations are positively correlated with tweets about the article, and we find little evidence to suggest that author gender affects the transmission of research in this new media
https://journals.plos.org/plosone/article?id=10.1371/journal...
(I've only skimmed the paper, a few months ago).
When someone is hired (whether for tenure or time-limited), their research as a whole has to be evaluated by external, independent committees who take into account the content of the research and do not base their judgment on indicators only. There is no shortcut around that.
The biggest annoyance nowadays is the decision-makers' insistence on "excellence", though. You cannot have only excellent people everywhere, as per the definition of "excellent", yet this demand is in every fucking guideline for postdocs and tenure-track position. It's absolutely ridiculous.
The problem with this assertion is: non-experts are paying for all of this.
Imagine you are an ordinary taxpayer. You are feeling the pinch yourself, you look around you and see infrastructure crumbling, every day the press says healthcare and this and that is underfunded. Now along comes some scientist, he or she wants a few billion for a new particle collider that will make no difference whatsoever to your life, and their only justification for it is "well other scientists say we should get all this money, and they're cleverer than you, shut up".
Can you see why funding something with no accountability might be considered problematic?
There's also the fact that an expert who can't explain something to a non-expert probably doesn't understand it very well themselves...
It took over 20 years after the Standard Model reached broad acceptance (ie., the experts thought the theory was probably right) for the first supercollider powerful enough to observe the Higgs boson to be financed. This was enough time for policy makers to reach high confidence that the experts had not gone badly wrong and for there to be reasonably well-informed public opinion on the merits of the search.
> Can you see why funding something with no accountability might be considered problematic?
Maybe you're confusing budget decisions with funding and hiring decisions. These are fundamentally different. Universities and research institutions, as well as national funding authorities, get budgets that are decided politically, i.e., by elected representatives. These can have broad categories and guidelines or preferred research areas (e.g. "excellence initiatives"). Budgets are usually allocated well in advance, for instance our national funding authority gets budget security for 4 year periods (if I'm not mistaken). How they spend it is dictated by political guidelines for the respective period and plenty of complicated national and international laws.
In contrast, I was talking about hiring decisions and decisions about individual funding. How can it not be obvious to you that these decisions need to be made by experts on the basis of CVs and scientific project proposals, not by politicians or other laymen?
E.g, for my company's twice-yearly evaluation, everybody writes up a short report on the most impactful stuff they've done including evidence, this is evaluated by their manager to give a score, and then there's a series of group meetings between managers to make sure that the scores are calibrated, including looking at all types of metrics that can be dug up and comparing to our written role descriptions for different levels. It takes a lot of time but creates fair scores.
This is extremely labor intensive, but that's the thing: To create anything resembling fair evaluation of a large group of people that do a large set of different things, you need to do things that are labor intensive. Using a simple set of metrics don't cut it.
- Conferences and participants have increased exponentially over the last few years.
- Students in AI/ML graduate programs have similarly increased.
- Huge numbers of companies are hiring AI/ML graduates.
- What constitutes a true advance AI/ML is difficult to determine. Deep learning is fairly ad-hoc method - a tweak to an existing method that allows you exceed soto (state of the art) is the simplest way to get attention. But that's not a method and so you're many others pushing similar tweaks also.
This environment in particular seems like it would exacerbate all the ordinary pressures to cheat found in the academic environment. It has something of the quality of the last blow-out of a bubble. And the thing is that even with deep learning being real, the dynamics seem fated to push things to the point that expectation are sufficiently far past reality that the whole collapses, for a bit.
Would it help at all if rather than participants reviewing 3 papers, each reviewed 2 papers and validated the review of 3 more papers?
This is computer science here, with things like the set NP whose defining characteristic is that it's easier to check a solution than generate it.
I'm imagining having some standard that reviews are held to in order to make them validatable. When validating a review, you are just confirming that the issues brought up are reasonable. Same for the compliments.
Sure, it's not perfect because the validators wouldn't dive in as deep or have as much context as the reviewers, but sitting here in my obsidian tower of industry, it seems like it would at least make collusion attacks more difficult. Hopefully without increasing the already heavy load on reviewers.
(It very much seems like an incomplete solution -- we only have to look at politics and regulatory capture to see how far wrong things can go, in ways immune to straightforward interventions. Really, you need to tear down as many of the obstacles to a culture of trust as you can. Taping over the holes in a leaking bucket doesn't work for long.)
> In a well-publicized case in 2014, organizers of the Neural Information Processing Systems Conference formed two independent program committees and had 10% of submissions reviewed by both. The result was that almost 60% of papers accepted by one program committee were rejected by the other, suggesting that the fate of many papers is determined by the specifics of the reviewers selected and not just the inherent value of the work itself.
With this much demonstrated discrepancy between two sets of reviewers, it’s hard to believe that adding a validation step would produce a consistent improvement. How can people be expected to find improperly accepted papers when they have less than 50% agreement on the acceptance of good-faith submissions?
Honestly I think this seeming randomness in acceptance is at the heart of why people might think cheating is acceptable. If the process is not reliable, why bother submitting to it?
I referenced Kahneham's latest book, Noise, above but this is exactly the problem he focuses on. There are solutions.
Suppose that two reviewers independently rank papers 80% on quality and 20% on chance factors. With good odds, the two reviewers will agree with each other on the relative rankings of any given pair of papers. But their lists of the top 10% of papers will largely not be in agreement with each other.
Back in grad school a colleague of mine spent nine months on an experiment in a new field and submitted it as a paper to a quality journal. Six months later, the paper was rejected for lack of novelty: One of the reviewers had found a paper with a figure-by-figure duplication of the same experiment -- published on arXiv a week before the rejection decision. Both the managing editor and the author on the arXiv paper were from Chinese universities.
We wrote a rebuttal and submitted a complaint to the journal editor, but no justice was forthcoming. My colleage switched research directions to avoid the collusion and now takes pains not to submit papers without a coauthor who has enough clout in the field to deter blatant research theft. He also avoids dealing with editors from institutions in China.
He ended up graduating two years later than planned.
Some have been posting their papers to pre-print servers and then skipping straight to commercialization attempts. This is especially concerning in the health and fitness world, where some supplement makers and fitness gurus are uploading documents to pre-print servers to give the illusion of being published authors. Casual observers may not be able to tell the difference between published, peer-reviewed papers and some random document uploaded that has a DOI on a pre-print server.
This doesn’t carry much weight in academia, but it can fool non-academic observers. I’m not sure if or how it will translate to CS papers, but I wouldn’t be surprised if skipping peer review becomes more common as the pace of publishing increases.
This is probably a controversial opinion, but I think this makes sense anyway. Peer review & editing from journals made a lot more sense in the world where physically publishing, printing & distributing papers made out of atoms was expensive and difficult. And where retractions and corrections were near impossible, and where we didn't have systems for tracking reputation.
More and more I imagine research becoming like blogging - where "publication" happens by first putting your work online, and then getting feedback in the public domain. And "journals" are replaced by sites like HN or Mastodon which aggregate content and form focal points for a given community.
It won't be perfect, but neither is the current system. And speaking as a mostly independent researcher, the idea of signing over copyright of my work for the privilege of putting my work on their website is preposterous.
Sunlight is the best disinfectant for the kind of corruption described here.
That honestly sounds like it either would be a step backwards, or it would end up re-inventing peer review. I think one of the basic functions of peer review is to provide a basic authoritative quality filter to help make the fire-hose manageable. As someone who lacks the infinite time needed to check everything myself, I find those kinds of filters valuable.
Peer review is good. Pre publication peer review is bad. Bring back the pre WW2 system. End the enormous waste of reviewer time and ludicrous delay. If it was good enough for Einstein it’s good enough. The only person deciding if it’s good enough to publish should be one editor.
IIRC, the current peer review system was created because academic specialization and the quantity of papers increased mid-century to a point where the pre-WWII system became unworkable. Specialization and volume have continued to increase, and it's hard to see how that makes the old system workable again.
We called it Assembl Chronos. It’s now available here at https://provenance.cerebrum.com. Please give it a try, I’d love to hear your thoughts :)
(More info: https://www.prnewswire.com/news-releases/assembl-chronos-a-b...)
It's easy to complain about China but that's what the Mainland Chinese system is designed to do. If they're able to break ours then that's just survival of the fittest. Adopt of perish. Who says you have to roll over and simply let them get away with it? Why are our universities collaborating with these "researchers"? Name, shame and blacklist them.
I wonder if it's worth staring a web-site where friendly scientists put papers side by side and say 'Was this a Rip Off'? Whereby at least the scammers get some form of possible public shaming for as long as the paper exists.
I can just imagine that site coming up in web searches ... how could people not click it?
I don't care how many graduate students you have, there is no way all of that is original research. And in my case, it was an obfuscated version of a very well known theory that any freshman would know.
Also, a DMCA request to arXiv would likely work.
> In no case does copyright protection for an original work of authorship extend to any idea, procedure, process, system, method of operation, concept, principle, or discovery, regardless of the form in which it is described, explained, illustrated, or embodied in such work
I wish I had something more substantive to add, but, all I can say is I hope your colleague knows that even if the swindlers and plagiarizers take our research, years of our lives, etc, be they in China or wherever the whole wide world else ain't able to take away the heart of a real one. They know the research ain't theirs, and they know they depend on real producers to be able to commit their crimes, or do anything actually useful and it's not the other way around. These plagiarists are parasites, and one day we'll be free of them.
As long as metrics like impact factor and citation count determine the trajectories of academic careers, plagiarism, collusion, and other forms of academic dishonesty will not go a way.
The fact of the matter is that most academics do not have time to dig into whether someone's work is intellectually dishonest. Being dishonest has a huge payoff as long as you don't cause a scandal.
Anyone who plans to stay in academia should understand that the metagame has changed over the last 50 years. Largely because human attention has not scaled with rate at which academics are exposed to and expected to assimilate new information. There is a larger payoff to exploiting the lack of attention than to earnestly carrying out some meaningful research program. At the very least, do both. By no means do only the latter.
There are metrics and citations being taken, of our dishonest acts. They determine the trajectories of virtually everything relevant to our lives, and can have momentous impacts on the lives of others. I cannot judge any other man or woman, but I know I would be a coward, a fool, and a fraud to diversify any kind of portfolio of mine in this life, by consciously choosing to sprinkle in some exploitation or lies.
I should have been clearer. If you want to stay in academia, not debasing yourself comes at a high cost in terms of your career. It is better to leave than to debase yourself.
Academia is a system worth destroying. Participate in its destruction. Leave. There are so many exciting opportunities to continue doing meaningful work outside of universities and research institutes.
Now there's a system worth destroying.
I used to be enamored of academia and its promise. I am now glad there are increasing avenues for success for researchers who choose to leave it, given the limits of their capability to express their ideas honestly with the hope of being recognized for valuable work.
I have no experience in this area, and I am probably going to ask a dumb question: why not always submit the paper to arXiv before submitting it to a journal to protect against this kind of theft? Wouldn't that clearly and indisputably establish priority?
I'm trying to understand your story because many details are very different from the things I know about scientific publishing.
Your colleague was a grad student who researched and conducted an experiment in a new field without a supervisor (you say there wasn't a coauthor with enough research cloud)? This never happens in my field and pretty much any technical field I'm aware of.
It then went to the review process and was rejected six month later because a reviewer found (a "copy" of) the paper on arxiv? And the rejection reason was lack of novelty? In many fields arxiv does not count as a "already published" . In particular if the submission date of the article is before the article showing up on arxiv.
Also you say the article on arxiv was very obviously copied. So did this result in plagiarism investigations? While there are many things wrong with the current review process, accusations of instances of plagiarism are typically dealt with very quickly and in my experience pretty much always results in the editors in chief getting involved. So did this happen? Also in my experience there is generally much more scepticism against chinese authors than western authors when this happens. The quality of research and publications from China has dramatically increased in the last 5-10 years though.
I clearly pointed out some gaps in the story that everyone vaguely familiar with scientific publishing would find, that's why I'm asking for clarification.
A supervisor can easily not have "enough clout in the field to deter blatant research theft". If you're a tenured professor from a random university in a random country and are not among the top few names in your field, you probably can and will supervise theses, but good luck convincing an editor from an elite institution to care about you.
And in your last two paragraphs, I think you're being a bit naive... what if the editor in chief is directly involved, or is a friend of the plagiarizer? Whom do you turn to? How do you prove that you did the experiment first if there is no proof but the submission, which is in control of the editor in chief?
I fortunately have never been the victim of such a despicable ploy, but I have seen all sorts of malpractice in journals, including editors that didn't care about blatant plagiarism. I don't think it's a predominant thing, fortunately there are plenty of honest journals and honest editors. But it happens. I'm quite familiar with scientific publishing, having authored a good number of papers, and I find the story perfectly believable.
This is so unfair, some cheaters from China got free research papers to their name and the legit author had to waste two years of their life
Could you clarify whether the duplicate paper was also submitted to the same journal or elsewhere, and if submitted whether it was accepted?
I'm trying to figure out whether the point was to steal credit, or to spike your colleague's submission.
> We wrote a rebuttal and submitted a complaint to the journal editor, but no justice was forthcoming.
Yeah this is the sort of thing that seems to only ever get resolved if a public stink is made, often on social media these days, because the integrity of both the editor and the journal is being implicated, which ends up with the can of worms[0] being swept under the rug if at all possible.
Of course people are going to keep tripping over the bump in the rug, and your colleague might not even have been the first victim.
[0] Having to check everything else the editor has ever done for similar misconduct, potentially retractions galore, making victims whole, process improvements, etc.
One critical vulnerability in the current reviewing pipeline is that the reviewer assignment algorithm places too much weight on the bids. Imagine if you bid on only your friend's paper. The assignment system, if they assign you to any paper at all, is highly likely to assign you to your friend's paper. If you register duplicate accounts or if there are enough colluders, the chance of being assigned to that paper is extremely high.
Fortunately, this is also easy to detect because your bid should reflect your expertise, and in this case it doesn't. What we showed in our paper is that you can reliably remove these abnormal bids. It's not a perfect solution, but it helps.
I wonder if we could randomly assign reviewers but allow the reviewers to self-report a level of familiarity on the subject matter in general (ideally in advance) and on the paper topic in particular.
"The colluders hide conflicts of interest, then bid to review these papers, sometimes from duplicate accounts, in an attempt to be assigned to these papers as reviewers."
It might also help to attack the ability to create duplicate accounts. Given how relatively few professors exist in the world I'd assume you could put a lot more effort into duplicate account detection than they are right now.
I was actually very surprised that it is possible to register duplicate accounts at those CSE conferences. We get send a single invite to our work address and need to lock into the system using that email. And we are being nominated to get onto the committee.
It's, of course, a very hard problem to solve. It takes a lot of effort to evaluate the real impact of research.
"How to get an objective rating in the presence of adversaries"
It is probably extensible to generic reviews as well... so things like the Amazon scam. But in contrast to Amazon, conference participants are motivated to review.
I honestly don't see why all participants can't be considered as part of the peer review pool and everybody votes. I'd guess you run a risk of being scooped but maybe a conference should consist of all papers with the top N being considered worthy of publication. Maybe the remaining could be considered pre-publication... I mean everything is on ArviX anyways.
So instead of bids you have randomization. Kahneman's latest book talks about this and it's been making the rounds on NPR, NyTimes etc...
https://www.amazon.com/Noise-Human-Judgment-Daniel-Kahneman/...
However, they review a limited amount of papers (e.g. 3) - "everybody votes" presumes that everybody has an opinion on the rating of every paper. That does not scale - getting a reasonable opinion about a random paper, i.e. reviewing it, takes significant effort; an event may have 1000 or 10000 papers, having every participant review 3 papers is already a significant amount of work, and getting much more "votes" than that for every paper is impractical.
It's unfeasable and even undesirable for everyone to even skim all the submitted papers in their subfield - one big purpose of peer review is to filter out papers so that everyone else can focus on reading only a smaller selection of best papers instead of sifting through everything submitted. The deluge of papers (even "diarrhea of papers" as called in a lecture linked in another comment) is a real problem, I'm a full-time researcher and I still barely have time to read only a fraction of what's getting written.
This issue reeks with the rank smell of base politics and in/out group dynamics, and humans have been fighting, in the abstract, these issues since the time Egyptians were building the pyramids.
How can there possibly be an "objective rating" when career advancement, peer respect, and big money all are in the mix depending upon results?
In other words objectives are deceiving and rating is based on objectives.
Example: mRNA inventor being sidelined at her university when her method wasn't famous
Example2: Schmidhuber inventing stuff and being forgotten because data and compute were just too small back then
It's all about building a diverse collection of stepping stones. Any new discovery might seem useless and we can't tell which are going to matter years later, but we need the diversity to hedge against the unknown.
Take for example a paper that presents a very innovative method, but with subpar results; and another one that presents an incremental improvement on some existing method, but with results that advance the state of the art. Which is better?
Even if you ask knowledgeable, careful and honest reviewers, you will get contradictory responses, because it's highly subjective whether you rate originality as more important than results or vice versa (and other factors, like whether you think the first method can eventually be improved to be useful or not, which is often just an educated guess). I see this happening all the time, and I don't think it's something that can be "fixed", it's just how humans work.
After that event, SIGARCH launched an investigation. After a couple years, here were the results of that investigation.
https://www.sigarch.org/other-announcements/isca-19-joint-in...
Worth noting is that the investigation actually initially found __no__ misconduct. Imagine that? A student kills himself, and you conclude it was the victim's fault, and not the environment that drove him.
It was only until this post [1] emerged that they relaunched the investigation.
[1] https://huixiangvoice.medium.com/evidence-put-doubts-on-the-...
> It should have been unnecessary that we expose these evidence and challenge the result of the investigation, if the committee can drive a responsible, transparent and thorough investigation
An interesting question : Did ACM require the followup Medium article to update their position? I don't know the details of the case. However, merely updating positions when situations are black and white are some of the easiest scenarios. I wouldn't be impressed if black and white situations are assessed as black and white. This doesn't mean that one shouldn't do so. I'd expect those scenarios to be a bare minimum requirement.
Did I miss something? All I can gather from the announcement PDF is that “several individuals” have been disciplined to varying degrees, the least severe being just a warning letter. No names named, no other details. Most of the announcement was just reiterating they took the investigation seriously. Kind of hard to determine from the announcement what details have and have not been considered, and whether certain individuals have been punished too lightly, no?
The announcement does mention a confidential report has been submitted for further review. Did anything concrete ever come out?
(I suppose it would at least be relatively obvious after a while which individuals are subject to a 15-year ban.)
You almost forget now days that "getting a PhD" or "getting a tenured position" has nothing at all to do with the process of scientific research.
As long as you are happy to explore a field that doesn't require millions of dollars of equipment and aren't competing for fame, there's nothing stopping you.
And being landed gentry probably went a long way as well.
And of course, I could choose to delegate the trust, and to “follow” someone, which would mean to incorporate their rankings, especially in areas where I don’t orient that much.
Do you think this would work?
I do agree that there's probably some cleverer solution on a personal level, but I think the journal system exists as a kind of guard against untrusted actors, and yet fails.
Are you asserting a 'trust' whitelist, a 'distrust' grey or blacklist, or some combination?
Are assertions meant to be linked to or backed by evidence?
Are these assertions publicly visible?
Is there a limit to the number of assertions that can be made?
I think you can see how such a system might devolve into formalized collusion.
*> And of course, I could choose to delegate the trust, and to “follow” someone
In the absence of explicit delegation or following, is trust/distrust intended to be transitive at all (ala PageRank or Advogato WoT)?
I would like to more about this. Can you point to any interviews or other materials that shed light on Burke's thoughts on academia?
This has nothing to do with collusion, but some in this thread are saying that just the number of papers is indicative of a problem.
Once it's no longer about being in the esteemed and scarce "10%", they won't bother because they don't need to. Imagine a process where the only criteria are technical soundness and novelty, and as long as minimal standards are met, it's a "go". Call it the "ArXiv + quality check" model.
Neither formal acceptance to publish nor citation numbers truly mark scientific excellence; perhaps, winning a "test of time award" does, or appearing in a text book 10 years later.
It's also noteworthy that it seems that Chinese engage in this quite a bit, either due to their culture that does not forbid cheating (e.g. see how they pass the GRE or other standard tests) or because there are a lot of them.
Chipping off the tip of an iceberg isn't a good long term strategy.
I guess the system is inherently broken after all?
"Academic politics is the most vicious and bitter form of politics, because the stakes are so low."
It has predictable results. Where are we going to get reliable research, and anything else, if we can't trust each other. Trust is an incredible business tool - highly efficient when you can take risks, be vulnerable, and don't have worry about the other person. Trust is an incredible tool for personal relationships, for the same reasons, and because if you can't trust them and can't be vulnerable, you have a very limited relationship.
I'm not sure that you intended it this way, but this reads as very oblique (i.e., "wink and nudge"). Which subculture are you referring to, and what particular relationship do you think they have to research in Computer Science?
I am referring to no particular subculture. Lots of people around me embrace it, including from all over the political spectrum (if that's what you are thinking).
I think the broader society sets the norms for computer science, as with everything else. For example, when star athletes like Barry Bonds, or entire teams like the Houston Astros, or much of college sports, cheat with few reprocussions (and in the past, that wasn't the case - players were banned and school sports programs were basically shut down, etc.) that affects computer science research.
Arguably, it's the opposite. Once people realized academics could do this kind of cool shit, they got showered with money and told to do whatever the they want. That's how we got the incredible scientific and engineering advances of the second half of 20th century.
Then the beancounters started asking questions about what the money actually buys, and research quickly turned into another short-term, self-contained, profit-chasing game, starved for resources and only occasionally producing something actually useful.
As it is, if our researchers are spending almost all their time thinking about and doing things other than research, what do we expect?
On a tangent, software industry has a bit of similar problem, with the best developers being forced to enter management roles[0] instead of solving technical problems. That is, a developer progresses from doing shoddy work to doing mediocre work and then, just as they start doing high-quality work, they get told to manage a new cohort of juniors doing shoddy work instead. I wonder if that's why so much software is hot garbage these days.
--
[0] - Whether proper ones on management path, or "fake" ones like principal developer, where you get all the managerial responsibilities with none of the authority.
Thaaaaannnkkk you. It has technically been reserved for Christian Aristocracy, which is practically a world where there is no desperation for calories. Instead many of the greats had an anxiety of their immortality (see Fourier).
It's up to you and me. Nobody else is coming to save us.
One way we can make the world better is by fixing the rules. Getting rid of ones that are unjust, and making the rest more consistent.
I once saw sign on a street: do not do something (do not remember what) and reference to a City Bylaw numbered as 37 thousand and something. That is just for one city. Good luck changing this sheer insanity.
And I think it is. At this quantity the quality will definitely suffer. Besides, I did read some bylaws at some point out of curiosity and without going into details many of them are outright unjust/deficient/etc (in my opinion of course)
The question of course is, peoples interpretations of which laws are just and unjust are subject to bias and individual incentives.
Incidentally, while I recognize thr popularity of this quote, its fairly ridiculous taken literally for laws which are prohibitory rather than obligatory.
Viewing a prohibition as unjust does not obligate me to violate the prohibition; believing people should be free from government constraint to do something doesn’t require me to do that thing.
“Disregard” or “discount” in place of “disobey” would be more generally valid.
So outlawing theft, rape, fraud, kidnapping, false imprisonment, etc. - that's all unjust?
https://en.wikipedia.org/wiki/Joseph_Tainter
Quoting from the book:
“Sociopolitical organizations constantly encounter problems that require increased investment merely to preserve the status quo. This investment comes in such forms as increasing size of bureaucracies, increasing specialization of bureaucracies, cumulative organizational solutions, increasing costs of legitimizing activities, and increasing costs of internal control and external defense. All of these must be borne by levying greater costs on the support population, often to no increased advantage. As the number and costliness of organizational investments increases, the proportion of a society's budget available for investment in future economic growth must decline.”
We used to have strong institutions that were supposed to help push groups into not choosing the bad corner of the prisoners dilemma, but they all seemed to have degraded. I suppose they could have always been like this and the curtain has just been removed, but I’d argue that the perception that following the rules was the best personal choice is almost as valuable as it being true
This animated gif makes it clear: https://invisible.college/reputation/declining-trust.gif
This trust is greatly eroded by calls to eliminate direct taxes on land holders. In America the local property tax is the most important thing holding society together. It provides residents with assurances that regardless of how corrupt the public process for distributing legal tender becomes, that the richest cannot simply buy the entire country and turn the continent into a private estate which their descendants will inherit in perpetuity without paying enormous taxes to everyone else.
When James Madison organized the assessment of property taxes at the national level during his presidency it resulted in the 'Era of Good Feelings' and a relative low point of political polarization. In contrast when state governments introduced sales taxes for the first time to reduce property taxes it prolonged the Great Depression, and when NYC gave the largest property tax abatements in the country to Donald Trump it lead to the Trump presidency which increased political polarization.
(1) If you lose trust in someone, you'll be less likely to try to find truth in their statements. Finding truth in someone's statements requires time and attention, and we'll invest that attention in people we trust.
(2) If you find falsehoods in their statements, you will lose trust in them.
Overall in Psychology, trust is a primary indicator of a relationship, marriage, partnership, or organization succeeding or falling apart. It's both a symptom and a cause of all other factors.
The classic example of the later are the politicians who rant “if only these people would get married and stay married, they wouldn’t be so poor” and neglect to consider that impoverished communities create the conditions for rampant single motherhood. The desire to raise children does not magically vanish merely because there are exactly zero worthwhile men in your community that aren’t your father’s age.
In the old days you had to know your place. WASPs smoked cigars and ran things. Those old guys drinking sherry and wearing tweed helped each other out. The Irish were cops, Italians firemen.
In tech it’s pretty obvious to see various constituencies doing dishonest shit help others out.
america has urbanized rapidly in the last half century, at the same time that family formation has broken down and life-long jobs have become a thing of the past. we are atomized and thrust into constant competition. i don't mean to idealize a past that i did not even experience, but there is something to be said for having roots and knowing your neighbors. we arguably have more opportunity at the cost of stable identity -- reputation and trust naturally accrete around the kind of stability we lack.
if you talk to older people, people around my grandparents' age or thereabouts, you will hear that they no longer recognize america, that it is fundamentally different than the culture they grew up in, in terms of values. i find myself thinking about this a lot.
I think knowing your neighbors is overvalued. My evidence is Tokyo and living in transient, largely ethnically homogenous sharehouses — in ethnically homogenous areas — for long periods of time.
From the outside this degradation of values and "social lawlessness" has been apparent for years now, especially since 2016.. Really hope this doesn't spill over.
Oh, and good lucking mending this.
More specifically, it's the perception of a breakdown that drives this behavior.
When it comes to cheating, there's a growing perception that "everyone else is doing it" and therefore it's not wrong to play the same games as everyone else.
The current political and social media discourse revolves around ideas that "the system is rigged" combined with a die-hard notion that anyone who disagrees with you is wrong and/or evil. When people are bombarded with these ideas every day on their social media feeds, cheating a little bit to get yourself ahead doesn't feel like cheating. It just feels like leveling the playing field.
Depends on how you define "lose". And also on what you think the "rules" are.
For example: most drivers in the US routinely exceed the posted speed limit on roads. Is that "violating the rules"? In a legal sense, it is, since if a cop catches you he can give you a ticket and you pay a ifne and points go on your driving record. But nobody considers you a bad person for doing it, and I would argue that doing it, if you don't cause an accident, is not harming anyone. However, obeying the speed limit is also not considered a bad thing; all it really means is you get where you're going a bit slower. The tradeoff is yours to make; you don't "lose" by choosing to obey the posted rule.
Now consider an example at the other end of the spectrum: all of the shenanigans with mortgages and the financial system that caused the crash of 2008. Those who "followed the rules" leading up to the crash--for example, those, like my wife and me, who limited our mortgage and the size of house we bought to what we could comfortably afford--did not "lose". Sure, the value of our home went down, but we had no need to sell it then. It was still a house and we could still live in it just fine. Sure, we didn't have a bigger house with more bells and whistles, but we also didn't have to worry about what might happen if the housing market crashed. In other words, we made a tradeoff not much different from the one made by the person who obeys the speed limit and just gets where they're going a bit slower--but still gets there.
Basically if all it takes is getting citations, then forming a citation ring is "Chabuduo" and you'll only lose face if you are caught (not good enough).
[1] https://news.ycombinator.com/item?id=27052249. This reminds me of the Japanese buzzword bingo of earlier decades.
You are absolutely right that trust and reliability are very valuable. Societies with high trust tend to be richer and much more productive than societies overriden by cheating and corruption.
Like the anecdote that he leased a new car every month so he could avoid registering it with the state.
Getting a divorce court judge to force you to sell your Enron shares at the top, nuking the regulator’s ability to charge you with insider trading, while you elope with your younger hotter high libido stripper nymph to the mountain you bought?
These are our role models
Why should it be possible at all to game Journals in this way? Particularly in Computer Science journals where people think about edge cases for a living...
Those systems that can be built to be resistant to greed would definitely benefit from it - but I think it's more of an issue with society at large.
(The redeem/blueteam thing is kinda unethical, so maybe the University of Minnesota should do it...)
Absolutely. I think people are intimidated, demoralized (de-moralized) and where once they believed anything was possible, any social problem could be solved (even those old as history, such as women's rights, human rights, etc.), now they've somehow drunk the wrong Kool Aid, some stuff distributed by Jim Jones.
Time to get to work.
I'm a US expat and escaping this culture is one of the things that's made me happiest - I tend to call it the bullshit culture because my favorite example is... writing a good paper for class is admired - but what's really praised is writing a paper that gets good marks without ever having read the subject matter. Being able to spin lies about a topic you've no understanding of and turn that into a marketable skill is a dark potent for the future of America. I think it's always been somewhat present, but since emerging strongly out of the business world in the eighties it's gained a lot of steam.
We are a society that can benefit from cooperation where everyone gets a fair slice of the pie, but that society is eroded if we praise and not shame those people who betray societal trust and cheat the system.
At least in academia, American universities (and Western universities in general) have had a good track record overall with academic integrity, and it has set them apart. This seems to have degraded recently, perhaps because of the increasing pressure of the “publish or perish” system (or dozens of other potential causes).
We do celebrate people who excel academically seemingly effortlessly. But we don’t celebrate bullshit artists so much in school. In business, and particularly tech, it’s another story.
The difference is that these collusion rings or bogus studies are discussed and exposed publicly. It's not the case in Asia.
You can read a bunch of articles on Bernie Madoff, Martin Shkreli and Billy McFarland that romanticize the cunning with which these folks exploited others. Most pieces on them will present an overall negative tone but often feature some pretty glowing admiration of them. Let's also not forget that tax evasion by Trump was praised repeatedly as him beating the system - that's a pretty common view point, more common (especially when it comes to taxes) than the view that those individuals are failing to pay their fair share from what I've observed at least. Trumps a complicated example due to all the political baggage around him so maybe just look at companies like Apple, Google and Facebook - they regularly offshore large portions of their profits and I really doubt the people working to those ends feel any shame, instead it's likely a "beating the system" motivation.
This is a bit of a public secret, but quite widely researchers don't really trust articles anymore, if they ever did. Maybe some plot or dataset may give some insight and maybe some discussion has worthy information to ponder on. But mostly they're just some ads to put in a yet another funding application.
Most articles are just churned out to get some lines to CV or to look good in some metric. Publish or perish has turned into full-on bullshit or perish. The whole peer-review system (which is just around 50 years old anyway) is on the verge of just grinding to a halt due to the stupendous volume of hastily hacked together manuscripts.
I think many are still sort of hoping that this will somehow sort itself out. But the collapse of the quality after the explosion of electronic journals, consolidation of publishing houses and overall structure that doesn't really care at all about what is actually in the papers doesn't give much realistic hope.
It should be noted that there's sort of a "parallel reality" in academia behind the publication show. The ethos for academic integrity is still quite strong, teaching tends to be valued by the community (but not by the system) and face-to-face discussions can be of very high quality. But the signal-to-noise is so low in publishing that it's not really worth following.
We really need to get some new arrangement so that we don't drown in all this bullshit. Word-of-mouth, open data repos, conferences and just blogging and pushing stuff to git repos probably is most that's needed. The publishing structure is becoming just plain unnecessary bureaucracy.
SEO is a kind of explicit analogy. Google pagerank was modelled on academic publishing, and it worked until it went live. From that point, links started to decrease as a quality signal.. spam. Publish or perish is a similar sort of dynamic.
Honestly, I think most legible systems for determining merit have these sort of issues. If advancement, accolade, grants or somesuch are determined by a formal system, whatever that system used as a signal or metric becomes corrupted. Hence why Word-to-mouth, open data repos, conferences and just blogging and pushing stuff to git repos does work. It's informal.
A sort of reputation system is in place in almost all peer-to-peer societies, it tends to form automatically. I don't think we really need any of this weird mess of a system.
We have Wikipedia, we have open source, we have OSM, we have all sort of things that should be "impossible" given the dismal perception people have of other people. This perception is just plain wrong and really harmful.
It's a hard sell though. The cost of metering is subtle. The do-nothing tenured professor is visible.
Wikipedia, OSS, etc really are the shining beacons. Existence proof for something better. Someone needs to write The Cathedral and the Bazaar, but in non geekish.
Perhaps surprisingly to some, many in academia would just like to research and teach with some quite modest salary and don't have to think about money at all. E.g. I would gladly and with no hesitations take a €2000/month tenure and keep on doing what I'm doing just more efficiently for everybody. I've been trying to pitch this idea to the funders here in Finland, but to no avail, they simply don't care if the funding system is useful or not for the academic community or humanity, they're focusing on playing the same old (maybe 10 years or so here) application lottery that's not only waste of time, but corrupts the whole community and even the very content of thinking in academica.
"Money" in academia is really abstract as well, and when discussed its not salary, but funding for projects or students or such. And because the funding structure is so bizarre and convoluted you just see big numbers with currency signs flowing everywhere, but this doesn't seem to have much to do with anything concrete happening around.
If academia becomes a place where you can get rich, the system will be in just years corrupted into some bizarre thing where advertisers advertise to each other for the sake of advertising.
Luckily cats can't be herded.
Some of it is intentional "motive hacking." As you say, prestige, research funding and the like are as (or more) operative as salary.
Some of it is unintentional. Before publish or perish, publishing volume probably was a signal for something. I doubt it was ever a signal for high quality research, but low (or no) volume may have been a signal for low quality. Also, formal decision making bodies (like grant makers or tenure committees) tend to gravitate to quantitative, legible metrics.
Whatever the reason initially, publishing volume became a hugely important thing with impacts on many aspects of research.
At the same time, in CS especially, the number of researchers has also ballooned. That's a whole other strain on a system of, at core, knowledge dissemination.
The antagonistic view is not just towards my colleagues, I'm not particularly proud of my own papers either. I find it more a nuisance to "pay the bills" and a lot of my research goes unpublished (at least in journals) due to all the IMHO unnecessary hassle involved. Just a blog or something would be a lot nicer and probably would communicate the work better, and would ease the pretension of objectivity which I find mostly causes wrong impressions and makes writing really a chore.
> It should be noted that there's sort of a "parallel reality" in academia behind the publication show.
So what should be the guidelines for someone who is not a researcher, but an engineer, and hopes to stay informed by reading relevant papers from a specific field. (You know the folks who should apply some of that in practice)
Also individual papers tend to focus on one very specific problem at a time. This is typically related to some actual larger "debate" and can be difficult to see if one's not familiar with the larger issue. Also especially conclusions tend to have quite heavy implied assumptions that are just generally accepted in the field.
I "stay informed" mostly by face-to-face discussions and emails and such. I don't read much papers myself, but many of my colleagues do and I just hear from them, or ask them if there is new stuff around related to something I'm pondering.
To get an overall view of "state-of-the-art" I'd recommend starting with masters' or doctoral theses. These typically require more elaborate presentation of the background and its typically put out in more readable terms with less assumptions of the readers background knowledge.
In some fields review articles are a good starting point as well, and they tend to briefly sum up the required background, but my understanding is that some fields don't do those much.
If you read "random" articles, I'd do a quick smell-test before digging in. See if code is available, ignore papers with clear hype in the abstract off-hand. You can also "navigate" the field by following citations, although this can be technically annoying as the publishing format is still tailored towards print, even though very few journals are actually printed anymore. If you hit a paywall, try sci-hub or just move on to a next one unless you're looking for something really specific.
If you have something more specific in mind, just email or call or go talk some researcher that looks to be doing something related to what you are looking for. Researchers tend to be quite eager to answer to the public of their stuff, and its seen as sort of a public service duty as well. Depends on the researcher quite a bit though. Maybe a good starting point would be somebody a bit "lower on the ladder". Maybe a postdoc or a PhD student (this depends on the country as well). Professors tend to be busier and actually may not be that up-to-date with their field (especially on technically detailed level) as they spend most of their time in administration and the funding ratrace.
Depending on the country you can just attend lectures too. At least in Finland university lectures are public by law (with some restrictions on e.g. practical lab stuff etc). You can see if the lecturer doesn't seem too busy after the lecture and just go and ask.
You can also just try go to conferences. They usually have a fee in theory, but I don't think you'll be turned away if you just browse around for posters or so, especially if its a smaller one. The fees are just sort of a scam (long and sad story) and researchers organizing the thing usually don't care about the fees at all.
For some fields there are some good youtube channels that provide summaries that can get you started. E.g. Two Minute Papers is good for machine learning/machine vision/"AI"/etc related stuff: https://www.youtube.com/channel/UCbfYPyITQ-7l4upoX8nvctg For computer graphics SIGGRAPH "video papers" are really nice and even entertaining: https://kesen.realtimerendering.com/sig2021.html
Hard to give more specific tips with such broad question. If you have a field or topic in mind, I could maybe give something more concrete.
My comment was on the historical status of academia in the US as a whole (think last 120 years), not just the current state globally.
You’re lamenting the quality of academic publishing in particular. The US now publishes less than 17% of science and engineering papers, but its papers are often the most highly cited. So yes, there has been a huge increase in the number of papers, and number of low-quality papers, but this isn’t necessarily being driven by the US, as the original comment would have implied.
You claim researchers don’t really trust articles anymore, and I agree that it takes a lot more work to filter out the noise now, and I’m less optimistic that authors are presenting an honest, objective appraisal of their results. But significant research is still happening, and academic publishing is still the primary way that information is disseminated. People seem to rely more on name recognition (author, school, journal) now. It probably varies field but field. I’m in a field where results are often proof-based and that tends to be harder to fake.
Internationalism is so ingrained in the academic culture (at least on fields I'm familiar with) that it doesn't even really register what country somebody's from or is working in. There are definitely some differences especially in the more "overt" parts of the culture (hats and robes and different titles etc), but these are of very little significance for anything but some ceremonies.
My working experience is from Finland, Sweden and UK, but in academia people come and go between countries very frequently so colleagues tend to be from all over.
There are at least some stereotypes that some countries are more prone to the e.g. citation rings, but I don't find that very relevant, as I think the whole system is quite broken and the publishing (at least in English language) forums are typically not country specific at all. Probably something like this happens in more or less any country.
This was not my experience of school in the US. Just as one example, two words: Cliff's Notes.
Really? Cause I'm an American who has lived, studied, and worked overseas. Let's just say it's not American (or generally Western) coworkers and classmates who are NOTORIOUS for cheating.
And I think many of us here who've attended "diverse" universities or work for companies with "multicultural" staff have a pretty damn good idea which cultures and nationalities are more likely to be cheating.
I don't think this is an American thing. In fact, I've never seen this praised anywhere outside of maybe a few people back in High School.
I worked for a company that expanded rapidly with distributed offices all over the world. One of the growing pains we had was that the managers from certain countries, America included, were very trusting by default. This opened the door to a lot of manipulation from employees in certain other countries (which I'm deliberately not going to name) where getting away with a lie was more or less considered acceptable as long as you weren't caught.
At once, such a great, and terrible, sentence.
At once such a great, and terrible, sentence.
Not that I ever believed him much about that (I do think he believed what he said...).
Anecdotally, I've heard stories about Zuckerberg confessing/bragging about all sorts of nasty things at these dinners.
Really, this stuff should just be shamed. Sadly, too often calling out bad behavior just gets you called a "hater"...
This coworker told me he and his "startup":
- routinely lied to potential customers on the size of their client list
- misled clients on the depth and completeness of their product
- blatantly broke CA laws to cut cost corners
All of this was done to secure contracts in order to secure more funding. "Always be selling", he said.
He literally fucking said to me that he learned to "be dubious, not deceitful" which is probably one of the most deceitful things I've ever heard.
Made me sick to my stomach and pretty much validated (1) why I never moved to SF in the first place, instead moved to NYC and (2) how much of a fraud YC has become. Absolute fucking madness.
This almost necessitates some cheating to survive and the only people left are those who survived this system and hence the culture slowly rots.
Both “Athens and Jerusalem” are principled according to honest toil. And look at the results!
Having 'hustle' is actually important at a startup, it's part of the essential aspect of it. I'd argue a 'hacker' has a kind of hustle.
I'm pretty suspicious of these things as well, but I've also come to believe in my many years that a bit of koolaid is fine as long as it has self awareness.
I wish Elon would stop posting all the stupid things he does, but I think that's what you get with 'all the other stuff'.
If Elon were fully polite, conscientious, a 'good listener' I'm not sure Tesla would exist. So we pay for the existence of Tesla by accepting that he's going to do dumb tweets about Crytocoins.
I think that's a BS excuse for bad behavior. How would being polite and concientious harm Tesla? It's simply that he has power and has low standards for his behavior.
I know plenty of successful people who do what Musk fails to do.
Except you don't know any successful people who've remotely done what he's done either.
Talking about 'Dogecoin' is a little unseemly, but it's nowhere in the realm of 'toxic' or 'bad acting'
Without massive public support and sympathy, Telsa wouldn't exist, it's a movement as much as anything, and so you need kind of a showman.
His appearances on SNL etc. are part of that public drama that keeps Tesla stock going with enough support to keep the legitimacy of the dream alive.
I'm seriously doubtful that a quiet, unassuming person would have been able to do most of that.
Expressive, bombastic characters will by virtue of the volume of their actions, sometimes creep up to the line. It's normal. There's nothing wrong with Elon, he's just a little cheezy and spouts too hard with some things.
(I'm trying to respect the HN tradition/guideline of not talking about votes. I don't really care about Internet points; I think the implied interest in the issue is worthwhile and applicable in this case.)
Only a tiny percentage of developers seem to actually enjoy coding - Most of them have no interest in it and only see it as a mechanism to acquire money, power and influence.
Disinformation is rampant because contrarians are punished and conformists are rewarded. The rot is deep in the guts of the industry. Those who have the most power and the loudest voices hoard all the attention for themselves and are unwilling to give exposure to any alternative views - Their deep insecurity drives them to surround themselves only with yes-people and to block out all critics; avoiding disagreement at all costs... Downvote, suppress, censor...
Powerful people in this industry need to put aside their insecurity by embracing disagreement, allow themselves to change their minds, and give a voice to contrarian views and ideas; even when it risks hurting their current interests.
Powerful people should seek the truth and try to promote the narratives which make the most sense; not the narratives which happen to benefit them the most. Everyone is free to move their money to match the changing narratives, so why do powerful people invest so much effort in keeping the focus on narratives which only maintain the status quo? To protect their friends? To protect the system? That is immoral - Capitalism was not designed for this kind of arbitrary altruism. For every person you 'help', you hurt 100 others.
As much as people love to bash Elon Musk right now, he should be applauded for constantly trying to adapt to the narratives which make the most sense as opposed to rotting in his own filth and succumbing to tribalism like everyone else.
I realize you might be a human with feelings, but your screed above has a curious structure that looks... off somehow.
Anyway, if you are genuinely angry and wrote the above, I acknowledge your emotions, and don't have anything else to add.
Definitely written by either a human or a really bad AI.
Most humans don't have enough background knowledge or sufficiently diverse life experience to make sense of this.
I'd have to write a whole book to explain my reasoning behind this statement. I got a lot of my knowledge from HN articles and comments so it should tie in nicely.
How many industries have you worked in? This is fairly common in many walks of life. As I get older I'm getting better at identifying the lunatics in charge. Often they are nice people, but also cause massive amounts of chaos, because they have no clue what is going on. No wonder they feel the need to exert micro-control.
Anyway, thanks for the rant. Always interesting to look at the bottom of the barrel for the HN rejects :-)
Fundamentally, there is no way to fix it, because the system is self-disregulating.
In other words, there is no mechanism to bring the academic system back to honesty over time; there is a mechanism to make it more dishonest over time.
This is inherent to government solutions. In modern times, democracy was supposed to be the regulating mechanism of the bureaucracy, but obviously isn't working. The bureaucracy itself certainly is disregulatory. Modern academia is simply an organ of the larger government bureaucracy.
In perhaps a generation, a similar specialty might be bootstrapped and begin to take on problems had been of interest in the old one. What to call the new specialty will be its smallest problem.
Why do you say that? Do you have any experience in the field?
Your position also paints CS in very broad strokes; in my experience, the only commonality between some subfields of computer science is that they use computers. Graphics, hardware architecture, programming languages, networks, and so on, are all essentially loosely coupled with their own organizing communities and directions. Some of these subfields are more closely tied to mathematics or electrical engineering than strictly to other parts of computer science. If there is an incurable "contagion" that afflicts all of these, I must admit it hard to believe that this contagion would not prove (if not already be proven) effective beyond the artificial confines of the term "computer science".
Which if one looks at the state of economics right now, is an object lesson on this kind of stuff never ending well.