ISCA'20 should really have been cancelled until this was resolved.
See a timeline of our attempts to get anyone at ACM/IEEE to care: http://pbzcnepu.net/isca/timeline.html
They can be easily found in grant webpages, eg https://www.nsf.gov/awardsearch/showAward?AWD_ID=1900713
As someone who received both their B.S. and M.S. at UIUC, I'm honestly disgusted at how various professors and administration have handled certain incidents that my friends have been part of
It is easier to fight things from the outside, as the internal higher-ups have less power to mess up your life.
It's interesting you mention UIUC, as traditionally that was the source of a lot of suspect papers. It was a running joke about the "coincidence" of how many people with UIUC connections managed to get papers into ISCA each year.
of course, nothing surprises me.
The reason people have latched on to the current set of allegations is there seems to be actual concrete proof of misconduct that can be acted on.
What is this insinuation based on? Do you have data on how many papers by "people with the right connections" are rejected, for example?
i'm not sure this data is useful, because we'd disagree on "people with the right connections".
another anecdote, when first trying to raise awareness of the issue, when mentioning it to members of the community I never got the response "wow, how could this happen in computer architecture?" rather the response tended to be "wow, I can't believe it finally got bad enough that a student died"
I know there are a lot of solid professors in UIUC like Joseph Torellas, Sarita Adve (C++ consistency models), Vikram Adve (who is the adviser of Chris Lattner - creator of LLVM). This is really bothersome and I hope it's not from one of those professors.
Both use modified "cycle-accurate" simulators for the results. For now let's ignore all the issues with the accuracy of these simulators.
Let's validate how "solid" the work is. Drop an e-mail to Torrellas and ask for the code they used so you can see if you can reproduce their work. Hopefully things have changed and they'll send you the code but in my experience they'll just say no.
So they got two papers in which are unverifiable, and none of the reviewers ever saw the code involved. This bothers some people, but it's not unusual at ISCA.
There are a lot of papers in ISCA that are based on cycle-accurate simulators. It has been like that since forever. How else would you evaluate new non-existent architectures that are bleeding-edge? FPGA? Most can't even afford that. Not to mention it's just not possible in a lot of cases. I agree with you that some work should be verifiable but your accusation is weak for this part. Maybe the community can push for verifiable work before getting published in ISCA since it is such a prestigious conference.
If you don't want cycle-accurate simulator based papers, you'd have to eliminate a large variety of body of work only possible with this approach. A lot of techniques in modern processors have seen their start in processor/system simulators and most of them are probably not cycle-accurate.
yes. Most "cycle-accurate" results are garbage. Do you show error bars on your results? Can you? Did you run on a variety of independently implemented simulators and show the results on all of them? Did you run the full reference inputs to SPEC all the way through? Why not?
The answer seems to be that it would be hard. But guess what, science is hard. Try complaining to a biologist sometime about your architecture paper that took so long to write, where you gathered all the results 2 weeks before the paper deadline.
It's fine if you come up with a new idea and run some simple proof-of-concept runs to show it might have merit, but don't pretend the results from an academic simulator hacked together by a sleep-deprived grad student have any real world merit.
2. i dont understand how you can pick on "cycle accurate" simulation when every simulator in our community has problems. at least in gem5 (one of torellas isca2020 paper), we as reviewers can look at the code. how about the famous "in house x86 pin based simulator"? most pin based simulators performance numbers should rightfully be joke, but we use them anyway, because we aren't going to rewrite them.
3. at the end of the day, most of our work is unverifiable, because we make so many approximations anyway. one young faculty told me "we just need to see the idea and determine if it makes sense". i just do not know if this is the right thing to do, or if its a lie we tell ourselves.