Ending an Ugly Chapter in Chip Design
spectrum.ieee.org
spectrum.ieee.org
There's an enormous amount of relevant details missing from this article and I don't know why. Why is there no response from Kahng to G&M's rebuttal of his experiments? What did G&M say when told their Github posting wasn't reproducible? Did they agree it was a problem or not? Why did G&M leave Google if the person who was personally undermining them was fired? What exactly did Chatterjee do to personally undermine G&M, and why?
ieee.org's editors are not doing a service to their members by making a chip design argument about AI a gender issue.
The tangents people, even here at HN, take viz-a-viz these topics, are often mind-boggling. And HN generally has less bot protection than places like reddit, too.
The independent reproduction that the IEEE article talks about also tested the method from the leaked paper from Chatterjee and concluded that his method indeed beats CT in almost all cases at a fraction of the computational cost.
The response from G&M to the independent reproduction emphasizes their method was used for production hardware at Google, was in a peer reviewed paper, and that their GitHub repo had a lot of forks and stars (I'm not making this up). Near the end it then raises several technical objections.
What is even the point of benchmarking against a learning based method of you don't let it finish learning? Quality of output matters way more than compute cost.
Some background is EE professional and academic culture has, for more than a half century been ... less than fine with having women colleagues. Lots of ugly. Decades of improvement too, reflecting a whole lot of years of work from lots of people and leadership. But... CS changed faster.
So an IEEE pub is pointing out that an IEEE-affiliated conference reportedly had... an excursion from professionalism baseline.
https://www.wired.com/story/google-brain-ai-researcher-fired...
The rebuttal was published less than 2 weeks ago, so that seems a bit early, and it is public, so reading it would answer several of your questions.
The ACM article also has some relevant details: https://cacm.acm.org/news/271439-more-details-but-not-enough...
They should have made it extremely clear in their original manuscript their CT results are against a non-pretrained version that was not always trained to convergence, and so the CT results presented are a just a lower bound on performance.
They got confirmation from a Google engineer on only one block, which they trained for 1 million steps. For the other blocks, they didn't train anywhere close to that many steps, and their tensorboards don't show convergence on total cost.
ML is difficult! It's not enough to confirm that your implementation and installation are correct - you also need to train to convergence. You are free to complain that this is onerous or expensive. It's a common problem in academia vs industry in other areas of ML -- ML takes a lot of compute, and academic budgets are extremely tight. That said, CT is still orders of magnitude smaller / cheaper than most modern ML models (not surprising! CT is pretty old at this point by ML standards (3 years is a long time in this field)).
Kahng's FAQ admits that pre-training led to a better final proxy cost in the Nature paper, and that it significantly reduced compute time, and that this is the method that the Nature authors recommend and used in their Table 1 evals.
This whole situation is such a shame. Prof Kahng wrote a really great article on CT a few years back, and it gave me hope that the whole field was going to embrace ML and really take chip design to the next level: https://www.nature.com/articles/d41586-021-01515-9
Most of the field in fact has, but Kahng himself seems to be backsliding here.
Unfortunately, I suspect that Kahng's been listening to some pretty toxic people (the same day his paper was posted to arxiv, it appeared in an SB author's court filing). It seems like his group ran into some difficulties jumping into machine learning, and rather than take that as an opportunity to grow and learn, they took that as an opportunity to attack.
While I'm glad that others in the field have built on the work successfully (see Appendix of https://www.annagoldie.com/home/statement ), it would be really tragic for Kahng himself to get left behind here, and so so unnecessary!
It is also covered in the ISPD'22 paper, "Scalability and Generalization of Circuit Training for Chip Floorplanning", by Summer Yue, Ebrahim M. Songhori, Joe Wenjie Jiang, Toby Boyd, Anna Goldie, Azalia Mirhoseini, and Sergio Guadarrama, which was written by authors from the Nature paper. https://dl.acm.org/doi/pdf/10.1145/3505170.3511478
In their ISPD'22 paper, link above, Figure 7 shows a 5x speedup but 0.3% worse placement results with pre-training on diverse designs vs. running from scratch. Pre-training on previous versions of the same design does improve results by 1.7% vs. running from scratch, and has similar runtime benefit to pre-training with diverse designs. Given that the Circuit Training approach requires 41x to 646x wall time and an even higher factor of compute time vs. other standard techniques, the 5x speedup with pre-training is insufficient to close that very large gap. Furthermore, pre-training does not improve results vs. training from scratch, unless you are pre-training on previous versions of the same design but even then a 1.7% improvement in the circuit training results is not adequate to demonstrate improvement vs. standard techniques as can be seen by examining the results from Professor Kahng's group's work.
This concern about pre-training has been answered in multiple forums, including the discussion in ISPD'23 Session 8, when Mr. Washauer-Baker raised the same issue. It's unfortunate to see Gabriel Warshauer-Baker, Anna Goldie's spouse, making repeated personal attacks on Professor Kahng. Please focus on the scientific discussion.
Thanks for what you do here, too, btw.