The Hardware Lottery (2020) | Hacker News Reader

The Hardware Lottery (2020)

hardwarelottery.github.io

47 pointsahamez9 comments

The Hardware Lottery - https://news.ycombinator.com/item?id=29409415 - Dec 2021 (1 comment)

Ride the Hardware Lottery - https://news.ycombinator.com/item?id=25043592 - Nov 2020 (4 comments)

The Hardware Lottery - https://news.ycombinator.com/item?id=24515035 - Sept 2020 (16 comments)

I am looking at solving this challenge in a specific way: using high-perf, GPU-accelerated HW simulators and ML algorithms to tune a new HW architecture automatically. Best ML HW => run on it the best ML models => produce new best HW (arch) => build new best HW => GOTO 10.

Reach out if you are interested in any way.

How would you define "new best HW"? Seems challenging, particularly if it's for generation N+1 ML models that haven't been created yet. Also, while there is work underway to use ML models to guide HW design, it's not clear to me that the best ML models for that == the best ML models for more general purpose tasks in audio, visual, and natural language processing, i.e. is HW circuit design done using transformers? What are the inputs, latent space, and outputs?

GPU development seems driven by more general computational principles that might be summed up glibly as "We're hitting fundamental physics limits for single-core processors, what is the maximum amount of data we can move per second through the maximum number of cores?" Perhaps there is a way to extend this with a view toward the challenges of current ML model training and inference, but I imagine trying to approach it as a black-box optimization problem could be quite difficult.

The issue is that there is no such thing such as "best ML HW" because different ML algorithms need quite different things.

The process you describe would work to produce hardware that's best for some specific type of ML models, while the whole point of this article is that the currently "best" ML models are there not because they are objectively best, but rather because they are what's best within the bounds of the hardware architectures we have (i.e. we're not looking at best ML algorithms but rather at ML algorithms that "won the hardware lottery"), and it seems plausible that there might be better ML methods if there was different hardware that enabled them - but the process you describe wouldn't possibly find that different hardware, it would instead tune the hardware even more towards the algorithm path we're already stuck on.

I wish for graph reduction hardware, we could build amazing things with that.

https://www.cs.york.ac.uk/fp/reduceron/

https://www.cambridge.org/core/journals/journal-of-functiona...

https://github.com/tommythorn/Reduceron

also: https://en.wikipedia.org/wiki/Graph_reduction_machine

I was about to post that third link as an answer, but you beat me to it by 8 hours :)

Anyway, I think an FPGA is an excellent piece of kit to start such development.

Related: The Bitter Lesson [0], and previous discussion [1].

[0] http://www.incompleteideas.net/IncIdeas/BitterLesson.html

[1] https://news.ycombinator.com/item?id=19393432

Thanks! Macroexpanded:

The Bitter Lesson: General Computation Methods Are the Most Effective - https://news.ycombinator.com/item?id=37893468 - Oct 2023 (1 comment)

The Bitter Lesson (2019) - https://news.ycombinator.com/item?id=37139223 - Aug 2023 (1 comment)

The Bitter Lesson (2019) [pdf] - https://news.ycombinator.com/item?id=37129921 - Aug 2023 (60 comments)

The Bitter Lesson – Rich Sutton (2019) - https://news.ycombinator.com/item?id=36017857 - May 2023 (8 comments)

The Bitter Lesson (2019) - https://news.ycombinator.com/item?id=34006388 - Dec 2022 (33 comments)

The Bitter Lesson (2019) - https://news.ycombinator.com/item?id=30889873 - April 2022 (37 comments)

The Bitter Lesson (2019) - https://news.ycombinator.com/item?id=23781400 - July 2020 (85 comments)

The Bitter Lesson - https://news.ycombinator.com/item?id=19393432 - March 2019 (53 comments)

Reply on news.ycombinator.com