DualPipe: Bidirectional pipeline parallelism algorithm

When my company was still working closely with CN factories a few years ago (before the bans / clients no longer wanting to work with companies working with china etc), the CEO's of the factories we worked with all were electronic engineers at that company or another before; they all could jump in, debug schematics, sold and write firmware themselves. And they did. These were places with massive campuses with towering buildings with robots and a few (relative to the massive space) employees doing maintenance etc + prototyping.

"developed" and "codes" have different meanings.

Maybe add Chimera as well?

https://arxiv.org/pdf/2107.06925

Off topic, but this is the Rick and Morty episode where Rick creates a perfectly level space.

The symmetry is uuugh.

It makes it so that having more GPUs makes inference run faster. Worst case has been you can only use memory from them and gain no speed at all

In very simple words: it is one way to reduce the white squares in the picture from @danielhanchen[1].

In more complex words: imagine a processor which is able to process every instruction in 10 clock cycles. But also the processor is able to get new input for this instruction on every clock cycle and starts to process this new input in a pipeline. After the first input you have to wait ten clock cycles. But if you feed the input line every time you will get the output also permanently.

In the case of GPUs, it is now not only a topic of a single pipeline, but multiple in parallel. Depends on your data and algorithm it can be thousands in parallel.

[1] https://x.com/danielhanchen/status/1894937006352031832

Middle out or something?