Fair enough. I should have clarified that “approximately the cost of a single kernel launch” is pretty much what I meant by “tiny”.
What I was getting at was that a “megakernel” and a captured graph should have similar launch costs.
What I was getting at was that a “megakernel” and a captured graph should have similar launch costs.
I'm not sure it applies so well in LLMs though (should read the paper...).