shreyansh26

What’s covered:

1. Hierarchical scans: block-local scan → write block totals → scan totals → carry-in add 2. Single-pass scans: the "domino" idea, and why naive inter-block propagation can stall / deadlock without the right coordination 3. Decoupled lookbacks: how modern single-pass scans coordinate across blocks safely 4. Warp-window lookback optimization: scanning lookback metadata in warp-sized chunks (and why it helps)

I also include H100 timings and compare against CUB for context.

A short deep-dive on Multi-Head Latent Attention (MLA) (from DeepSeek): intuition + math, then a walk from MHA → GQA → MQA → MLA, with PyTorch code and the fusion/absorption optimizations for KV-cache efficiency.

A simplified implementation of FlashAttention in PyTorch for educational purpose.

Haha, so all one has to do is to add some fancy terms like LLMs, vectors, hyper-fast and 300+ Github stars. What a a joke lol!

OpenAI is actually the one which has made AI so open and accessible for all. Perhaps not open source but all the developers building cool AI products these days are mostly due to OpenAI.

That's helpful. Thanks!

Reproduced on Edge on Windows 11 as well. What a bug! Really need to know the root cause of this.

Someone post something about astronomy also please!