On that note, I have a relative comprehensive intro to PyTorch in the Appendix (~40 pages) that go over automatic differentiation etc.

The alternative, if you want to build something truly from scratch, would be to implement everything in CUDA, but that would not be a very accessible book.