That's why I enjoyed the "hackers guide to neural networks" so much - it builds everything from the ground up