I'm not sure using pytorch counts as "from scratch" anymore. I'm not saying you should avoid the stdlib or anything crazy, but at the point where you're pulling in for-purpose libraries it really doesn't seem like "from scratch" to me.