Interesting how you haven't used any machine learning framework :) Did you do it purposely to get a better understanding of neural nets? Are you going to move to frameworks to speed up your iteration speed? Or are you more interested in maybe refactoring your code into a reusable framework? :)