Edit: If your algorithm is not using neural networks, then libraries like TF may or may not be a good fit, it depends on the algorithm. Writing custom low-level code can still make sense in those cases.
http://blog.rogerluo.me/2018/10/23/write-an-ad-in-one-day/
http://blog.rogerluo.me/2019/07/27/yassad/
Although the endpoint is likely to be a better understanding of the choices made by a mature implementation, and of the work involved in fixing up edge cases.
And yet they love to ask you to do exactly that at technical interviews... coming up next: what ML algos you need to know to ace that interview.
What I have to say is this: please don't build your own.
Some people do. It's a good challenge.[0]
But the parent of the comment I was replying to clearly had the former in mind, as a subsequent comment showed.[0]
I had a lesson in writing crypto once, when I made what I thought was a good enough secret mixing procedure to encode some data I wanted to email outside of a company that didn’t allow web access. (Long time ago, circa 2000). It all looked undecipherable and I sent most of the data before I discovered that strings of binary zero were leaking my secret key. Oops, pretty stupid.
D'oh! Good point though lol.
I am sure you could write stuff like Diffentiable Processors or the like from scratch with numpy but if you respect yourself and your time, you won’t. Complicated architectures are orders of magnitude harder than writing feed forward networks from scratch. For example, see the Merlin paper.