in my experience, it is more useful to view neural nets as geometric transformations - via stateful functions - that map stuff in input space (eg a sentence written in english) to stuff in some other space (eg the same sentence written in french).

by viewing neural nets (and machine learning, in general) from a mathematical perspective, you can readily exploit an entire field of tools and techniques (eg numerical optimization) and clearly define objective functions to train against - benefits that you dont necessarily get by viewing ml from a biological perspective.