Have you seen nx[1]? It looks like the first step towards this!

1. https://dashbit.co/blog/nx-numerical-elixir-is-now-publicly-...