Have you measured performance? If mutating from Elixir like this can bring serious benefits, maybe there's a place for mutable versions of libraries like Explorer and Nx.
In my case the data I'm dealing with is more of a store than a single data item, so I'm leaning on the example of things like ETS. Also it's within a single application rather than being a large generally-available library, so the trade-offs are different. It would be interesting to know if they did tests though.