I think you could do that already. Just store your weights in a table...
1. Write a query planner that compiles SQL into CUDA kernels
2. Write a VRAM-resident backend for Postgres
3. ???
4. Profit!
> Matrix multiplication is after all nothing but an aggregate sum over a cross join.
They converted linear algebra expressions into relational algebra expressions, optimized them using equality saturation and converted back into linear algebra. With some noticeable speedups.
They used relational algebra for optimization because relational algebra has canonical form - there is only one most optimal expression.
And, equality saturation is most efficiently implemented using generic join algorithm.