Perhaps the sanest approach is we type every column of X and Y separately...
Typing every column of a matrix is not sufficiently general. Each element of the matrix can have its own type.
Suppose you're solving a linear algebra problem in two unknowns. Say it's an electromechanical system, and we want to find a voltage and a pressure that gives us some desired flow rates. So let x [=] (V, Pa), for example, and let y [=] (mA, kg/s).
Then the matrix-vector product Ax = y requires the units:
[ mA/V , mA/Pa ] * ( V ) = ( mA )
[ kg/V/s , kg/Pa/s ] ( Pa ) ( kg/s )
Of course those units in A have nicer reductions, but it doesn't teach us anything to figure them out. The point is that the matrix A represents the system design, so the units don't have to have familiar names, let alone be consistent within a column. If the system involves Johnny reading a value off a meter, and then phoning somebody up to tell them to pedal a stationary bike a little faster, that process has some kind of units associated with it, and the matrix A must include those units in order to represent the system.