train 2 matrices, add their product to the pretrained weights, and voila! Someone correct me if i m wrong
If we can perfect methods to fine-tune large models for specific task while reducing the overall model size, then it can fit into more consumer grade hardware for inference and can be broadly used. The objective is to prune unnecessary trivia and memorization artifacts from the model and leverage LLMs purely for interpreting natural language inputs.
You don't need additional layers. After training, the product of the two matrices is added to the original weights matrix, so the model size remains the same as the original during inference.
Some annotations:
- The labels in the orange boxes mean "A is initialized with random weights (in a gaussian distribution, B is initialized with weights set to zero".
- d is the number of values of the layer's input and output. (The width of the input and output vectors, if you will.)
- r is the number of "intermediary values" between A and B. It's expected to be a lot smaller than d, hence "Low Rank" (apparently LoRa even works with r = 3 or so), but it can be equal to d, though you lose some of the perf benefits.