In RAM, yes. But if you compute an activation, you need to load the weights from RAM to the GPU core.