Yes, parameters are usually stored as float32, activations as bfloat16 or float16.