Took me a while why Office is still using Intel code. Turns out they have "Open with Rosetta" enabled. I disabled Rosetta on those apps and it went to use Apple Silicon code. Now it is snappier than using Rosetta.

Thanks for letting me know!