Grinding the olivine to this study's grain size of 83 microns[0] will consume roughly ~14 kWh/tonne[1]. One tonne of olivine absorbs about 1.25 tonne of CO2, so that's 11 kWh/tonne for grinding, which is indeed likely to be the majority energy consumer.
Clearly the solution is... build roads from olivine! :D Collectively, roadways are "naturally" subject to a huge amount of mechanical weathering, which currently mostly just generates asphalt microplastics.
No doubt the cost/geotechnics doesn't work out, but it's an amusing thought nonetheless. Maybe it could even be made workable, somehow...
[0] https://iopscience.iop.org/article/10.1088/2515-7620/acfd89/...