Try to update your Python environment and also install ALL recommend dependencies for Pandas (and for Geopandas if you use it). I had one old env in Anaconda (Python 3.9.12, Pandas 1.4.2) and then created a new one in Mambaforge (Python 3.10.6 with Pandas 1.5.0). It gives me speed up one of experiment project from ≈30 min to ≈5 min. Use Python code only for glue and leave extensive calculations for C/C++ code. Pandas and Geopandas have the recommended C/C++ dependencies which dramatically speed up calculations and in new versions as I guess they improve integrations with these dependencies.

PS I also recommend to every one to use Mambaforge instead of Anaconda, because it uses Mamba dependency solver written in C++ and it is in orders of magnitude faster.