It's probably a git/libgit issue. I reported a similar issue(Cloning/Pulling commits from github took forever, Transfer speeds going as low as 1 kbps when my connection was 50mbps).

They asked me to run some commands and log output to measure packet loss/speed etc. and then told me that my connection had 2-5% packet loss which was causing all the problem and I need to talk to my ISP or get a better connection.

I switched my ISP after that and its not a problem anymore.