"I think I just configured sendmail to only use ipv4 and that solved the gmail issue."
This was my experience, too. I took this advice [1] configuring postfix and Gmail started to accept my emails.
[1] https://christian.skala.me/blog/gmail-why-are-you-doing-this...