I was convinced that somehow the line from our house to the pole in the alley was bad and when the sun was warming it up it caused the issues. The cable company refused to fix it since they never saw a problem on their end by the time I would reach customer support.
My solution was to upgrade to business cable Internet, which was about the same price we were already paying, just required signing a 1 year contract.
First day the new service was active, called business support and 2 days later they came out and replaced the line. No issues after that.