Seasons 3 and 4 had basically steady viewership through their whole runs, but each had way fewer viewers than the preceding season. So a lot of people who watched S2 until the end didn't even start S3, and many who finished S3 didn't bother with S4.
I wonder why this is? If viewership dropped through the season it'd be more obvious - people just didn't like the season. But to me this indicates that it lost mindshare between seasons. Was it interesting enough to watch as it was going, but not interesting enough to wonder what would happen next? I've definitely had that happen, where I binge watched a season of a show, but didn't bother (or dropped off quickly) when a new season started, just because I had sort of lost interest during the break and didn't get pulled back in the same way.