The longer a disease takes to show symptoms means it's had a longer time to multiply inside your body.
Once your immune system detects it, you're going to get the standard flu/cold symptoms (fever, runny nose, etc).
During this period your body is actively losing its protection due to putting so much resources into fighting the viral infection in your body.
Other deadlier diseases can now enter your body and essentially hide behind the fact that your immune system can't respond to everything at once.
So you get more complications as you're sick for longer.
The viral infection itself does have an impact on organs, specifically the lungs, but it's usually secondary complications that kill.