I remember dating a guy who bragged about how some of the songs on his favorite album was 5 minutes long. I thought that was such a strange detail to care about.
Personally I think it comes from not being exposed to a lot of music growing up (my mom is hard of hearing), and just assuming that almost all songs were 2-3 minutes. I'd say that the thought process behind it also has something to do with longer songs requiring more time or dedication to learn all the lyrics, so it feels more special when you know and really enjoy a longer song.