It could be that what they wanted to do just couldn't be done in 1 frame. Or often it could, the games were just poorly optimised.
There were plenty of games where the code didn't care to pick up the joystick movements in a reasonable time, making the game feel sluggish, even if the joystick movements can be read directly from the CIA port.
Compare games with perfect framerates like Turrican 2, Superfrog or Leander, with games that don't, like Magic Pockets, Xenon 2, Driller. In fact, compare the official version of Stunt Car Racer with the cracked version, where the crackers massively improved the drawing speed and thus the framerate!
Compare the scrolling of https://youtu.be/V1R7Pd329Bc?t=1969 or https://youtu.be/wFtLGDywZlg?t=553 with https://youtu.be/_4U4ZW37haU?t=114 or https://youtu.be/BXR2yswfC6E?t=862 -- it's purely the programmer's fault.