I think the scrum team size is determined in part because communication overhead is exponential, getting beyond 6-8 people means the whole team has to spend a lot of extra time coordinating. This is true regardless of whether you're dong scrum or not. It's true some teams may be correctly sized at 20, but I would suspect that breaking that into 2-3 other teams would be a benefit most times.

Story Points represent effort and uncertainty, not business/customer value - those are Business Value Points. Devs are only committing to X story points per sprint. If refactoring is needed, it's built into the story points, or added as it's own task. And as others have said if the Product Owners don't take engineering input on paying down tech debt/infrastructure/internal tooling, then you have a broken company no matter what process you're following.