If you want to use it's internal iDP - you can. If you want to authenticate against GSuite or another SAML provider, you can do that, too.
And behind the scenes, AWS SSO sets up the exact same SAML infrastructure that is available to you already in IAM, just with roles with unpredictable names (so that it's difficult to programmatically attach policies) with "DONOTDELETE" as part of the name but no actual SCP in place to prevent the role from being deleted. Because it's the same exact SAML infrastructure, but with additional redirects to allow you to login through the AWS SSO start page instead, it's slower compared to setting up SAML access per AWS account directly.
AWS SSO is a horrible product that actually encourages poor security practice (i.e. AWS managed policies, because a single inline policy is not large enough) and really the only reason why anybody bothers using it is because SAML login from the AWS CLI tooling is not well-supported by AWS.
We've since adopted AWS SSO and Control Tower, and to address the problems with it that you mention, we configure the SSO permission set so that the only thing the user can do is assume their identity hub role. Because you can chain `AssumeRole`, it's an extra indirection, but everything works just like it did before.
So while I agree that AWS SSO is not the right place to configure fine-grained permissions, I think that overall I prefer the benefits of Control Tower and the relatively easy SAML integration provided by AWS SSO over the previous bespoke SAML configuration we used. It's a bit inconvenient to set up the extra hoop to jump through to go from SSO permission set to identity hub IAM Role, but that's a one-time cost and is trivial to maintain.