I have this setup pretty well in my code now but getting there wasn't simple or easy from my perspective and keeping the list of secrets your IAM user can access up to date can be a pain as well.
I'm working in a lambda environment so my options might be more limited but I'm interested to see how other people are solving this issue (maybe specifically for small/side projects). As it stands my lambdas all get a role applied to them that gives them access to the secrets but something not AWS-specific would need a "bootstrap secret" to be injected before the code could call out to the third-party to get the other secrets. For Lambdas I suppose I could inject that "bootstrap secret" in via environmental variables but now I've got a new issue to deal with. Injecting at build time via something like GitHub Actions Secrets is an option I guess.
All that to say, while I agree secrets should never be in source, in practice it's not super easy (I'd love to be proven wrong, maybe I'm not doing it right).