Depending on your use case it's worth thinking about the expiration time. I assume that's checked in your client but do you also need to invalidate tokens or downgrade permissions before they expire? In that case you might want to work with smaller expiration times or get a denylist.
Assuming you also validated the expiry date.
Assuming you also verified that the issuer matched the expected one, and that you didn't fetch keys based on the issuer that is in the token.
Also assuming that the crypto alg is the one you expected.
I have seen the two authorization approaches get co-mingled, which leads to issues.
Also, often scopes are two large.. but that's mostly an implementation choice, like read all, rather than allow access to only a specific item - maybe scopes are just less easy to handle/maintain/define for the average developer.
Correct, for me this is a must avoid at all costs. I can't imagine how hard / complex it is to manage / audit two authorization system. Unless your application need more detailed permission / smaller scope, example later.
Personally I just use the one that comes from the identity provider. In my case, keycloak's model is sufficient for my use case.
And you're right, scopes are hard and unless it's global scope, you'll need to roll your own. One good example case is github/gitlab. Global scope is the administrator access, and it's easy to set it in identity provider (as "administrator" access maybe).
However for each group / repository level, you'll need to roll your own validation.
EDIT: More often than not, it comes into business domain than technical / programming one. If you're not experienced with the business domain, no wonder you'll find it hard.