If you are just looking to run a custom OAuth/OIDC server, Ory Hydra is ridiculously easy to set up.

Keycloak comes with the full IAM stack, so if you need that good; if you just need the OAuth server, its a bit much in my opinion. Have not tried Dex so can't speak to that.

https://github.com/ory/hydra https://github.com/keycloak/keycloak