The biggest one is that you're essentially trusting data that the client is providing (Apple gives user id to the client and the client sends it to the server). Unless you can verify the token and exchange it for your own session id, you're opening up your users to be easily impersonated (if they get a hold of the user id).

Other than that, Apple also provides server-side verification for the validity of the token. Without that, the client could send a random string and the server wouldn't know the difference.