That's a good question. It comes down to how much you want to control the devops vs how much you want someone else to handle it for you.
App Engine is a platform-as-a-service (PaaS) which generally means a lot of things are done for you. You give App Engine the code, and App Engine builds it, deploys it, scales it, monitors it, applies OS patches, and generally provides a bunch of related services for you.
Compute Engine is more infrastructure-as-a-service (IaaS) which means the service gives you VMs, and you handle installation, build, deployment, and setting up load balancing, scaling, monitoring, etc. the way you want.
There are also hybrid products that fall somewhere in the middle. Kubernetes Engine is an example. Indeed, you can locate most hosting platforms somewhere on this spectrum.
Note also that there are trade-offs specific to Elixir as well. PaaSes generally have to be opinionated about application architecture. Most PaaSes assume a stateless web application well suited for container-based deployment, which runs somewhat counter to Erlang's stateful services and hot code swapping. If you're writing a typical web app (say, using Phoenix) then a PaaS like App Engine will often work very well. But if you are doing something different and managing persistent nodes and processes, you may need to control deployment yourself, which makes a lower level service like Compute Engine more useful.