First, there is/was no single algorithm, but the core ideas driving a lot of recommendations is:
1. Create user taste vectors
2. Match those vectors to other users or collections of tracks
3. Use that information and combinations of other things to find recommendations.
Each step of the process is constantly being experimented with. Different custom playlists might be using a different combination of tech doing those basic steps.