Depends completely on the context. In an ideal world the pianist leads the tempo.
However, sometimes the composition makes this near impossible (i.e. there is a large section of the orchestra that has to start playing in the new tempo, such that a conductor couldn't both react to the tempo of the soloist and show it clearly to the orchestra). If it's impossible for the conductor to get the tempo from the soloist, then usually there is time before the first rehearsal to discussing those spots, and discuss ideal tempos for the soloist. As you can guess, there is usually the possibility of negotiation, either verbal during this meeting, or during performance rehearsal both finding a more ideal tempo for the hall, orchestra, and days sensibilities.
Schumann piano concerto is a decent example. Second movement tempo is totally decided by the pianist, no discussion is necessary prior to rehearsal (unless they want some particularly abnormal). The second theme of the second movement is with solo piano in accompaniment position. So while we want to accommodate their musical needs, ultimately the musical burden is on the orchestra (in this case the cellos), as a result the tempo will be mostly decided (contextually of course, it has to fit with the first theme) by the conductor. Third movement beginning in theory is lead into by the conductor, however after the downbeat the soloist can show their tempo and the conductor responds.