This is related to the conservation of Etendue [1] in an optical system, which is basically a statement of conservation of power: you rightly point out that radiant flux is determined by the source and constant – and for that reason, the primary numerical aperture or f-number of the lens is ultimately what really matters – assuming, as you point out, that you want to use the narrower DoF that arises (in which case SNR scales as the square root of sensor area).
However, sensors get noise from different sources: and while you're right to point out that you might be up against photon shot noise, read noise goes down with pixel area: so, as long as pixel area scales with sensor area, and that scaling is performed by uniformly scaling the pixel, the larger sensor is intrinsically "a little bit better". Quoting shamelessly again from wikipedia [2]
> The read noise is the total of all the electronic noises in the conversion chain for the pixels in the sensor array. To compare it with photon noise, it must be referred back to its equivalent in photoelectrons, which requires the division of the noise measured in volts by the conversion gain of the pixel. This is given, for an active pixel sensor, by the voltage at the input (gate) of the read transistor divided by the charge which generates that voltage, CG = V_{rt}/Q_{rt}. This is the inverse of the capacitance of the read transistor gate (and the attached floating diffusion) since capacitance C = Q/V. Thus CG = 1/C_{rt}.
As capacitance is proportional to area, pixel area matters here – read noise is proportional to it linearly. In low-light conditions, read noise dominated most cellphone sensors (mostly for the above).
[1] https://en.wikipedia.org/wiki/Etendue
[2] https://en.wikipedia.org/wiki/Image_sensor_format#Read_noise