You’re both correct—optics are constrained by sensor size, so while the amount of background defocus is determined directly by optics, it is determined indirectly by sensor size.
The amount of blur at infinity is proportional to the ratio of the aperture size (e.g. measured in mm) to the subject size (e.g. measured in mm). For a given lens design, the aperture size is proportional to the sensor size.
You do get higher aperture numbers for smaller sensors but we’re talking about a factor of, say, 2x weighed against much larger numbers. This is just because wide-open lenses are impractical to manufacture at larger sizes (weight goes up with the cube, after all).
I see phone cameras with lenses at like ƒ/2.2 or something thereabouts and it’s not radically different from what you’d see on larger cameras until you get to LF, but if you dive into vintage LF lenses you’ll see “barrel lenses” with crazy, like, ƒ/2 designs for shooting on LF film. These are not scaled-up versions of sensible, modern designs. Instead, they are really old designs, and they are not gonna be sharp.