You can correctly calculate the relativistic mass of a photon in other ways, such as from its momentum or its gravitational interaction, and doing so gives you the energy you'd expect from E = mc².
Of course p = h/λ. Thus the relativistic mass m is given by p/v = p/c = h/cλ. And a photon's energy is given by E = hc/λ. This is all in agreement with the relativistic mass you'd calculate from the photon energy E = mc².
Heavens, which of these do you not believe?
https://en.wikipedia.org/wiki/Stress%E2%80%93energy_tensor
For that matter, you've already noticed that the Lorentz transform becomes nonsensical as v → c when starting from rest mass, whereas the transform works just fine when using relativistic mass as the starting point, and nothing blows up, and you can avoid any confusion.
Whether trendy or not, relativistic mass is however a very useful concept, because in many cases it is the relativistic mass, rather than the rest mass, that behaves the way we expect mass to behave; mainly that it is additive. For example, if you weighed a mirrored box full of bouncing photons, the scale would measure the mass of the contents as the sum of the relativistic masses, not the sum of the rest masses.
Rest mass has the awkward problem that it depends on which particles you consider to be "part of the system": A photon going left has no rest mass, a photon going right has no rest mass, but a system of two photons, one going left and the other right, does have a nonzero rest mass.
String theorists would even consider much of a particle's apparent rest mass to simply be another manifestation of relativistic mass, indicating the presence of an invisible periodic motion along a hidden dimension, resulting in kinetic energy even within a particle that seems to be "at rest".
(Edited for clarity)
And which kind of mass is a proxy for the total energy of a system? That's right: relativistic mass, not rest mass. And this is the sum of its parts, as it should be, because energy is always conserved. Relativistic mass and energy are the exact same thing, up to a constant factor of c². Isn't energy a useful concept?
Furthermore the history of physics has often revealed what we thought of as atomic particles to actually have systems underneath. In the course of time, it seems more likely to me that we will do away with rest mass than with energy, in which case all mass will be relativistic.
When considering systems there is only one mass. It's called mass. It's neither relativistic nor rest mass. It's just E/c2 where E is the total energy of the system. It's not the sum of the parts because it's dependent on interactions between the parts as well. The mass of deuterium nucleus is not the sum of relativistic masses of the proton and neutron. It's less because they are bound.
Rest mass is a sort of useful concept because it is the bit of energy that's still there even when object is at rest. Apparently it has something to do with Higgs field which I'm not super keen on because I don't understand it but nothing seems to contradict this so far.
But talking about relativistic mass doesn't make much sense. The velocities are involved in mass only because equation for the energy of a particle has momentum term. It doesn't create anything qualitatively different to be worth naming it. Relativistic mass is only talked about to avoid contradicting the intuition of mass being quantity of matter which was introduced earlier to weed out the intuition of mass being weight which kids get from everyday language. It's really not a great idea because it misleads people who try to think for themselves a bit further. For example what would happen if you tried to push fast moving object sideways. Would it be harder because of higher innertia because of higher relativistic mass? After all trying to push it in the direction it travels should be harder because of high relativistic mass. That's what people say when trying to explain why you can't accelerate matter to the speed of light. Suddenly relativistic mass becomes a directional property. When you use the correct equations in their full form istead of trying to shortcut to skip momentum and vectors all of this confusion disappears.