It will give a short term boost, but eventually you will have to look beyond your data and explore new things your customers really want.
The parable of the drunk found looking for his keys under the light instead of inside the dark bar where he lost them comes to mind. What's needed is a flashlight, however imperfect.
It's a bit like moving the furniture around the same room.
Someone has to have multiple good ideas and the ability to carry them out properly before an A/B test is even valuable. Otherwise you're measuring one uninformed random change against another. A/B confirms whether you've succeeded in improving something. It can't really suggest what to try next.