Focus on measuring search quality and methodology first. Be a scientist. Great search teams obsess about methodology. Treat everything you try as a hypothesis, not guaranteed to work. Create a feedback loop that improves the pace of experimentation.

Other than that, the solution space is just as wide open as regular programming. It's just in many ways more frustrating because nobody knows what they really want from search, they just "know it when they see it" and no two users really can agree on what a good result is! :)

In addition to @softwaredoug's comment is his book "Relevant Search", it's a great starting point! https://www.manning.com/books/relevant-search

I found "Search User Interfaces" by Marti Hearst very informative. It's available online for free: https://searchuserinterfaces.com/

Perhaps https://yacy.net could help you.