Would love help simplifying this code (it looks really ugly right now)

I am considering redesigning the api using lxml instead of BeautifulSoup for performance reasons.