It's interesting, because through my readings in linguistics, I got the sense that no language is "objectively" harder than any other, that everything is relative to the other languages you know.
Not really. A language's difficulty is often proportional to how much contact it has with outsiders, i.e., how many people are learning to speak it as adults instead of as children. Isolated groups can develop truly esoteric and labyrinthine languages. When a language is being learned a lot by adults the hard parts tend to get polished off.