In fact, for most of history, and with very great many minds throughout history agreeing to that, it would be your objection that would be considered bizarre.
You don't have to believe in a specific creator or any specific religious mythology to agree with such a statement as "this world is fallen". Even as atheist like e.g. Emil Cioran could agree with such a statement. But even if someone actually believes e.g. in the Christian god, it's almost irrelevant as to the validity of their belief that "this world is fallen".
It's just a way of seeing the world -- as permanently imperfect and with fatal flaws in its construction. In fact most of our history and even personal lives agrees with this observation as well: we are ignorant of most things, we age, we get ill, we suffer, we kill each other, we are petty, and so on. And in the end, we just die, along with everybody we knew and loved. All that in an indifferent universe, that could e.g. wipe thousands of us out tomorrow with an earthquake, a tsunami, and so on. Or something of our own making, like a nuclear war.
These things can be summed up in "this world is fallen", and there's nothing bizarre or unconnected about it.
One might disagree and see progress etc, and even be optimistic that everything will prevail, or even see no issue with death and illness and the ills of old age, and so on. But in the main there's nothing much bizarre in saying that "this world is fallen" -- e.g. that things are incomplete, broken, and more often than not disappointing.