It seems like your complaint is that 2 dimensional data is more difficult to work with than 1 dimensional data, not a problem with the format itself.
I think the answer is that it's arranged that way because that arrangement makes it easier for humans to interpret. Because humans prefer to see data in such 2D arrangements, file formats like PDF were developed so that 2D layouts could be defined and preserved across platforms.
Hence the fundamental problem here is that we want computers to be able to read data that is designed for human readability. That is not an easy problem and it is in fact a large part of what people mean when they talk about "AI".
Though as a counterpoint, separating content from presentation is already a fairly common practice.
A lot of the text we encounter in the world does depend to a significant extent on 2D layout to facilitate its comprehension. Examples include many (perhaps most) websites, product labels, tax forms and receipts.
None of that is the fault of the PDF format which is intended to faithfully represent 2D layouts. As you mention, if the goal is accurate labeling of content other formats exist which are more suitable to that purpose (though I think the concept of separating content from presentation has always worked better in theory than in practice).