He could simply have chosen a different macguffin, but Tolkien had a weird way of fixating on things that he'd already written. It's almost scientific: you explore the space of stories, and whatever hangs together consistently must be "true".
He solved that not by deprecating the original book, but by rewriting it, and then including a preface in The Lord of the Rings about how the original book was Bilbo's self-promoting propaganda.
This is emblematic of his work. He kept re-writing and re-writing to find something that felt "true". Which is partly why he never really settled on anything. He kept assigning characters different names, and then assigning some names to other characters until they fit. In the early Hobbit, Thorin was called "Gandalf", which makes it a hell of a thing to read. (And the wizard was named "Bladorthin". Which is almost as bad as "Bingo Bolger-Baggins", the original name of Frodo. Or "Trotter", the original Strider, a Hobbit with wooden feet...)
Given that The Hobbit was not actually intended to be connected directly to his Middle-earth works except retroactively, and that The Silmarillion was not published until after his death (with significant editing, because he kept re-writing it until it was no longer compatible with The Lord of the Rings), in a sense there is only one truly "canonical" Middle-earth book: The Lord of the Rings.
Tolkien was not only an extraordinary genius, but we have an incredible collection of his draft works to see how his genius worked. I can't think of any other author whose process is simultaneously so remarkable and so well-documented.