Earliest fossil for homo sapiens is 315000 years old, earliest neanderthal fossils are 430000 years. Keep in mind these are fossils and they need certain conditions to fossilize to begin with.

That's a huge range of time. And the evidence that we can find is very limited. This has nothing to do with what we expect to find when we have so thoroughly documented that humans were exceptionally capable of reusing pretty much anything.

So I don't buy the certainty that there was no agricultural society. We simply cannot know that anywhere past 100k years. We can say there were no globally spread agriculture, because that would have had a much higher chance to leave some evidence, but that isn't the same thing. An agricultural society doesn't have to be global or be dominant to exist.