I don't think anyone is saying they FOR SURE didn't cultivate a couple flowers and magic mushrooms.
If they were an agricultural society resembling anything like the ones we see later, there are several things you'd expect to find.
That's a huge range of time. And the evidence that we can find is very limited. This has nothing to do with what we expect to find when we have so thoroughly documented that humans were exceptionally capable of reusing pretty much anything.
So I don't buy the certainty that there was no agricultural society. We simply cannot know that anywhere past 100k years. We can say there were no globally spread agriculture, because that would have had a much higher chance to leave some evidence, but that isn't the same thing. An agricultural society doesn't have to be global or be dominant to exist.