If the criterion is that it's to some extent imaginary, we already have a word for that: Fiction.
(The lines get blurrier when talking about imagined historical fiction, or even things like alternative fiction.)
Even things like Tau Zero are using relativistic time dilation as the plot driver.
I agree, and sometimes the line is drawn between SF being "things that are theoretically possible" vs. Fantasy where things are impossible. But then you have things like Egan's Clockwork Trilogy, which is "what if the laws of physics actually worked a bit differently in this specific way" but which I assume anyone would consider SF. As opposed to Brandon Sanderson's books, which could be described in a similar way, but are usually categorized Fantasy.
At the end, it's mostly a marketing and feeling thing. As one of my favorite authors put it, the different between SF and Fantasy sometimes comes down to - are you putting a tree or a spaceship on the cover of your book?
What would you say is the reason for categorising works differently? Can you see differences there or do you also think it's mostly marketing?
An author once wrote an intro to some short story. The story is part of a much larger futuristic scifi universe in which people have developed telepathy and other things through genetic means. And the specific short story was the first one he wanted to publish, and it was about a specific planet in that world, in which the whole story is basically a telepath coming into town and interacting with the population.
And the publisher returned the note that this wasn't scifi, it was fantasy. Because of course he did - stripped of the broader futuristic setting that the story takes place in, it's just a story of a wizard coming into town. Never mind that there are solid science fiction explanations for the "magic" - you don't get that in the short story.
While Tau Zero that was mentioned elsewhere is believed to not match the laws of nature now, the science the entire plot rests on was considered scientifically plausible at the time it was written.
It was speculative, but it explicitly did not set out to make up a world in which some scientific law is different.
In other words, that isn't a defining factor of SF.
The speculative nature of it is closer to it - hence the shared label of speculative fiction often used to group SF and fantasy.
Even the "hard" sci-fi tends to comprise of the author's one area of expertise or hyperfixation while everything else is nonsense. You'll have descriptions in intricate detail of how the spacecraft are engineered down to the self-sealing stembolts, but biology is basically magic.
A common sf theme is "here is this change to the laws of physics, what would our universe then look like". Eg Arrival (and the story it's based on), tons of books by Egan, any book with FTL.