I agree with your expectation of clicking the 'X' going back to the results.

But it's a bit strange, why have the 'X' there at all when the browser back button will do exactly the same thing more naturally?