Not long before the Assange case ended, Swedish police kidnapped two asylum seekers (and yes, kidnapped is the right term - it was not an arrest, and has since been found to violate Swedish law) and handed them over to the CIA who handed them over to the regime they fled, which subsequently tortured them. The kidnapping and rendition was both a violation of Swedish law, and a violation of international treaties, and yet nothing happened to the people responsible to my knowledge.
You might see why someone who believed - rightfully, as it turned out years later - the US wanted to file charges and get their hands on him, might worry that the Swedish government would turn a blind eye to him being handed over to the US.
Whether or not he was right to fear that, it seems reasonable to believe his fear was genuine, not least because he chose to put himself in a situation that has kept him imprisoned in inhumane conditions far longer than he would've ever served if he was convicted of what he was accused of in Sweden.