It was the same for me. I was in Manhattan two years ago, and I was amazed with how much worse the problem is in SF.
This was a sharp contrast to the mid 1990s, when I lived in New York. I found aggressive panhandling to be more prevalent in NY than SF then, with panhandlers following me for blocks, approaching me at the counter of coffee shops (once crowding me as I attempted to get my change from the cashier, quickly picking up it up when I dropped it, and then asking me for it while holding it away from me).
I'm always a little skeptical about anecdotes, though, as the "homeless" population, ranging from non-addicted or mentally ill people who are temporarily without housing to severely addicted and mentally ill people engaging in criminal level harassment of people, often clusters in a few neighborhoods in a city, sometimes at the outskirts.
I met a man from Paris who refused to agree that San Francisco has a worse problem than Paris. He said that he believes that in Paris, the severe poverty is pushed out to the edges of the city. I don't know if this is true, or if he didn't want to criticize too much, but it is an interesting observation, at any rate.