It's of course totally possible to learn a language without learning to write, since that's historically how people have always done it.
But note that modern societies aren't very accommodating to illiterate people. Even language-learning resources like dictionaries tend to assume that the user is able to read and write.
On the other hand, if you just want to visit as a tourist, you could do that without any knowledge of the language, just relying on English signage in touristy areas and Google Translate for simple conversations. It might detract a bit from the experience if you visit an old temple and can't read the inscriptions, but then again most Chinese people can't read the Manchu inscriptions on Qing dynasty buildings and still enjoy visiting them.
But if you want to stay longer or want to travel off the beaten path, not being able to read and write can be a real hassle. Fortunately, 4000 different characters isn't actually that much. You probably know tens of thousands of English words you learned over the course of a few decades. If you learn five to ten new characters per day, you can reach 4000 in a year or two. The hardest part of language learning is actually doing that little bit of daily practice.
When I started learning Chinese, I used https://apps.ankiweb.net/ with the Chinese support addon and added five flashcards every day (not always new characters, most words have characters in common) then worked through the scheduled number of reviews, using a notebook for writing practice. After two years, I moved to China as an exchange student and was able to get around fine, even though I still had a lot to learn. Sometime later my notebook was completely filled with writing practice, at which point I decided it wasn't really necessary anymore.
If that much grinding turns you off, try applying it to something you enjoy. E.g. pick a pop song you like but don't yet understand, get the lyrics and work through them word by word. Or watch a drama with target-language subtitles and keep your ears and eyes primed to notice common words. (Although those might not always be the most useful, e.g. the protagonists' names are probably not something you need to remember.)
But I don't know what has made you struggle with language learning in the past, so maybe that advice is not at all useful to you.