I am not sure if I get your point. Yes, having a small base vocabulary has trade offs and yes it is true that you will have to not just learn the meaning of every single word but also common idioms.
You can be up and running very fast in Toki Pona being able to produce and understand simple sentences about broad topics. If you need to be very specific, you will have a hard time but the philosophy of Toki Pona is to strive for simplicity and to focus on your core message.
You would have a hard time writing a scientific paper in Toki Pona but that is not what is was designed for anyway.
What language people have trouble learning can be a bit subjective and also heavily depend on the learning material and other outside factors. Sounds like Toki Pona didn't "click" with you which is fair enough. Maybe you want to give it another try or maybe it just is not for you, there are other interesting conlangs.
Toki Pona is relatively easy to learn not just for the small base vocabulary but also the relative simple phonology (Only 9 consonants and 5 vowel, always stress on initial syllable). Learning how to pronounce and write Toki Pona is extremely easy and regular.
Vocabulary is probably what takes the longest time in language learning, so you get a huge head start. The grammar might be very different to English or Dutch but has many similarities to Asian languages and is not that complex. As I argued for learning Toki Pona for mind expansion, I would argue that the difference is a good thing anyway.