Some back of envelope math tells me that a solar panel the size of my phone would roughly take 4-5 days to fully charge the phone.
If you are interested, you can buy solar charger phone cases - some even with fold out panels. But even then, it's hard to imagine any real utility from them unless you are actively backbacking.
Two, you never get optimal conditions, especially if you're out hiking. Because of the way solar cells inside a module are connected, partial shade (your finger, a tree, a passing cloud) will plummet the output. If you shade just 20% of the solar module, you don't get 20% less power, you get something like 80% less... the tiniest bit of shade has huge effects on power output.
Three, it's terribly inconvenient. If you want to optimize charging, you can't be actively using your phone (because your fingers would shade the module), can't strap it to your backpack (the straps would shade it), can't hike through wooded areas (the trees would shade it). All of this would happen during the day, when you're active and wanting to use the phone. And if it gets too sunny and hot, you'll overheat your phone and either cause it to thermally shut down or else just damage it.
Generally it's just much more convenient to get an external solar panel, maybe one with a built-in battery pack. For backpacking, they can both be in the same portable unit. For car camping, you can have a huge (foldable) solar panel plus a big external battery. Here are a few of both kinds: https://www.outdoorgearlab.com/topics/camping-and-hiking/bes...
That way, you can just leave the solar charger at camp, in the sunniest spot (or dangle it from your backpack), where it'll slowly charge its own battery pack. At night (or whenever), you can plug in your phone to the battery and quickly charge your phone from the battery (which acts as a buffer/cache between your device and the sun) -- all without really changing your phone usage habits.
btw, thanks your question hints me another solution, if solar pack can be more slim and foldable, it will be great to be a "solar-phone-case".
Like...
Why can't we use the same chair we sit in to type at a desk when we drive in a car?