SolarPi experiment 1: The PiJuice Fiasco

Suspect that particular solar panel is bad, until you've tried a couple of others at least.

But going to a bigger system is probably better anyway. They make "solar electric fence chargers", look to those for examples of how to do it.

Lead/Acid batteries are great for this kind of application because they're robust and cheap, you can often treat the battery and charger circuit as one problem and then just tap the battery as if it was an automotive application (needing beefy filtering anyhow).