Do you block Google trackers aggressively? reCAPTCHA uses that very heavily: if you allow all of their stuff and they track you across the web, you'll have to basically never do more than click the button. On the other hand, if you take your privacy seriously and are aggressive about tracker blocking, you'll have a pretty awful time.

I imagine hCaptcha doesn't have enough trackers sprinkled around the web to use those as signals for this.

This is under the control of the site with hCaptcha, so you'll tend to see more variety in difficulty levels depending on their settings.

There will always be some individual variance, but when we've tested this people always solve hCaptcha faster than reCAPTCHA on average.

(disclosure: work there)

Same. I heavily block google scripts and hate reCAPTCHA with a passion, but hCaptcha really takes the cake for the most painful captcha experience.