that's a lot of weight to add for backpacking. every bit counts when you're going 20 miles up and down hills.
The thing is though, I've only found it really useful if I'm doing stiff ascents and am not comfortable with my training. Mostly just to keep an eye on my heartrate, if my training/preparations are adequate for my trip, I don't use it at all.
If you are really UL, you can ditch the phone and just use the Ultra. Apple Watch + small battery is lighter than a Fenix + iPhone + battery.