Silk's just easier in general. Unlike vegetable fibers, silk can be reeled to yarn/thread directly. With cotton (or wool), there are about 4-6 other processes before you can even get to spinning. Hemp's worse still, I think there's a retting process in there like with linen.
Or perhaps, it's more true that the labor-intensive portions are up front... you're feeding the dumb little worms up to 4 times a day early on, transferring them off of the eaten leaves (so as to avoid disease), and this only lightens up to twice a day later in the lifecycle. And this isn't tapping out a little pinch of food like for a goldfish bowl, they'll often do 10,000 worms at a time just to have some modest quantity of fiber (perhaps enough for 5 or 6 yards of fabric). This ends up being like 60+ lbs of mulberry leaves there at the end.
All told though, I'd much rather raise the worms than sit there for six months trying to card cotton by hand.