It concludes "There’s further optimization left to be made by locking individual HashSets instead of the entire follows or following set, but we can leave that for a later day."

The other optimization that I would try is replacing those u64 keys with u32. There are only a few million accounts at present so there's a lot of growth room before 32 bits become inadequate. RAM is cheap, but making values smaller allows you to fit more of them into CPU cache, which often gives worthwhile speedups.