You can get CMTS equipment like Arris C3 and run your own DOCSIS headend if you want, but it's a lot of complexity to the point where it's not worth the effort unless it just happens to exactly fit your niche. Due to the way that cable plants work (upstream channels) the 'provider side' equipment must be complimentary to the 'customer side' equipment. That is there is no method for peer-to-peer traffic between cable modems in the DOCSIS standard even if the hardware is theoretically capable.
OTOH you can get a 2.5gbps MoCA adapter for like 50 bucks, and it can be installed, working, and reasonably secure in about 30 seconds. So I don't think DIY DOCSIS is a worthwhile pursuit outside the novelty.