For starters, they should actually follow the DMCA. The DMCA gives affected people a defined way to counter-notice, and then the entity who filed the initial DMCA notice can either sue within 14 days or the content gets restored.

Instead Google chose to evaluate "disputes" themselves, with algorithms and "AI", and reject disputes. Rejecting counter-notices is absolutely NOT something the DMCA mandates or even suggests. Moreover Youtube essentially used to allow (probably still does) alleged rights owners to reject disputes in essentially one click, while the DMCA would require them to bring a law suit. Some copyright owners therefore created bots doing the clicking for them.

They could also be more lenient to "established" players as a first step, especially when it comes to counter-notices/disputes. Factor in previous history google has with an alleged infringer (alleged by their own algorithm by the way, not even by a third party) when considering a dispute, like account age, channel age, number of previous videos without problems, "we do know the customer" e.g. to pay out ad money, etc. And then maybe not outright reject it, but leave it to the alleged copyright owner to file a law suit (as the DMCA states) or at least refer it to actual human beings for further evaluation.

Of course, Google could hire people to check up on their own algorithms and decide on disputes instead of machines. Youtube had $6 BILLION in ad revenue in the last quarter (not year), so they could certainly afford to hire some people. In the end it might even be a profitable investment, as fewer good content is pruned from Youtube for wrong copyright issues, leading to more ad revenue.

Youtube right now seems pretty content with their quasi-monopoly, to their own detriment in my opinion. As unlikely as it may seem that people will create competitors, it can happen, ask mighty MySpace about it.

Add friction to the process of submitting copyright claims. Reduce friction in the process of appealing copyright claims. Relax draconian rules like the copyright strike system leading to account closure. Adjust the content ID system to reduce false positives (assuming no magical improvement to the system this will come at the cost of increased false negatives; that seems like an entirely reasonable trade-off)

They just stop and wait for the DMCA take downs and copyright owners have to make their own claims? I guess.