The high speed rail authority is not only tasked with building and operating the final high speed rail route but also has had to deal with land use politics of dozens of jurisdictions along the alignment. HSR has no at-grade crossings which means everywhere along the route that it crosses a road there needs to be an overpass or underpass. In the central valley where most construction has been happening that's at least one every mile for about 150 miles. They also had had to do multi-year environmental impact studies (because California law requires it for any major change from the status quo) which have to be incredibly thorough because people can just appeal any part of the project for any arbitrary "environmental" reasons such as "the overhead wires alter my view of [something, usually nothing]" if they can claim that it wasn't considered in the study.
Can we build it? Yes. The issue is mostly that we have so much process that delays building and scope creep for what does get built.