Assertions about the difficulty of implementing effective parallelism reflects assumptions about software architecture that are common but not universal. The tools most people reach for tend to be either multithreaded locking of some type or functional "everything is immutable" style, neither of which scale well on real hardware. Complex applications that do use large numbers of cores efficiently use software architecture originally developed on supercomputers but is equally suited to ordinary machines with many cores: fine-grained latency hiding. In practice, this means extremely cheap logical concurrency, like proper coroutines, combined with a latency-aware scheduler (the harder part). This may not be "parallelism" in some strict sense but it has the same effect in terms of utilization and is sometimes used on supercomputers to maximize throughput.

Programming languages often have some level of support for necessary primitives, such as goroutines, but then hide them behind an unsuitable scheduler. On the machines where latency-hiding was originally developed as an architecture (notably Tera's MTA), optimal scheduling was done by the hardware itself which had perfect visibility into relative latency with no overhead. Things are not that simple on x86/ARM. If you want to implement effective latency-hiding, you'll be implementing an equivalent scheduler in software, and often by inference because sometimes the only way to know the true latency of something is by doing it. I don't think it is that difficult but schedule design is not something most programmers know how to do well -- it is magic buried in operating systems and libraries. Nonetheless, you see more examples of this kind of software design for performance reasons.

There are very few real-world applications that are so serial that latency-hiding is not effective at using all available cores, particularly in the server realm where the workload is inherently concurrent. The original target application for latency-hiding architectures was scaling graph analytics, which are often not amenable to more classical embarrassing parallelism.