You're absolutely correct about using non-parametric tests, and more scientists should be using them. The normality assumption is flat out laughable when using real-world data most of the time.
You're also correct about multiple hypothesis testing. Accounting for familywise error (e.g., Holms adjustments) can help to keep your p-value reporting honest.
That doesn't negate the underlying problem, though. A p-value is simply an indication, nothing more. The p-value never promised to be more than that. The issue isn't in the p-value's construction, the issue lies in its misuse and how easily it can be abused in statistical reporting (see: p-hacking).
The p-value as a test statistic is perfectly honest in my opinion. But like many other statistical methods, it comes with its own set of baggage that I feel gets conveniently glossed over more often than it should.