Are humans even capable of building perfectly reliable systems?
That was not the only thing about the apparatus, just one of the most important, but the point is: we cared, we cared a lot, and we cared a lot about a lot of things.
Even if there are bonuses, all the bonuses would be given on the basis of solving known problems. There is literally zero monetary incentive to solve unknown problems; only owners care about that stuff. Employees don't care about that. They might point out some whilst not mentioning others.
I've been a software engineer for 15 years and my experience is that unknown unknowns in tech are like a secret piggy bank for engineers to tap into in times of need. They are rarely brought up; only in the event that an engineer feels the need to justify their value to the company/project. The accumulation of secrets gives engineers differentiable value within the project/company and makes them hard to replace... So they tend to keep at least a couple to themselves... They ration it; like retirement planning.
The individual incentive to conceal issues until a time of personal need (when layoffs are looming) outweighs the collective incentive to identify new issues to generate more billable hours of work for the team.
If you're the only person who knows the issue which prevents a multi-million dollar project from coming to fruition, that is valuable information and gives you a lot of leverage in terms of job security... Which is the biggest reward you can get in this line of work.
And it's not malicious; the people who know about those issues probably also solve more issues and contribute more solutions than anyone else. The concealment may be happening subconsciously; they won't admit to themselves that a specific kind of problem exists.
Like when I worked in crypto, all the engineers there believed that the project could scale and also be decentralized... When it couldn't, given the chosen approach. The belief is still there years later; though all the rational engineers left, they were promptly replaced by believers.
I think intellectual integrity has been on the decline overall.
That said, the issue in this case is almost certainly systemic, not new physics. But I don't think you should consider the misfortune of the data-driven calculation typical; rather, it's notable specifically because of the discrepancies. Most calculations do not have such large disagreements. The lattice results, in contrast to those from the data-driven approach, all line up very nicely.
(This is more common than you'd think.)
LOL. No, no they are not. There's a lot of reasons I'm out of that game, and this is one of them. $1400 a month (2010 dollars), unlimited working hours, unlimited international travel. Sound fun? No?
> bonuses
I think I got a $5 Starbucks gift card once. I guess you could count the free leftover pizza as a "bonus"?
To address the heart of your argument, Science is not structured like software engineering, and if you're thinking it is, you're making a lot of mistakes. Scientists often do well as software engineers, should they change careers [waves hand!]; economics aside, software engineers very rarely make good scientists. The mentality is simply different.
As regards complexity management and dishonesty; yes, straight-up dishonesty does occur sometimes. We are all human, after all. But science is done in the physical world, and unlike software or mathematics, the physical world does not admit unlimited complexity. So things must be divided up in a manageable way right from the very beginning. On a big experiment, if your little "box" of responsibility does not work, you can and will be replaced. I have seen it done. Even to or by me. (If involved, I have historically been the one doing the replacing; but I've been on the other end, too.) And you will be cross-checked by independent people, if it is critical: I have seen this too. You had better get it right.