How do you make 15 million of anything without realizing there's an issue, let alone the most scrutinized thing on the planet right now?
It'd be like the difference between burning dinner 7 nights in a row, versus making a single mistake during Meal Prep Sunday and ruining a weeks worth of dinners all at once. The latter is much more understandable.
Microbrewers often use fermenters larger than that.
This is akin to me using a large 64 core server for an app, instead of say 4x16 . Sure, the larger server can easily handle the load, however the 4x16 setup is more redundant and avoids many scenarios where all users will be out of service
Just because there are high capacity machines capable of doing that much does not mean it should be used especially a With a new process trying to scale up
(If you're using a 4x16 server, whatever that is, to provide isolation in your system, you may want to talk to additional devops folks for other options.)
Yes fuck ups happen at plant and in planning, and yes we are not experts in the field to comment in detail on what is best approach. However if you are looking at it as an admin, clearly risk was not well managed.
Obviously, the mistake is bad. But this is something with a very high fan out. And the smaller number just doesn't get the clicks. :(
And it's put other production from the facility into question.
Obviously, it should be investigated such that were can reduce the chance is mistakes in the future. But piling on with blame will just convince folks not to try.
And then there's the much greater value of the doses once they have been administered.
HACCP would prevent this in any number of ways.
What makes them different is 1) there are ample checks in place to prevent mistakes from ever leaving the facility, and 2) there is process that reduces the chances of having it happen in the first place.
But all of these places will have incident logs of mistakes like this, along with corrective actions that change the procedures and make it less likely to happen a second time.
They shouldn't happen in a 150yr old, multi-billion dollar pharmaceutical company.
How hard is it to restrict the ingredients available to the "chef" to the ones needed for the current product? Not hard at all unless you want to save a few dollars cutting corners.
Some other comments have already pointed out that 15M doses is probably only one or a few batches. Combined with the fact that the procedure documents for this particular line are probably still warm from the laser printer, I'd say let them iterate a bit more to get the kinks out.
[ Procedures are not (should not be) engraved on stone tablets handed down from on high. If you try it that way, you are just as guaranteed to fail as when you had no procedures at all. ]
As to what went wrong at this facility (besides overpromising?); we'll have to see. I wish there was such a thing as open source procedures!
In a situation where maximizing vaccine production sooner saves lives, pinning this on saving a few dollars doesn't make sense. Whatever they were doing cost them quite a few dollars in addition to the lives lost from slower vaccination.
I don't know their process, but suppose the "restricting the ingredients" phase happens. What if the ingredients were swapped at the time of restriction? Same problem happens.
Chef goes to find eggs as required by the recipe for his cake, and finds that there are none. Instead there is pepper, an ingredient not required by the recipe.
Or have the employee log what he is doing in real time BEFORE he does it. I am adding sugar, I am adding eggs, I am adding wheat flour. Then an alert is triggered when the system (which knows what product is being made) recognises that the ingredient doesn't belong.
This is a multi-billion dollar company... they could have the process live-streamed to 30 independent chemists each one watching the "chef's" every move (this is not a serious proposal).
This stuff is all hashed out when one writes up a HACCP process.