By "the patent office" you mean taxpayers, right? Because we're the ones who foot the bill for any judgement against the government. I can't see any situation where individual patent clerks would be held accountable. First, it goes against established case law regarding civil cases against federal employees [1, 2]. Second, if you amend the law so that patent clerks are an exception and can be held individually liable for potentially tens of millions of dollars, only absolute idiots would agree to take that job.
[1] https://www.justice.gov/jm/civil-resource-manual-33-immunity...
[2] https://www.ojp.gov/ncjrs/virtual-library/abstracts/personal...