So are you saying they shouldn't be accountable to elected officials?
Libraries need to follow constitutional principles and serve their whole community, not just the loudest subset. The broader community clearly supported keeping their award-winning library intact.
1. Book banning/burning is really bad (duh).
2. The worst book banners/burners throughout history have employed the same justifications as you now employ.
To the question about accountability: Who knows, but it isn't as important as avoiding book banning/burning.