Hacker News Reader

Curious about: does it have fault tolerance if one of the machines goes down mid-inference? Can it dynamically reroute, or does it just retry?

It can dynamically route. If a machine drops out of split, the topology is recalculated and the request is automatically retried.