This blog post clearly targets VCs, but what they are doing is legit and can improve the performance of local models on low-end hardware as well, especially since their priority is to optimize non-batched inference.