Part of the motivation for SymForce was in fact serious attempts (multiple months) by us to do similar things with TensorFlow XLA and being stymied by some of these effects. Many of the problems we deal with, and many problems in robotics in general, deal with small "tensor sizes", where dimensions of individual variables are often <10, and batch sizes aren't huge either. SymForce-generated C++ code is typically one machine instruction per scalar per "op" - compiling TF / JAX etc using methods like you mention definitely helps a ton, but it doesn't reach that level of efficiency for small problems. And computing Hessians for second-order optimization tends to blow up op counts even more.

Similarly, SymForce examines sparsity of individual matrices and jacobians at code generation time, so that operations with sparse matrices are similarly one instruction per op - at runtime there's no looping or indirection through sparse data structures or allocations of sparse matrices in the optimization loop. There's definitely a cross-over point where dimensions become large enough that TF / JAX / etc will be the tool to use - we'd love to provide some more benchmarks and comparisons here. For the first two experiments in the paper (see Tables II and III), we do compare to JAX and amortizing across a batch size of 1000, SymForce is still way ahead. We tried to be careful, but the code is available so please come beat up our numbers.