Hey, I looked at the solution, Seems like you are using JSON forms to validate the response from LLM and iterate it until it matches the response. Again, I don't see why would I use this in my app as the API call will increase the latency and I can use the same method to create Rust or Go code from GPT4 or other code based LLM's. Am I missing something here?
We actually leverage LLMs very sparingly. We do not generate the transformed data directly as this would not just introduce significant latency, but also quality and reliability issues. Rather, we use LLMs to produce high-level mapping/transformation logic in a language of our design that is deterministically executed to produce your desired data. So this means LLMs are used only when you introduce new data formats that requires new logic, and used surgically even then. The vast majority of usage so far in terms of volume leverages the logic created in the underlying pipeline and will not have latency issues. This also allows for building reliable and stable pipelines with our APIs, which is a requirement that's difficult to meet considering the non-determinism of LLMs.