Thanks, all good points that would seem to make this library a good fit for certain use-cases.
As with the other poster, I’d be interested to hear a bit more about point 1.
As with the other poster, I’d be interested to hear a bit more about point 1.
Does this help clarify?
I think what factors in even more when you use the API is that you do not have fine-grained control over the generation process. If you follow the MS guidance approach, you fill in structured text yourself, and then let the model generate only the value parts, e.g. up to the next quote. To do that more or less word by word, you have multiple API calls, and have to be very smart about providing the right stop tokens.