Here is a very real example where 32k tokens is not enough: predicting discharge disposition for a long stay inpatient. A 30-day hospital stay can produce over 1,000 clinical notes.
For extracting what it was if it's in the document, you can use the indexing approach described in the article. For predicting what it should be, you'd probably be better off using llm as an extraction mechanism and then using structured data models to classify discharge disposition
But yeah my money is on context windows getting bigger and frameworks smoothing out how to do the above automatically, So it feels like a point in time optimization right now That will be tech debt by the end of the year
A lot of factors go into predicting the post-acute care route that results in the best outcome. You are correct that it would be possible to create features using an LLM, but that is a very difficult problem compared to simply treating the problem as a sequence classification.
"enhancements to automatically draft message responses"
"Another solution will bring natural language queries to SlicerDicer"
neither of which needs 32k token nor to see the patient records
also, foot note 1
"users of Azure and Azure OpenAI Service are responsible for ensuring the regulatory compliance of their use"
there is zero claim of actual compliances of these services for handling sensitive or regulated data.
now I understand the enthusiasm, but at least don't waste people time with sources that are, at best, tangential, and don't provide any substance to the discussion
"The second use will bring natural language queries and "data analysis" to SlicerDicer, which is Epic's data-exploration tool that allows searches across large numbers of patients to identify trends that could be useful for making new discoveries or for financial reasons. According to Microsoft, that will help "clinical leaders explore data in a conversational and intuitive way.""
Do you not understand what natural language queries on external data by access of/enabled by GPT-4 means/entails ?
The only thing worse than being condescending is being condescending and wrong.
that means natural language will understand user request and write queries for SlicerDicer data backend, not that data is sent to the llm for analysis.
it writes queries, don't receive data.
> That being said, you can run LLMs on-prem.
how is this relevant at all? it's just dodging the question, since the claim was about gpt4 32k tokens and not the crop of generic models that we have now availabe on prem (and which don't support 32k token anyway)