> (1) Is there such thing as a "end of reply" token ? (I'm not taking about End of Sentence <EOS> which is obviously different)
EOS as end of sentence? IIRC its end of sequence, which may also answer the question IIUC.
EOS as end of sentence? IIRC its end of sequence, which may also answer the question IIUC.
So, if we simply wait for <EOS> to be sampled in a block, that could be the place to stop. My answer above should be the same (modulo this correction) -- simply keep obtaining the blocks in a semi-autoregressive fashion.