Is it expected to be slow? Each request takes several seconds to reply.
Also how recent is the training data? When I asked what’s the date today? I received 22nd of March 2018.
Interesting choice of using streamlit for interface.
Also how recent is the training data? When I asked what’s the date today? I received 22nd of March 2018.
Interesting choice of using streamlit for interface.
Probably, yes. The slowness is not on the Streamlit end, but on the Replicate API end. The docs for the 13b API [0] say:
> Predictions typically complete within 9 seconds.
Whereas for the 70b API [1]:
> Predictions typically complete within 18 seconds. The predict time for this model varies significantly based on the inputs.