Sound awesome! Looks like you invested a lot, e.g. inference batch is not easy at all.
Few follow up questions:
- Do you really retrieve embedding on every batch? Doesn't it make sense to keep them in memory (must not be more than few gigs)?
- Do you have/plan to incorporate inference feedback loop for retraining? Or you call it metric as well?
Thanks for an answer!