Efemarai: Stress Test and Validate Your Computer Vision Models
github.com
github.com
Me and Svet(co-founder) have worked in industry (building self-driving cars, to cancer-research to different DARPA projects) and have come to the realization that similar to how software/EE/Aerospace is tested/QA'd as part of the deployment process, we need more rigorous steps in the ML domain (beyond test/val datasets or thumbs-down).
Our hope is to allow ML teams to build test suits for their models, embed them as part of their CI and give another layer of confidence that when you re-deploy, you're not going to regress and to the infamous one step forward, two steps back.
You can register at https://ci.efemarai.com and easily submit jobs through python/cli (pip install efemarai). You'll get access to: - Operational Domain Finetune how the images should be transformed, such that they cover the variability the model is expected to see in the real world. We know datasets cannot have it all, so we're releasing a tool that has helped us a lot in being able to both encode "business level performance SLA" - it should work with this small objects, in darkness, under lense flare, when the face is x% rotates or form this azimuth (and not plain mAP or accuracy)
- Support any Input and Output data types Not only do we support tasks such as classification, object detection, instance segmentation, keypoints detection, regression, but also any combination thereof, with any type of input - single image, multi-image, video, text, or anything that combines those. Right now this is such a major pain point from arbitrary open source datasets, loaders, we really hope to provide an easy way to encode the teams internal structure to something that is generalizable.
- High efficiency There is so much gain to be made in being thoughtful in the data that is being used, rather than randomly augmenting data. With Efemarai you can find and fix failure modes of your model given the degradation of performance that is extracted from purposeful transformations.
We look forward to seeing you at Discord https://discord.gg/cWQC3rrB and chatting with you!