Prodigy: A new tool for radically efficient machine teaching

Fei-Fei Li gives a good sense for this in her history of ImageNet [1][2].

[1] https://qz.com/1034972/the-data-that-changed-the-direction-o...

[2] http://image-net.org/challenges/talks_2017/imagenet_ilsvrc20...

For the specific questions:

1. The built-in web views all have binary annotation interfaces. This is more of a design choice than a fundamental limitation, and the front-end is extensible --- you can add your own web views if you need to.

The binary interface is sort of a position statement. We think this is The Way, so we want you to try it. We'll have more input components in future, but at the start we want to guide people towards the intended workflow.

2. The beta focuses on NLP support, but there's a front-end for image classification, and a workflow page: https://prodi.gy/docs/workflow-image-classification .

3. You can usually get some accuracy improvement by retraining once all the annotations are available. I've not found a streaming SGD algorithm that works as well as the simple iterate-and-shuffle batch process. Batch training also lets you tune the hyper-paramers. You can read more about this here: https://prodi.gy/docs/workflow-named-entity-recognition#trai...

I would suggest writing a Prodigy recipe to do the batch training. That way you can pass in the dataset ID, instead of exporting the annotations. There's no problem with exporting the annotations and running a script, though. Again --- it's all on your computer. You can run it however you like.

Sounds like marketing BS. what about OpenNLP and Stanford's for NLP?

I do think there's a pretty common failure mode for teams who don't have much experience with ML, though. Teams who don't have much experience with ML often take "We don't have much data" as a parameter of their problem, and don't see that this is something they can decide to change. This can lead to a lot of time spent experimenting with different unsupervised approaches that are a poor fit for what they're trying to do.

We're mostly worried about saying we "support" a language when we've just trained a tagger on a UD treebank, though. We like at least having the stop words and tokenizer exceptions filled in by a native speaker, so the usual flow has been that someone needs the functionality, and they make a pull request.

If you just need the UD model for say, Bulgarian, you can do:

    python -m spacy train xx /path/to/output_model /path/to/bulgarian-train.conllu /path/to/bulgarian-dev.conllu --no-entities

It looks like it is online during typing of annotation, trying to predict annotations.

When it says teaching, it means teaching the AI. When it says "radical" it means ... getting slightly more data input, and in an online manner.

Examples for text data: topic tags, marking mentions of companies, finding descriptions of protein interactions. For images people mostly do segmentation (finding object boundaries) and classification.

If the model is generating images or text, you usually need to do annotation to evaluate the model. For instance, you need to know whether the translations are grammatical, whether an image is coloured correctly, etc.