syllogism

You could do something like `value = table.get(some_function(), MISSING_VALUE)` and then have the conditional. But let's say for the sake of argument, yeah you need to assign the value up-front.

Let's say you're looking at some code like this:

    if value in table:
        ...

If you need to change this so that it's `some_function(value)`, you're not going to miss the fact that you have a decision to make here: you can either assign a new variable, or you can call the function twice, or you can use the '.get()' approach.

If you instead have:

    try:
        return table[value]
    except KeyError:
        ...

You now have to consciously avoid writing the incorrect code `try: return table[some_function(value)]`. It's very easy to change the code in the 'try' block so that you introduce an unintended way to end up in the 'except' block.

I still think using the exception mechanism for things that could just be conditionals is bad. I elaborated on this to a sibling comment.

Yes there's been various recommendations of this over the years and I think it's really bad.

Using try/except for conditional logic gives the developer a spurious choice between two different syntaxes to express the same thing. The reader is then asked to read try/except blocks as meaning two different things: either ordinary expected branching in the function, or handling exceptions.

I think it's definitely better if we just use conditionals to express conditionals, and try/except for errors, like every other language does. Here's some examples of where this duplication of syntax causes problems.

* Exceptions are often not designed to match the interface well enough to make this convenient. For instance, 'x in y' works for both mapping types and lists, but only mapping types will raise a `KeyError`. If your function is expected to take any iterable, the correct catching code will be `except (KeyError, IndexError)`. There's all sorts of these opportunities to be wrong. When people write exceptions, they want to make them specific, and they're not necessarily thinking about them as an interface to conveniently check preconditions.

* Exceptions are not a type-checked part of the interface. If you catch `(KeyError, IndexError)` for a variable that's just a dictionary, no type checker (or even linter?) is going to tell you that the `IndexError` is impossible, and you only need to catch `KeyError`. Similarly, if you catch the wrong error, or your class raises an error that doesn't inherit from the class that your calling code expects it to, you won't get any type errors or other linting. It's totally on you to maintain this.

* Exceptions are often poorly documented, and change more frequently than other parts of the interface. A third-party library won't necessarily consider it a breaking change to raise an error on a new condition with an existing error type, but if you're conditioning on that error in a try/except, this could be a breaking change for you.

* The base exceptions are very general, and catching them in code that should be a conditional runs a significant risk of catching an error by mistake. Consider code like this:

    try:
        value = my_dict[some_function()]
    except KeyError:
        ...

This code is very seriously incorrect: you have no way of knowing whether 'some_function()' contains a bug that raises a KeyError. It's often very annoying to debug this sort of thing.

Because you must never ever ever call a function inside your conditional try block, you're using a syntax that doesn't compose properly with the rest of the language. So you can either rewrite it to something like this:

    value = some_function()
    try:
        return my_dict[value]
    except KeyError:
        ...

Or you can use the conditional version (`if my_dict[some_function()]`) just for these sorts of use-cases. But now you have both versions in your codebase, and you have to think about why this one is correct here and not the other.

The fundamental thing here is that 'try/except' is a "come from": whether you enter the 'except' block depends on which situations the function (or, gulp, functions) you're calling raise that error. The decision isn't local to the code you're looking at. In contrast, if you write a conditional, you have some local value and you're going to branch based on its truthiness or some property of it. We should only be using the 'try/except' mechanism when we _need_ its vagueness --- when we need to say "I don't know or can't check exactly what could lead to this". If we have a choice to tighten the control flow of the program we should.

And what do you buy for all this spurious decision making and the very high risk of very serious bugs anyway? Why should Python do this differently from every other language? I don't see any benefits in that article linked, and I've never seen any in other discussions of this topic.

The examples in the article recommend doing the right things, but I think the initial discussion of the LBYL vs EAFP discussion could be better.

The article says basically "LBYL is bad", but this isn't a good description of what the author does in the later examples. Following "EAFP" without exception is also really bad.

The simple policy is that you should use "LBYL" when you're dealing with local state that can't change out from under you, especially when it's just properties of your local variables, but you need to use "EAFP" for anything that deals with remote state. Function calls can go either way, depending.

Blanket advice against LBYL leads people to use try/except blocks to express basic conditionals. If you want to ask something about your local variables, just ask it in a conditional --- don't make the reader infer "ah, these are the situations which will raise the error, and therefore we'll go into this block if the values are this way". If you want to do something when a key is missing from a dictionary, don't write:

    try:
        value = table[key]
    except KeyError:
        return default_value

Just write:

    if key not in table:
        return default_value
    value = table[key]

(If you really need to avoid two lookups for efficiency, you would do something like value = table.get(key, MISSING_VALUE) and then check 'if value is missing_value'. But this optimisation will seldom be necessary.)

The example the author gives about interacting with the file system is a good example of where indeed you really should use EAFP. The file system is remote state that your function does not own. Similarly if you're doing database operations, calling a remote API...lots of things.

There's various middle ground when you're calling functions. Often you don't want to worry about whether that function is going to be interacting with remote state, and you want to treat it totally as a black box, so you just use EAFP. But if the function documents really clear pre-conditions of your variables you can just go ahead and check, it's better to do that.

Maybe "trust" in a purely technical sense, but we live in a society. If a vendor has a part of their contract that would put them under penalty of fraud, customers don't usually worry whether they can independently verify it's true.

For instance, if you write in your agreements that data goes into this bit of open-source code and leaves it and goes nowhere else, and you list out some processes you take to ensure this is true, and the whole thing is an outright lie, that's really bad for you in ways that are clear and enforceable. It's also hard to hide from your own employees, and people come and go. You're unlikely to make that claim unless it's true.

The gold standard they're comparing against was done by humans though. And a task-specific model trained on that data will be better at that task than GPT-4.

What's definitely true is that getting decent data often takes some care, especially in how you define the task. And mechanical turk is often especially tricky to use well.

You probably know this but you definitely don't have to run into that problem. In practice most people who use one component to produce features for another will take care to ensure errors are present in the pipeline. So the conceptually simple (but operationally annoying) way to do this is to train your POS tagger or whatever on multiple folds, and predict the missing fold. This is known as "jack-knife training" in the literature.

In spaCy what we do is just train the components in sequence. So everything is trained at the same time, and in the early iterations the model is seeing samples with errors. I've always found this to be good enough.

There's definitely active research on it. But here's the basic recipe that gets state-of-the-art accuracy on most tasks. It's been around for 5 years or so now, which is why I said "just".

You take an encoder transformer architecture like BERT and train it on a language modelling objective. Then you discard the part of the network that does the token prediction, and stick some randomly initialised network that does a specific task on. This is generally kept minimal. For classification tasks, often people just connect a linear layer to the vector that represents a dummy sentinel token after the sequence.

The general goal is to exploit the representations learned during the language modelling task, so that the network starts out knowing general grammatical structure of the language, multi-word expressions, can distinguish word senses from each other, etc. Then it needs far fewer examples to learn a specific task. There's definitely transfer learning going on: if you initialise the network randomly instead of via the LM objective, results are very much worse.

This is a good blog post from when the recipe was still fairly new: https://www.ruder.io/nlp-imagenet/

(Author here)

About the speed, the idea is that if you break down the task, you can very often use much smaller models for the component tasks. LLMs are approaching prediction tasks under an extremely difficult constraint: they don't get to see many labelled examples. If you relax that constraint and just use transfer-learning, you can get better accuracy with much smaller models. The transfer-learning pipeline can also be arranged so that you encode the text into vectors once, and you apply multiple little task networks over the shared representation. spaCy supports this for instance, and it's easy to do when working directly with the networks in PyTorch etc.

This is where the terminology being used to discuss LLMs today is a touch awkward and imprecise.

There's a key distinction between smaller models trained with transfer-learning, and just fine-tuning a smaller LLM and still using in-context learning.

Transfer learning means you're training an output network specifically for the task you're doing. So like, if you're doing classification, you output a vector with one element per class, apply a softmax transformation, and train on a negative log likelihood objective. This is direct and effective.

Fine-tuning a smaller LLM so that it's still learning to do text generation, but it's better at the kinds of tasks you want to do, is a much more mixed experience. The text generation is still really difficult, and it's really difficult to learn to follow instructions. So all of this still really favours size.

Nothing's to stop us, and in fact we can do that now! This is basically what the post advocates for: replacing the LLM calls for task-specific things with smaller models. They just don't need to be LLMs.

(Author here)

It's true that getting something going end-to-end is more important than being perfectionist about individual steps -- that's a good practical perspective. We hope good evaluation won't be such an enormous effort. Most of what we're trying to do at Explosion can be summarised as trying to make the right thing easy. Our annotation tool Prodigy is designed to scale down to smaller use-cases for instance ( https://prodigy.ai ). I admit it's still effort though, and depending on the task, may indeed still take expertise.

I think the 'industrial' or assembly-line analogy is probably good, and I see what you mean about the alternative system design. Thanks for explaining the other approach.

I'll put it this way. If you want to integrate ML into a product, or even a system with lots of internal users, you end up increasingly towards the 'factory' approach.

I'm making the stronger claim that fine-tuned BERT is almost always better than ICL GPT4. That is, you beat ICL GPT4 with a model of less than 1b parameters. These are the results in the literature.

Fine-tuning GPT3 is actually a strange middle-ground: you're constrained to solve the tasks via prompting, rather than attaching output layers and doing gradient descent over a loss function that directly matches the task. So fine-tuning LLMs for these classification tasks is indeed often worse than just using the larger model.

The other thing is, as you scale up the model the ratio of output weights to the whole model weights gets quite out of balance. This introduces some technical challenges, and so it's kind of tough to make it all work with medium size LLMs of around 7b parameters or so. In contrast the process of fine-tuning BERT-sized stuff is really well explored, and it's easy to find more or less push-button solutions.

I know what you mean about the domain knowledge, and it's a thing that's a bit different from the previous situation with the feature engineering. The problem with feature engineering for linear models was you really had to understand the domain _and_ the ML.

I do think there's a similarity in how creative you need to be though. It means that applying LLMs to new problems isn't as straight-forward as people make it seem at first, and isn't necessarily reliable. In contrast, labelling data is something that has a much smoother effort to reward curve for most problems. The experience of labelling the data, training a model and getting it hosted isn't as seamless as it could be -- we're working on that.

I do think classification is pretty fundamental though. The way I see it is, model outputs can be either human-facing, machine facing, or both. If you're going to feed the output into another system, that system wants the data to obey some limited schema, so that you can run logic based on it. For instance, let's say you want to trigger some alert when a particular kind of article is published or a particular kind of message is sent. Triggering the alert is a boolean thing, so that has to be a classification task. You might want to also attach text in the alert, so that's a human-facing part.

I agree that there's lots of ways that LLMs can be used iteratively, allowing more trade-off of computational cost for accuracy. I just think in a lot of cases, the best way to exploit that is to trade towards as much accuracy as you can get, and use that to create training samples. You can then manually correct the training samples as well --- if they're mostly correct, reviewing them is pretty quick. You can then train and evaluate a smaller model.

(Original author of spaCy and Explosion CTO here)

Okay so, first some terminology. LLMs can mean a bunch of different things, people call models the size of BERT LLMs sometimes. So let's talk specifically about in-context learning (ICL) with either zero or a few examples. So we'll say LLM ICL, and contrast that with techniques where you annotate enough data to train with, which might only be something like 10-40 hours of annotation. The something you do with that data is probably training a task-specific classification model initialised with weights from a language modelling objective. This is sometimes called "fine-tuning", but fine-tuning can also mean taking an LLM and adapting its ICL. So we'll just call it "training", and the fact you use transfer learning like BERT or even word vectors is just tactics.

Okay. So, here's something that might surprise you: ICL actually sucks at most predictive tasks currently. Let's take NER. Performance on NER on some datasets is _below 2003_. Here's some recent discussion: https://twitter.com/mayhewsw/status/1700139745769046409

The discussion focusses on how bad the CoNLL 2003 dataset is, and indeed it's a crap dataset. But experiments have also been done on other datasets, e.g. check out the comparison of ICL and training in this paper from Microsoft: https://universal-ner.github.io/ . When GPT4 is used this one paper reports it slightly better on some tasks: https://arxiv.org/abs/2308.10092 . Frustratingly they don't do enough GPT4 experiments. This other paper also does a huge number of experiments, but not with GPT4: https://arxiv.org/pdf/2303.10420.pdf

The findings across the literature are really clear. ICL is generally much worse than training a model in accuracy, and you generally don't need much training data to surpass ICL in accuracy.

For tasks like text classification, ICL sometimes does okay. But you need to pick the problem characteristics carefully. Most text classification tasks people actually want to do have something like 20 labels, the texts are kind of long, and the labels don't capture the category especially well. Applying ICL to such tasks is very non-trivial. Your prompt balloons up if you have lots of classes to predict between, and providing the examples is hard if your texts are even a few hundred words.

Let's say you want to do something ultra simple: classify articles into categories for some news site or blog. This is the type of problem text classifiers have been eating for breakfast for 20 years. This is not a difficult problem -- a unigram bag of words does fine, and the method of estimating the weights can be almost anything, like just averaged perceptron will be totally okay.

But how will an LLM be able to do this? Probably your topic categories include several different types of article under them. If you know what those types of article are you can separate them out and make sure they're represented in the prompt. But now we're back at needing a tonne of domain knowledge about your problem -- that's like having to write custom features to make your model work. We all switched to deep learning so we wouldn't have to do that.

LLMs build a much more sophisticated representation of the meaning of the data to classify. But then you give them very few examples of the problem. So they can only build a shallow function from this deep representation. If you have lots of examples, you can learn a complex function from shallower features. For a great many classification tasks, this is better. The rest of your system usually needs the classification module to have some sort of consistent behaviours anyway. To do that, you basically have to make an annotation manual, and then you want to annotate evaluation documents. Once you're there the effort to make training data and train a model is minimal.

The other elephant in the room is the expense of the LLM solutions. The papers are missing results on GPT4 not because they're lazy, but because it's so expensive to use GPT4 as a classification solution that they want to get the rest of their results out the door.

The world cannot migrate all its current NLP models for text classification and NER to ICL. There are nowhere near enough GPUs in the world for that to happen. And I don't know about you, but I expect the number of text classification and NER models to grow, not shrink. So, the idea that we'll stop training small models for these tasks is just implausible. The OpenAI models that support batching are almost viable for prediction, but models like GPT4 don't support it (perhaps due to the mixture of experts?), so it's super slow.

The other thing is, many aspects of language that are useful as annotations are consistent linguistic features. The English language codes for proper names and numeric entities. They behave differently in the grammar. So some sort of named entity annotation can be done once, and then the model trained and reused. This is what spaCy does. We do this for a variety of other useful annotations across languages. We actually need to do much more: we need to collect new annotations for these models to keep them up to date, and we need to do this for more tasks, such as semantic role labelling. But it's definitely a good way to reuse work. We can do this annotation once, train the models, and users can reuse the models.

The strength of ICL is that you can get started very easily, without doing the work of annotation and training. There's lots of research on making ICL few-shot learning less bad on arbitrary text classification, NER and other tasks. We're working hard to take these results from the literature and build best-practice prompts and parsers you can use as a drop-in annotation module in spaCy: https://github.com/explosion/spacy-llm . Our annotation tool Prodigy also supports initializing the annotations from an LLM, and just correcting the output: https://prodigy.ai . The idea is to let you start with an LLM, and then transition to a model you train yourself, which can be run much faster.

The idea is to hash the string into a 64-bit key. You can store the string in a value, or you can have a separate vector and make the value a struct that has the key and the value.

The chance of colliding on the 64-bit space is low if the hash distributes evenly, so you just yolo it.

If you have very large dicts, you might find this hash table I wrote for spaCy helpful: https://github.com/explosion/preshed . You need to key the data with 64-bit keys. We use this wrapper around murmurhash for it: https://github.com/explosion/murmurhash

There's no docs so obviously this might not be for you. But the software does work, and is efficient. It's been executed many many millions of times now.

Sexual control and restriction is a pretty widespread feature of premodern cultures. I think you should stick to the soap. The porno is probably trouble.

> Behavioral modernity in humans including the way we use language is thought to have emerged 70000-55000 years ago.

Citation for that? There's clear evolutionary selection for language capabilities a long time before that isn't here?

It's a venture fund, as a lot of large companies have.

I don't really get the argument for it, I think it's pretty dumb overall. But it's not that uncommon.

The paper trains an XGBoost model to predict investment outcomes for startup investments from the time period 2014-2016. The model is trained on investments prior to 2014, and uses features from Pitchbook. It says the most important features are text descriptions of the company and CEO, which are featurized as unigram and ngram TF-IDF counts.

The paper avoids the very first methodological problem one would check for, which is using a held-out set that is mixed in time. That is: at least it uses a heldout _period_ (2014-2016), instead of individually held-out samples. Using individually held-out samples would let the model learn about future trends. If you know some companies that will be successful in 2026, that will help a lot in deciding what to invest in today, even if you can't pick those companies. So at least the paper doesn't do that.

However, the paper doesn't seem to have limited itself to information written and documented about the companies at the time of the investment decisions. Descriptions about the CEO will change over time. Once a company is looking to go public, maybe they emphasise that their CEO has an MBA. If your company died at seed, maybe this detail isn't included in the text. Even the people listed as the founders of the company is something that might be revised over time (Tesla is a famous example).

The paper really doesn't give us much insight into what the model learns. This means that we need to consider competing explanations for the model's accuracy --- we're not forced to accept the author's conclusion that the investments were indeed "predictably bad" in a way that could be profitably implemented as a trading strategy.

By itself, the fact that a black-box machine learning model with a large number of features can distinguish between two classes on a held-out set really isn't very strong evidence that the model has learned something of practical importance, that will generalise beyond the train and held-out sets you've collected. You need to show us what it's actually learned. If it's supposed to identify wolves, is it looking at the canines or the snow? For a study like this, I'd want to see feature pruning down to where the model is still accurate but with only a few textual features, and then show us what those features are. If you can't do that in a way that actually leads to human insight, it's extremely unlikely that the model is learning anything real.

More like kin selection: https://en.wikipedia.org/wiki/Kin_selection

You might be interested in pex: https://pex.readthedocs.io/en/latest/

They didn't just put out Docker with a sign up that said "free stuff" though. They actively solicited the users; they told the community "Hey, use our software, you can trust us". If you do that you do have an obligation.

It's like, if you beg your friend to let you give them a lift and don't show up, you can't turn around and call them a "choosey beggar" because the lift was supposed to be free.

The response in that thread isn't _remotely_ reasonable, of course you can't make a developer tool like this auto-update.

Oh, I didn't take it as a pointed criticism or anything. It just seemed like it would be an instructive example.

The underlying point I often make to people is that Python's slowness introduces a lot of incidental complexity, and you find yourself fiddling with numpy or something instead of just writing normal code and expecting it to perform normally.

(I'm the creator of spaCy)

If you want to email me at matt@explosion.ai , I'd be interested in the specifics of the algorithm and why the implementation was slow.

The idea for something like that keyword extraction algorithm would be that if the Python API is slow, you should just use Cython. The Cython API of spaCy is really fast because the `Doc` is just a `TokenC*`, and the tokens just hold a pointer to their lexeme struct, which has the various attributes encoded as integers.

I've never really done a good job of teaching people to use the Cython API though. I completely agree that it's not productive to have slow solutions, and using too many libraries can be a problem. The issue is that Python loops are just too slow, you need to be able to write a loop in C/Cython/etc. Thinking through data structures is also very important.

I get very frustrated that there's this emphasis on parallelism to "solve" speed for Python. Very often the inputs and outputs of the function calls are large enough that you cannot possibly outrace the transfer, pickle and function call overheads, so the more workers you add, the slower it is. Meanwhile if you just write it properly it's 200x faster to start with, and there's no problem.

> I'm not sure how to best reconcile these two competing interests.

What would help a lot is if the requirements were specified outside of the actual artifact, as metadata. Then the requirements metadata could be updated separately.

Everyone needs commands like "pip install 'spacy==2.3.5'" to work reliably in the future, so that you can go back and bisect errors. You need to be able to get back to a particular known-good state, and work through changes

I'm not saying we pin our dependencies to exact specific versions, but we absolutely do set an upper bound, usually to the minor version.

I think you're really under-rating how important it is to be able to do something like "pip install 'requests==1.0.5'" or whatever, in order to reconstruct the past state of a project. If requests hasn't pinned its dependencies, that command will simply not work. The only way you'll be able to install that version of requests is to manually go back and piece together the whole dependency snapshot at that point in time.

There's pretty much no point in setuptools automatically installing library dependencies for you if you expect the library dependencies to be unpinned. In fact it would be actively harmful --- it just leads people to rely on a workflow that works today but will break tomorrow.

You're asking for an ecosystem where there's no easy way to go back and install a particular version of a particular library. That's not better than having version conflicts.

The other thing I'd note is that it's quite an understatement to say that pinning dependencies makes life "slightly easier" for library developers. We're not going to accept builds just breaking overnight, and libraries that depend on us aren't going to accept us breaking their builds either.