lgessler

I'm afraid I feel that this piece confuses more than it clarifies.

First, saying a model "scours through its vast training data" is misleading at best: at inference time, LLMs no longer have direct access to their training data; they only have access to it insofar as it's been encoded in its parameters.

Second, saying "Every instance of an answer in the training data is like a vote" doesn't give the full picture. First of all, there are embedded contexts where "votes" can be negated: consider saying "the Earth is flat." vs. "We know it's false that the Earth is flat." or "Only a fool thinks the Earth is flat." All three contain the substring "the Earth is flat.", but both humans and LLMs are able to use context to understand that the latter two sentences are doing the opposite of endorsing the proposition that the Earth is flat. You could even imagine an extended satirical bit with "the Earth is flat" embedded within it where it is clear to a reader that all its content is intended to be taken as farcical, and I'd wager that an LLM would in many cases recognize this. So the voting metaphor breaks down here--it makes you think that the LLM is just keeping a tally of propositions, but really, it is doing something a bit more sophisticated.

I don't disagree with the premise, of course. LLM overhype is real. But we should be skeptical for the right reasons. Anna Rogers and Sasha Luccioni have a paper I really like: https://openreview.net/pdf?id=M2cwkGleRL

IMHO reviews are almost indistinguishable from noise at the AI conferences I'm familiar with these days anyway, so I don't see much of a value add.

GP wrote "simple to use API". You can attribute many qualities to TensorFlow, but this is not one of them.

I liked Yoav Goldberg snarky's quote tweet:

> next paper: transformers can solve any problem but on some of them they may compute indefinitely and never provide an answer

> (and you cannot tell in advance which is which!!)

https://twitter.com/yoavgo/status/1835802380589203802

Most LLMs determine their token inventories by using byte-pair encoding, which algorithmically induces sub-word tokens from a body of text. So even in English you might see a word like "proselytization" tokenized apart into "_pro", "selyt", "iz", "ation", and non-English languages will probably (depending on their proportional representation in the training corpus) also receive token allocations in the BPE vocabulary.

Here's actual output from the GPT-4o tokenizer for English and Hindi:

    >>> [enc.decode([x]) for x in enc.encode("proselytization")]
    ['pros', 'ely', 't', 'ization']
    >>> [enc.decode([x]) for x in enc.encode("पर्यावरणवाद")]
    ['पर', '्य', 'ावरण', 'वाद']

A division of the NIH (NCBI) maintains a repository of open access publications in life sciences called PubMed Central (https://en.wikipedia.org/wiki/PubMed_Central) and this article was in a qualifying journal.

Yep, came here to say this. The big thing about the results here that might not be obvious to someone not in AI is that the models being trained in this paper are very many orders of magnitude smaller than the LLMs we've all heard so much about recently, and they're also being trained on specific tasks instead of general language modeling.

So I'm not expecting this will find its way into a LLaMA near me any time soon, but maybe this is an interesting result for people working in the specific domains represented in the evaluations.

So you're thinking of expressions like "Winston said he had noticed a continued decrease in his suicidal ideation" (real example). I don't know if I'd agree that it's clear a rule-based system would be able to catch everything like this. Consider:

1a. (original) Winston said he had noticed a continued decrease in his suicidal ideation.

1b. (simplified) Winston noticed a decrease in his suicidal ideation.

2. Winston's suicidal ideation decreased.

3. There was a decrease in the suicidal ideation Winston felt.

4. Suicidal ideation decreased in the patient.

Many more syntactic repackagings of similar propositional content are possible, too. For perfect recall, rule-based systems need to grapple with this, and additionally, in a real setting you're probably getting predicted (rather than gold-standard) POS tags, relations, etc., adding additional noise.

My expectation (as someone who's mildly bearish on LLMs, btw) would be that if an LLM appears to be doing worse than a rule-based system then you probably haven't tried some low-hanging fruit yet such as adjusting prompting strategies, fine-tuning, using different pretraining data, etc.

I don't personally have a glottal stop in "look cool". I think it's the same as "lukecool" except for one vowel, at least in my English:

* lukecool /luk kul/

* look cool /lʊk kul/

This might have something to it, but there are plenty of common expressions that have the same phonetic properties you're describing (IPA transcriptions provided):

* fast track /fæst træk/

* swim meet /swɪm mit/

* Fat Tuesday /fæt tuzdeɪ/

* hat trick /hæt trɪk/

* fake coin /feɪk kɔɪn/

* drip pan /drɪp pæn/

BTW, I don't know if your account of how the phonetics of such words works out is correct. The traditional account is that in situations like this where the same consonant appears on either side of a word boundary, English speakers actually hold the consonant for a "double length", which is called gemination. (This is in fact very common in some languages, such as Italian, e.g. "sette" 'seven' vs. "sete" 'thirst', but English only recognizes consonantal length contrasts in consonants in this limited sort of situation.) The Wikipedia page has a good discussion: https://en.wikipedia.org/wiki/Gemination#English

You can verify this for yourself by comparing some minimal pairs. Consider:

1a. fast track /fæst træk/

1b. fast rack /fæst træk/

2a. fake coil /feɪk kɔɪl/

2b. fake oil /feɪk ɔɪl/

In my personal English, each a-b pair differs only in the amount of time I produce the /t/ or /k/ closure for at my alveolar ridge or velum.

I tried to get into emacs a few years ago and just got fed up with the incredible regularity with which packages broke during routine updates. It was at least an hour a week on average, I feel, conducting bug hunts for either minor (this or that keybind/function doesn't work) or major (I can't get any packages to load) bug hunts. Was I doing something wrong? How do emacs people deal with this?

Admittedly, I was using Spacemacs which probably has way more packages than a bespoke emacs configuration, but you must understand if people get turned off when they ask you how to use an editor and your answer is "first, spend a few weeks setting it up and understand how each of your many packages works".

The gold standard is immersion, everyone agrees on that. Most people can't have that, so a mix of language instruction, conversation with native speakers, and exposure to appropriately challenging texts with memorization tools like Duolingo or Anki as a _supplement_ is the next best thing.

IMO this is a misunderstanding of the product.

This might be a cynical take, but I think that for most users of Duolingo, it functions more as a game than as a tool which produces substantial gains in language knowledge. I'm a computational linguist, and whenever on hearing this people ask me what I think about Duolingo, I like to say it's language-flavored entertainment.

> The sad truth is - all of classical nlp is Dead, with a capital D.

If your language is well served by LLMs. Which, generously, is true for maybe 20-50 of the world's 7000 or so languages. For all the rest, "classical" NLP is still how things get done at least at present.

Sure, but I don't think even Bender or Marcus would deny their potential for practical utility, right? I think they mean to say that LLMs are not exactly as miraculous as they might seem, and very capable of misbehavior.

Idk if this work really bears on that argument. I can imagine Bender reacting to this by observing that this is a clever way of finding the needle (a good heuristic algorithm for an NP hard problem) in a haystack (millions of other heuristic algorithms that are bad, very bad, or maybe not even syntactically well-formed), and then saying that the fact that the model still produced so much garbage is proof that the LLM still doesn't really know how to reason or access meaning.

And I think that'd be a good argument. OTOH, this system is not just an LLM, but an LLM and a bunch of additional components on top of it, and there might be different things to say about this system considered as a whole.

Tl;dr in my words after a skim: this is a method for using LLMs to find new, SOTA (or at least very good?) heuristic algorithms for NP hard combinatorial optimization problems. They achieve this not by making the LLM itself smarter but by finding a way to keep only its best ideas. (Haven't had a chance to read carefully yet so caveat lector.)

Pretty impressive, but don't panic--we're not at the singularity yet.

In a sense you're not wrong, but would you say the same thing about refined sugar? Both of these substances, in terms of immediate physiological effects, are known to be unambiguously bad for you past a certain low threshold. But immediate physiological effects are probably not the terms with which we ought to evaluate our lifestyles.

It's been a while since I worked with any of them so it'd take some review form me to even write a blurb confidently. Sorry! On your latter point though, NLG from AMR is a topic that's been studied since its inception. See the AMR bibliography[1] and CTRL+F "generat".

[1]: https://nert-nlp.github.io/AMR-Bibliography/

AMR: https://github.com/nschneid/amr-tutorial

UCCA: https://aclanthology.org/P13-1023.pdf

And yeah, you could consider UMR an improved version of AMR. AMR is often lampooned with the false name "American Meaning Representation", referring to its unapologetic focus on English only. UMR tries to undo this sin and has a lot of the same people behind it, as well as some new features such as document-level context.

CC BY SA 3.0: https://github.com/amir-zeldes/gum/blob/master/LICENSE.txt

I didn't know about that project, that's really cool! I'd be curious to know whether the person who devised this scheme was aware of structured meaning representations (UCCA, AMR, ...), and if so, why they chose to create a new meaning representation. Maybe the goals of the project and/or the constraints of Wikidata necessitated this.

Anyway, GUM (and its sister corpus EWT) does have a lot of parsed permissively-licensed text, so whoever's in charge should definitely consider using them. (Amir, the maintainer, is also super friendly and would respond to an email.)

Nobody asked, but here's a syntactic parse of a portion of this article in Universal Dependencies done by yours truly :) https://github.com/amir-zeldes/gum/blob/master/dep/GUM_bio_g...

I think you're right. Here's my math:

https://www.wolframalpha.com/input?i=%28%281+square+mile+*+1...

Not surprising, imo—voice assistants got big at a time when there was no way to implement them except by what I would call a massive catalog of hacks. Now that LLMs with really good NLU for major languages exist, Amazon must have made a determination that either voice assistants are no longer worth actively developing, or that LLMs ought to serve as the core of the NLU component of their dialogue systems.

How are conflicts resolved?

YMMV but this doesn't seem true to what I've witnessed in my own life. I can think of many friends (I'm 29) who found either an LTR or their spouse using dating apps. In many cases they were single for long stretches and complained bitterly about the process of finding a partner, but in this respect, I'm not sure if apps are different from any of the alternatives.

I'm a linguist, let me try to explain. It's true that in some cultures, especially very economically developed ones, it's common for adults like parents or schoolteachers to teach kids "language". But this is more about familiarizing them with spelling, literature, writing style, and prescriptive norms (e.g. don't split your infinitives), which are all from a linguist's perspective on the periphery of "language" in the narrow sense, which consists just of core grammar. And when it comes to this sense of "language", kids learn it all pretty much with no explicit instruction: the order that adjectives go in, how to form relative clauses, etc.

Consider these sentences:

1. The angry man complained to the manager.

2. The angry about his food man complained to his manager.

3. The man angry about his food complained to the manager.

I won't get into the details of everything that's going on here, but (3) is much "better" than (2), and kids naturally learn that you need to move "angry" or any other adjective to the other side of its noun when it gets a prepositional phrase complement. No parent is ever telling their kid (except a linguist parent, maybe, since they love making their kids subjects), "Remember, Jon, when the adjective has a _complement_, it needs to come _after_ the noun rather than before it."

As someone who's casually been eyeing up CRDTs for two years I've wondered continually what the story for authorization is, and this article seems to suggest (in line with my own understanding) that a CRDT library on its own like Y.js can't really handle applications where mutations need to be checked for authorization because it lacks a central authority (i.e., the server) while Reflect assumes that a server will be mediating client interactions. Is this all correct?

IMO, as an NLP researcher, it's a product of the "fast science" culture of AI where there are more papers than anyone could possibly read even in a given subfield at a particular conference, and an "old" paper is anything that was published over two years ago. (Who knew Bollywood movies and AI publications would have so much in common?) Maybe there are good reasons why publications should move so much more quickly in AI as opposed to, say, scholastic philosophy, but it's undeniable that it has become a struggle to get others to even read an abstract of your work, and as such we should not be surprised when sensationalization verges on becoming a requirement for your work to get engagement.

So to answer your question--perhaps cynically--“Tiered memory layers to provide extended AI context windows” is not the best title you could have for an AI paper because it has all the rizz of a shipping manifest. If you want to maximize citations, you need to market more.

I also considered whether to blame the field's very young reviewer population for not having the proper disdain for sensationalization that I'd expect from an older researcher who would surely have more restraint than to speak so much more of the sizzle than of the steak, but then I remembered that the paper which introduced the Transformer model was titled "Attention Is All You Need".

SEEKING FREELANCER | USA | REMOTE

We are a couple of academics who run a popular Olympiad-style competition for high schoolers nation-wide. Our current software system for administering this competition is quite old and idiosyncratic, and we want to do a rewrite.

The administration of the test itself is not handled by our software (it's done on paper), but all of the organizing around it is handled by the software. This is all pretty standard CRUD stuff: there should be a user system that distinguishes at least three kinds of roles (admins, who need to create competitions and publish standard text-based content like announcements; proctors, who need to be able to be associated with competitions and dates they'll be administering tests and the students who'll take it; students, who need to be able to register for tests at certain times and locations).

We have a lump sum of $30,000 for this work, but it needs to be disbursed by the end of this year or else it disappears, so this project needs to get done on a tight timeline. Also, we should be upfront and tell you that you'll be getting paid through university bureaucracy, which is not difficult per se, but does involve a decent amount of paperwork. We think this is a great opportunity for a senior web developer who has done a lot CRUD stuff like this before. We do not have any requirements about use of any particular tech stack, though all else being equal, we'd mildly prefer Django in Python.

If you're interested, please contact me at lukegessler (αt) gmail (dοt) com, with [HN] in the subject line so that I know you've read this posting, and write a paragraph or two explaining your relevant experience for this opportunity. Our engineering expert (me) and our domain expert (my colleague) would then work quickly with you to hammer down a list of requirements to deliver, and you'd be off to the races!

PS: we need to check on this, but we suspect you may need to be a US citizen, unfortunately. Will update this post ASAP.