Comment by code51 | Hacker News Reader

Comment by code51 | Hacker News Reader

Correctly lists US presidents in chronological order (which is an important test to me).

However for "List Led Zeppelin albums in chronological order"...

Here is a list of Led Zeppelin albums in chronological order: 1. Led Zeppelin (1968) 2. Led Zeppelin II (1969) 3. Led Zeppelin III (1970) 4. Led Zeppelin IV (1971) 5. Houses of the Holy (1973) 6. Physical Graffiti (1975) 7. Presence (1976) 8. In Through the Out Door (1979) 9. Cymbals and Wine (1980)

It's missing "The Song Remains the Same", weird but important.

GPT-4 is also missing it: 1. Led Zeppelin (1969) 2. Led Zeppelin II (1969) 3. Led Zeppelin III (1970) 4. Led Zeppelin IV (1971) 5. Houses of the Holy (1973) 6. Physical Graffiti (1975) 7. Presence (1976) 8. In Through the Out Door (1979) 9. Coda (1982)

"The Song Remains the Same" is a live album of the "concert film". Both GPT-4 and Mistral don't seem to treat it as also an album.

When asked as a follow-up, GPT-4 says:

""The Song Remains the Same" is a live album and soundtrack by Led Zeppelin, released in 1976. The list you asked for was comprised of their studio albums only." (note that I didn't specifically say studio albums only)

Mistral says something else:

"The Song Remains the Same was released as a single in 1976, not as an album. As a result, it is not included in the list of Led Zeppelin albums in chronological order."

Same behavior, different explanation.

Interesting to see alignment is this hard, even in basic stuff.

It seems many lists online don't include "The Song Remains the Same" in typical main discography lists, including on the main article of Wikipedia.

When I ask this question to Mistral 7B (running right now through ollama on mac) I get the exact list of studio albums as seen on Wikipedia. It got 'Coda' but it actually says "Please note that "Coda" is a compilation album and not an original studio album" which is correct.

If you think about these models as a knowledge compression it's pretty wild it's able to hold all this in only a few gigabytes and we get subtle understandings of human language, and some basic general natural language processing capabilities to go with it... all spitting out of 4GB (when quantized). Like, sure you could compress most of the text in wikipedia to 4GB but that alone wouldn't get you to the level of natural language reasoning that this gets along with the data.

For me, it's much less about what definite facts are stored losslessly in the model (like Led Zep albums) and more about the abilities to do one-shot learning and reasoning about data in context... the power there is useful to many applications.

English-language publications, databases, etc. have largely settled on the term 'Albums' referring by default to only studio albums. Live albums, EPs, etc. are referred to more specifically.

It's not universally true, but if you go to the Led Zeppelin wiki page, the short form discography does not list The Song Remains The Same - and you can find many other examples of this sort of distinction, all the way to audiophile-focused music torrent sites like oink and it's chain of successors.

In light of this, it's not surprising that many models follow the same pattern.

MS Copilot is apparently a music fan.

Most LLMs have problems with subtly, such as compound anagrams tend to repeat the same words in reverse order rather than reversing all of the letters in all words.

Reply on news.ycombinator.com