Recently I found that GPT4 can't even reliably create a list of german nouns with a given article (der / die / das).
It will mess up a simple list - if you ask it to analyse it, it'll be able to tell you that it's wrong.
Then you get it to correct the list and it may still be wrong.
It can take several iterations to make the list correct. I would have thought this would be a super easy task for it, but apparently not.
Seems they're enshittifying already, I hope a competitive model is released soon.
If my Tesla went 80 mph and started going 15, we wouldn’t attribute that to the-nature-of-Teslas, or a software update: there are vast incentives for anyone who knew that was done to share that publicly.
Instead, we know something is wrong with the individual car and we take it to the dealer.
Here, the missing part is objectivity via a speedometer: I have one, I know outputs are consistent on 0315 models at 0 temperature.
Oh, let me guess... because OpenAI told you so. OpenAI, the one istitution with the most strong incentives to tell people the model is not getting worse.
People are incredibly silly, myself included, you get old enough and see an _insane_ influx of new people, you figure pretty much exactly this is going to happen. 95% of people don't know what temperature is. Of the 5% remaining, 4.9% think its something you just tell ChatGPT to adjust.
It makes total sense that they would throttle due to the incredible global demand. They can't build super computers fast enough now.
> Erstelle eine Liste mit 10 Nomen, die den Artikel "der" haben.
Maybe "reliably" is doing a lot of the heavy lifting?
It's still extremely helpful, I just found it strange that it seemed like a simple task - for something that has been fed millions of documents, for it to still give some incorrect results - especially AFTER it had analysed its own results and found some noun artikels to be incorrect.
But as you say, they are still extremely helpful anyway.
I actually didn't realise it was giving me incorrect info until my gf started looking at it!
(I was trying to use it to help me learn german)
The list was mostly correct, but yes it added nouns that were not "der" nouns into the list.
It then attempted to correct the list and failed at correcting it.
In terms of output, it also didn't want to give a list of 200, but I did manage to get a list of around 100 back.
It was unwilling to write "Der Jahr" when "Das Jahr" is correct.
GPT answer:
It's difficult to determine the "200 most common" nouns for a particular gender in German since the frequency of word use can vary greatly depending on the context (e.g., conversational, academic, technical). However, here's a list of 100 common masculine nouns, which use the definite article "der".
der Mann (man), der Tag (day), der Freund (friend), der Vater (father), der Bruder (brother), der Sohn (son), der Monat (month), der Kurs (course), der Tisch (table), der Stuhl (chair), der Garten (garden), der Zug (train), der Platz (place), der Himmel (sky), der Wagen (car), der Brief (letter), der König (king), der Schuh (shoe), der Baum (tree), der Hund (dog), der Bär (bear), der Fisch (fish), der Vogel (bird), der Schüler (pupil), der Regen (rain), der Wind (wind), der Wald (forest), der Berg (mountain), der Fluss (river), der Junge (boy), der Kaffee (coffee), der Lehrer (teacher), der Arzt (doctor), der Student (student), der Polizist (policeman), der Künstler (artist), der Teller (plate), der Löffel (spoon), der Geldbeutel (wallet), der Laptop (laptop), der Schalter (switch), der Fernseher (television), der Schrank (cupboard), der Helm (helmet), der Hut (hat), der Schmuck (jewelry), der Ring (ring), der Sport (sport), der Schaden (damage), der Boden (floor), der Sand (sand), der Sturm (storm), der Preis (price), der Feind (enemy), der Held (hero), der Raum (room), der Morgen (morning), der Abend (evening), der Unfall (accident), der Vortrag (lecture), der Urlaub (vacation), der Ausflug (trip), der Hafen (harbor), der Bericht (report), der Name (name), der Bauer (farmer), der Rechner (calculator), der Traum (dream), der Anzug (suit), der Geschmack (taste), der Eindruck (impression), der Zweck (purpose), der Vertrag (contract), der Krieg (war), der Kunde (customer), der Arbeitgeber (employer), der Mitarbeiter (employee), der Kollege (colleague), der Bewohner (resident), der Fahrer (driver), der Gast (guest), der Kritiker (critic), der Profi (professional), der Sieger (winner), der Kandidat (candidate), der Beamte (official), der Insasse (inmate), der Zeuge (witness), der Beweis (proof), der Schatten (shadow), der Zweifel (doubt), der Trauer (grief), der Frieden (peace), der Nerv (nerve), der Horizont (horizon), der Gedanke (thought), der Lohn (wage), der Antrag (application), der Verlust (loss), der Betrag (amount),
There's a common pattern in GPT discourse (here on HN, but elsewhere too): somebody describes a limitation, somebody else goes "no such limitation exists: look, here's GPT output", and a third person goes "no, here's why your example demonstrates the limitation".
It's interesting. People find it very hard – or are disinclined – to distrust a charismatic robot, even when warned.
> Trauer is a feminine noun. Remember that, in German, both the spelling of the word and the article preceding the word can change depending on whether it is in the nominative, accusative, genitive, or dative case.[1]
[1]: https://www.collinsdictionary.com/dictionary/german-english/...
A good AI should be smart enough to know that if you ask for "German words with the article 'der'", you're most likely to want to be given a list of masculine nouns.
d) GPT4 is significantly worse in german and most other languages, than in english anyways.
(I use normal machine translation API for a lot of this, but you can also ask it in another context window to translate the text to other languages. I use this approach for e.g. sindarian)
Otherwise my current strategy is to put it in an analysis loop until it deems the list to be correct.
Why did you think that? This isn't meant to be critical, but I'm honestly curious, what led you to believe that technology underlying GPT-4 made it a good fit for this or any particular task?
It has probably seen the correct nouns used millions of times in the training data - and asking it to produce the correct nouns for a bunch of words is really just "tell me which case you saw most during training", which is something LLM's are really good at.
It is a purely statistical model. It does not know any "rules" about the language (it doesn't know any language at all), but it is fed data and derives from that sophisticated probabalistical relationships between words.
It shouldn't have much of a problem to generate the correct grammatical formulations, as it has been extensively trained on them. Moreso than any other technology neural networks are suited for this kind of tasks where hard rules do not exist (as a German I couldn't tell you why "rain" is masculine, but "machine" is feminine), but lots of data correctly implements the rule, does.
(https://github.com/openai/gpt-3/blob/master/dataset_statisti...)
Also tbh, with all the hype of LLMs one would think that such a task would not be such a challenge.
The strange/sad thing is that despite being "large language models" they're often hypermyopic on English..
I've done some measurements comparing generation between various languages in the prompt and no matter what I do half the time i cannot get them to not include english text or comments in code unless the request is made in japanese, chinese, or a similar very-different-language.
If you train and evaluate something mostly on a language loke English, you're going to end up with a model that thinks everything works like English, which means among other things very little morphological complexity.