Hacker News Reader

> For example, recently psychologists discovered a simple way to test for creativity. The simple test is to come up with a list of 10 words as different as possible from each other.

When Robert Frost was the poet laureate of the U.S some news magazine (I think NewsWeek, although it might have been Time) did a creativity test on him, using inkblots iirc, and he scored particularly low.

Frost was irate afterwards and remarked something to the affect that to be creative required that you not see things in the inkblot, that is to say to not try to see things in distractions, that he had spent years training himself not to see things in meaningless shapes.

I think that there is quite a lot of truth in this, also if someone wanted to give me a simple test to come up with a list of 10 words as different as possible from each other I would probably get verbally abusive - at least call them an idiot.

> For example, recently psychologists discovered a simple way to test for creativity. The simple test is to come up with a list of 10 words as different as possible from each other. According to the test, GPT-3 is far more creative than the average person, scoring in the 85-95th percentile.

This is the dumbest thing I read all week. My god, how can somebody be so obtuse and so... narrow-minded?

An excellent argument in favour of including some humanities course in tech degrees at least.

> An excellent argument in favour of including some humanities course in tech degrees at least.

What are you seeing that I am not?

It looks to me like the author of the article went to a liberal arts school (Hampshire College) for a BA in cognitive science and philosophy before studying a PhD in neuroscience, and is a published fiction writer (both short-stories and a novel).

I suppose what a tech degree is needs to be defined, but in Canada a computer science degree would involve "breadth electives" that cover humanities and the social sciences. It is at least two of each depending on if one is studying for a BSc or a BA (a BA would require more). Even a degree in computer engineering in Canada would require at least one course in the humanities. I think this applies in the US as well. I believe computer science and computer engineering are the stereotypical tech degrees people picture when the argument is made that tech is lacking in an understanding of the humanities and ethics.

It seems that the author of the article is about as far from a typical tech degree as I've outlined above, but you may be seeing something else that I've missed?

Church turing paradox but in comments ~ ;p - to elaborate

One take is that it’s bollocks- a mechanistic measure for a mechanism to “win” at has nothing to do with human creativity. One would argue that the arts should be taught with stem to help engineers interface with humanity- especially the indefinables.

Another take is to ask what it is the machine isnt quite doing. As i believe its been said before “tell me exactly what the program can’t do and i will make it do exactly that”.

But where does it end? To satisfy the argument we will need to define creativity, humanity, arts, not to mention bicker on the semantics of building the necessary shared ontology!

The synthesist in me thinks the next Shakespeare surely could benefit from n+1 “monkeys and typewriters”

The tasks asked to name 10 nouns which are as "different" from each other as possible. I chose 10 common words which all belonged to different concepts, I scored 75%.

The results then stated that "different" refers to the probability of two words appearing closely to each other in known texts.

With that in mind I retook the test and chose 10 words which are rare or vulgar. Scored 93%.

I got a dual-major in psychology in undergrad, and while it's not been an exceptionally useful degree, it did bequeath me with an enormous dose of skepticism toward psychometrics like this. The replication crisis solidified that skepticism

Being able to choose less "related" words doesn't sound like it has much meaningful basis at all in what we care about when we say "creativity". This metric, as many others like it, was likely chosen and popularized because it is easy to measure

I can see it being strongly correlated, as that human has more items to draw from for figuring out solutions, but I don't see how it can be used to test computer creativity. Python's random surely isn't creative.

That's the problem with metrics though. You can make anything sound plausible. Once a model has been presented to you, you can come up with an endless stream of just-so stories for why it measures what you were told it measures. Nonetheless, some metrics simply aren't that useful, and get adopted anyway because they produce a bunch of data. Psychometrics like this have an awful track record for standing up to more meaningful tests like "Can we drive any outcome we care about with this"

That’s a good way to demonstrate where we are with AI; it can beat a human on any given narrow definition, but still fall far short on any task that requires some ineffable creativity, synthesis, or understanding of the word. We can make a Python script more “creative” than 95% of humans (by a meaninglessly narrow definition of creativity), but all our effort as a society has still not produced a car that can drive better than your average teenager.

Arguably that line of Python wouldn't do very good - you'd want choices that maximize the difference as opposed to be merely something random, this is harder than it appears; and the distribution of words is also uneven enough (both in relative frequency of word usage, and also in lexeme "density" w.r.t how many terms there are for some domains and how few for others) that pure random sampling would be not a good option.

It does pretty well on this (https://www.datcreativity.com/task?), at least with the below code. Consistently getting 80-90%.

https://github.com/MattGaiser/CreativeWordChoice/tree/master...

I just grabbed a random list of nouns, narrowed it by the params specified, and fed them in as required. You may need more than 10 as it rejects some, but in that case I just re-ran it and put in the first nouns to come up.

Word frequencies follow a power law, so nearly all words are used very infrequently. So a randomly chosen word is likely to be very uncommon. Also there are many more specific concepts than general ones (because each general concept encompasses many specific ones). So a randomly chosen word is more likely to be specific than general.

Random sampling is likely to give you unusual, highly specific words. I don't see how that wouldn't do very good. Sure, you could do better by mapping each word to a concept, then selecting ten random concepts and selecting a word for each concept. But why do that when you get 99% of the way there with just random choice.

To show this experientially, I let python choose 10 random words from the first English word list I found. I got "epilithic, codirects, serologic, capsian, digestibility, yasna, samsara, supergun, hepatoptosis, kyurin". Sure, capsian and kyurin are somewhat related (names of places/people), as are serologic and hepatoptosis (both medical terms), but it's a pretty good spread.

Not poster but I think the point is you could do better, not that this doesn’t work.

You can imagine having some discrete optimisation maximising the dissimilarity of words, though that’s hard to do, in particular in a discrete space.

The example you got was the exact thing I had in mind - there are "terminology-heavy" fields with lots of terms, and a random draw is quite likely to draw two items from a single field which IMHO should count as quite related. The automated scoring method they're using seems to say that all obscure words are unrelated to each other, though, but I'd use that as a criticism for that evaluation algorithm not as praise for the evaluated method.

On the other hand, probably there are many humans who would pick two local animals or two office-related objects that happen to be in their room, so that cuts both ways.

Yeah, a random sample won't be (numerically) ideal, but given the size of the set, probably safe.

A really quick and easy way to optimise this would be to just grab a well-trained word embedding space and pick _k_ furthest words by your favourite distance metric.

Also the test I took[0] scores this very poorly, by determining how often those words are in the same section of a text as each other, meaning choosing uncommon words (crenellations, gravedigger) will show much more creativity than common words (castle, grave) that are similar conceptually.

[0]: https://www.datcreativity.com/task?

My naive take is that if you can keep 2N concepts in your head at once, the number of possible connections among them is much more than twice as big than with N concepts, so you have a much wider graph to analyze and possibly fish out interesting ideas from.

(Unless N is 0, of course.)

I agree, but that is a two stage process. Sure the machine can beat us on concepts, but can it make meaningful connections?

I would just like to say, the insight to try just a random choice of words and see how well that satisfies the criteria demonstrates a kind of creativity that isn't captured by having people generate a list of "unrelated" words from memory.

Memory is much more correlated with creativity than one would think.