Human or Machine: Can You Tell Who Wrote These Poems? | Hacker News Reader

Human or Machine: Can You Tell Who Wrote These Poems?

npr.org

66 pointsevo_959 comments

6/6, and I didn't read more than half-way through most of these. What's the tell? Focus. The human-written poetry develops a single, coherent subject with each passing line. Each new line adds new light to the ones that preceded it. Any single line of the machine written sonnets, considered alone, might appear to be written by a human but, when taken together, they simply don't add up. The progression of a single idea just isn't there.

I freakin' hate poetry. Give me prose or give me epigrams! To me, a poem is an awkward combination of prose without clarity and and epigram too bloated to be stated succinctly. I point this out, because I want to be taken seriously when I say that this machine-written poetry is bad in a way that makes me appreciate human poetry just a little bit more.

Same here! 6/6 within five minutes. The computer ones are just laughable gibberish, reminiscent of word salad generation from some 1993 dated Perl tutorial.

> The human-written poetry develops a single, coherent subject with each passing line

Yep, thats it. The machine-written poems make seemingly correct sentences, but fail to create a coherent sense. Its quite easy to spot. The machine-written parts use: The Pythonic Poet (http://ischoolreview.com/iSR_Grav/entries/entry-6). It works with markov-chains and support-vector machines.

Same, it took me just a few minutes. The human writers all have a point, whereas the machine writers (although they sometimes do have some beautiful imagery) obviously lack a plot or overall structure. It's just words without purpose.

Not all the computer generated ones seemed human even on the level of the individual lines. I read all but one all the way through just in case something at the end managed to bundle the chaos into something coherent, but Sonnet #5 has this line:

then comes a small bright spark comes wandering

There's no way any human would have written a line like that, it looks the kind of error you get when you phrase a sentence one way, then rephrase it but fail to fully delete the existing words. The two occurances of "comes" stick out to any human when read aloud.

Poetry that scans correctly should be fairly easy to write for a machine - just find words with the right rhythm patterns and rhymes. Poetry that connects on an emotional level must be a lot harder.

I was tricked by "A thousand pictures on the kitchen floor, Talked about a hundred years or more."

I got 6/6

I didn't even read that far. It was obvious by here:

SONNET #2

    The dirty rusty wooden dresser drawer.
    A couple million people wearing drawers,
    Or looking through a lonely oven door,
    Flowers covered under marble floors.

This is exactly how a computer algorithm would conflate drawer with drawers. There is no obvious theme to this verse, no high level construct.

Every human poem in this example has a high level theme, which is non-obvious to a computer. None of the computer ones do.

That's how I got 6/6 as well. It's less about the actual words and more about if the poem was actually talking about anything coherent.

All the computer poems were just rambling and rhyming and weren't really even trying to say anything. A lot of the time they phrased things VERY awkwardly too, to the point of being incomprehensible.

Sounds like rap music :)

Also interesting to compare with less thematically structured work. Probably not the greatest example but what comes to my mind is some of The Mars Volta's lyrics; a lot of it is pretty abstract and stream-of-consciousness and it's hard to really pinpoint a narrative as such, compared to the example poems which were pretty straightforward

    "Last night I heard lepers
    flinch like birth defects
    its musk was fecal in origin
    as the words dribbled off of its chin
    it said I'm lost
    I'm lost
    
    Now I'm lost
    
    Dolls wreck the minced meat of pupils
    cast in oblong arms length
    the hooks have been picking their scabs
    where wolves hide in the company of men
    it said I'm lost
    I'm lost"

Yep, the human poems would often refer back to an idea or word mentioned before. The computer poems read like something from a good markov generator.

Actually, you can kind of reverse engineer the algorithm behind Sonnet #2. It seems like they first pick a bunch of related words that rhyme (drawer, door, floors, apartment, wall) and then build up the line backwards. The problem is that even though the words are related, the actual lines don't form any single cohesive image.

>This is exactly how a computer algorithm would conflate drawer with drawers.

Because human poets never do wordplay like that?

Because when humans do, there is an obvious (sometimes less obvious, but present) reason to do so.

There is no such link made in this example between the usages.

You haven't read a lot of schoolkid poetry posted at public libraries, have you?

I'm not sure that the majority of that really qualifies as poetry, at least not in a man vs. machine context.

That's what I found. All the "human" ones were easy, but then you had to play "machine or awful poet?" on the rest.

I answered 'human' for drawer/drawers, because "surely an algorithm would be designed to avoid this kind of thing? surely it'd be better than that... must just be a bad poet"

Are you implying that awful poets are in fact not human?

I believe I could make a solid argument for that. :P

But no, it's just that all the bad poets in the original article were machines in the end.

I got this one wrong. I thought no way a computer would be so dumb to have drawer and then drawers in the next sentence and this was a trick question to get one to say machine. Guess I overestimated the algorithm !

Inability to differentiate homonyms is one of the current weaknesses of word embeddings, isn't it?

But like most art, whether it connects on an emotional level is mostly dependent on the consumer. Once an artist gains fame, then it becomes a matter of fashion signalling to agree with others about how good that artist is or to deride that artist for making something that doesn't live up to previous works.

I found a lot of the lines from The Policeman's Beard is Half-Constructed by the program Racter to connect with me on an emotional level, despite knowing that (a) it was likely heavily edited by the program creator and (b) much of it just mimicked the unusual word choices and rhythmic qualities of poetry, not the intentional conveyance of a message or theme.

That's the interesting thing about looking through the output of any computer algorithm. I have often done this with random walks using markov chains.

By editing 200 lines into just 3 or 4, you are actually adding a lot of information. It makes the computer seem smarter than it is.

If we could create a "good poem" evaluator + editor and tack it on the end of a "random poem" creator, we could perhaps get some good results.

>But like most art, whether it connects on an emotional level is mostly dependent on the consumer.

Consumers. The best art affects a lot of people. And they don't need to be told to like it. It "just works."

Art audiences follow multivariate statistical distributions. It's very much not just a personal reaction - it's a personal reaction shared by many other people having broadly similar reactions.

Some of it is social signalling, and some of it requires education and experience.

But valued works tend to persist in culture because they reliably create reactions in a statistically significant proportion of all possible audiences.

Right, because you have to convince the high-status authorities to say that it connects on an emotional level so that everyone else will nod their heads to avoid looking stupid.

Truly, AI is a long way from beating the Emperor's New Clothes game.

But frankly, there are easier, more useful AI goals to target.

I'm guessing you don't like poetry.

>Right, because you have to convince the high-status authorities to say that it connects on an emotional level so that everyone else will nod their heads to avoid looking stupid

Honest question -- did you get the same emotional read off of the machine-poems as you did the human-poems? If not, perhaps there is something more involved in poetry than stringing words together!

>Honest question -- did you get the same emotional read off of the machine-poems as you did the human-poems?

Yep -- none.

Hmm, well I was able to correctly identify every single sonnet. The machine generated ones I would read and think "What the heck was that poem about? It just rambled on about random things" whereas the human ones you could tell that all the verses were interconnected.

Take Sonnet #6 for example. You can tell there is intelligence behind the sentences being written. Each line builds on the line before it. Compare to sonnet #5 which rambles on about nothing and uses duplicate words (then comes a small bright spark comes)...

You've never seen a human poem that just seemed to ramble about nothing?

Bad ones, sure. But I got all 6 poems right in that case as well. I had to read them carefully and think about it, but I still got them all.

In this particular case I think it was fairly obvious, not so much because of the meaningfulness of the human-written poems (though the poems that were written by humans did clearly have meaning associated with them), but more because the machine-written poems were very awkward.

Sonnet 1 was about a person mourning their mother who died in a car accident. It read well and made sense, so I would have been very surprised if it were written by a machine.

Sonnet 2 didn't make sense, but more than that there were very awkward constructs. Rhyming "drawer" (the furniture) with "door" and "drawers" (the undergarments) with "floors" in the same quatrain reads very poorly. "Sergeant" does not rhyme with "apartment".

Sonnet 3 was about something (a growing plant), and it reads well, so it was written by a human.

Sonnet 4 was about something (Joy, a personification of the emotion), and it read well. "Joy" being consistently capitalized throughout was a subtlety that convinced me it was written by a human.

Sonnet 5 had more awkward constructs. "Shame" and "Tame", "Inflame" and "Came" in the same stanza. "Bring" and "wandering" don't really rhyme. The last stanza is just weird -- "hear" is supposed to rhyme with "cheer", "cheers" (which is the line right before "cheer"...) is supposed to rhyme with "roars" (?), and "nights" is supposed to rhyme with "respects"? Again, it didn't really make any sense.

Sonnet 6 was similar to Sonnet 3. It's about something (a growing plant again), and it reads well again.

From my perspective, sonnet #6 was easily the best technically. That doesn't mean I connected with it emotionally, but someone could.

Generally, there's a poem (more than one even) to fit any person's taste. patio11 has quoted one before, which is meaningful to him:

    I went to find the pot of gold
    that's waiting where the rainbow ends
    I searched and searched and searched and searched
    and searched and searched, and then
    there it was, deep in the grass
    under an old and twisty bough
    It's mine, it's mine, it's mine at last
    What do I search for now?

Rudyard Kipling wrote a number of poems about pragmatism, among other things; here's a verse of Dane-geld:

    It is wrong to put temptation in the path of any nation
    for fear they should succumb and go astray
    So when you are requested to pay up or be molested
    you will find it better policy to say:
    We never pay anyone Dane-geld, no matter how trifling the cost
    For the end of that game is oppression and shame
    and the nation that plays it is lost!

Here's a verse of Rising of the Moon that appeals to the same (very popular!) basic themes as jingoism does:

    All along that singing river, that black mass of men was seen
    High above their shining weapons flew their own beloved green
    Death to every foe and traitor, whistle out the marching tune
    And our army buys her freedom at the rising of the moon

All of those are high-quality poems.

The problem comes in with an upper-class movement to distinguish themselves from the rubes who appreciate good poetry by removing all the things that make it good. The problem SilasX identifies is real, and you can earn a lot of acclaim today writing awful, rhythmless, rhymeless, themeless stuff and calling it poetry. Compounding the problem, this terrible high-prestige form is what you'll learn about if you decide to take a poetry class. But we shouldn't let it ruin our view of what poetry can be.

> this terrible high-prestige form is what you'll learn about if you decide to take a poetry class

While there are classes in post-modern poetry, most poetry classes are not that.

How many poetry classes have you taken?

Fair enough. My experience with poetry classes is limited to elementary school and middle school. Our middle school curriculum consisted of badmouthing poetry that somebody somewhere might have actually liked. The school was proud of having classes taught by a published poet.

It's entirely possible that you were taught by a pseud. That's unfortunate if so.

I did a degree in English lit. The poetry classes I took looked something like this:

- Old and Middle English Poetry (Beowulf, Chaucer, etc.)

- Elizabethan Poetry

- Paradise Lost

- The Shakespearean Sonnet

- 18th Century Classicism (Alexander Pope, etc.)

- Victorian Poetry (Tennyson, Matthew Arnold, etc.)

- 20th Century Irish Poetry (Yeats, that kind of thing)

- 20th Century American Modernism

Only in the last class was there was anything like you described, and even then it only made up a fraction of the material covered.

Here is one of the most critically acclaimed poems of the last 40 years. It's perfectly easy to understand. http://www.poetrybyheart.org.uk/poems/a-martian-sends-a-post...

Here I was, writing a draft response about this personally unprecedented phenomenon of someone who lacked any sense of poetics at all, but I quickly realized that isn't necessarily the case. No one has asked you the right question -- could you tell the poems apart? I bet you could.

Did you get them?

Did you recognise, for instance, that the first poem is the poet's recollection of cancer ravaging her mother, and of how she walked to the hospital every day to see her?

Did you understand and just not care, or did you not understand and so find nothing to care about?

I'm not a high-status authority and I was still able to connect emotionally to the human ones. Or, at the very least, each line contributed to a consistent mental picture or idea.

I found an overarching theme or message above and beyond the words in the human written ones (guess 5 of 6 correctly). This isn't just a theme tying together the lines, it's more of an arc of understanding.

Of course, I did incorrectly guess that one of the poems was written by a machine, when it was a human.

>Poetry that scans correctly should be fairly easy to write for a machine - just find words with the right rhythm patterns and rhymes

Indeed, Pentametron [0] ('With algorithms subtle and discreet/I seek iambic writings to retweet') has been up since 2012. Despite the occasional serendipitous pairing, it obviously doesn't make much sense, as didn't the obvious machine-poems in TFA.

[0]https://twitter.com/pentametron

Just to point out, human poems, and in fact the poems here, often don't scan or rhyme correctly.

This is another variation of the Turing test. Time to realize: the Turing test is useless because: (a)the goal is to game the jury and (b) humans don't necessarily make sense either (example: bad or even just confusing poetry) so the human baseline is not always well-defined. Need Turing Test 2.0

I found a website once advertising "have a conversation with me or my chatbot, and try to guess who it is".

My counterparty quickly made some bizarre grammatical errors, prompting me to guess "chatbot" and be rewarded with a screen saying "No, you fool! See, telling the difference is harder than you think!"

Telling the difference between a robot and a human pretending to be that robot is apparently viewed as a Turing test, but it isn't actually the same thing.

Similarly, there's a Turing test that's run every year with very strict rules on what you can and can't say to the other side. This defeats the idea of the test (while making the event less nonsensical, in that the rules allow for people to misidentify chatbots some of the time).

I guess my point is, your point (a) is an artifact of a certain culture that's trying to appropriate the concept of a Turing test so that they can declare they've beaten it. Point (b) is better.

Turing Test is a personal experience. For example, I will only believe TT has been passed if I personally had a chance to talk to the "Entity Under Test", and I can't tell it's human or not. I don't really care whether other people have been fooled or not.

Obviously, the EUT should be, or pretend to be, my intellectual peer (e.g. similar education, age, social status, country, etc). It should be a native English speaker, or at least as fluent as I am. I should be able to converse with EUT for as long as I need to (several conversations if needed, perhaps limited to a couple of weeks).

Most importantly, the human participants should be motivated to convince me they are human (e.g. they only get paid if they manage to do so).

A Turing test is meant to be both the AI and the human trying to convince the tester that they're human. To do it properly your human testee should be a third party not related to the AI developer, and should be independently motivated to 'win' by convincing the tester that they're the human.

Having the human pretend to be a chatbot is just disingenuous.

As others have mentioned, lack of an overall theme is an easy way to distinguish the human from machine poetry.

Keep in mind, the machine poems are those hand selected by humans, twice filtered -- this publication and the original publication. That means these poems have a high degree of coherency that you wouldn't otherwise see.

People are reporting 6/6 ability to distinguish. If you had a person and an algorithm each generate two (or even one) poem without any filtration from a field of poems then it would be even more obvious which was machine and which was human.

Unfortunately the problem reduces to qualia. Poems that have a theme evoke the experience of that theme. Likewise for generative music and art.

If you have no model for experience (it doesn't have to be metaphysical - something mechanical would do) you can't include that element in the output.

Except by accident - which is how Deep Dream works. The designers found a model that was good at mimicking a certain kind of experience - partly through luck, and partly through analogy.

But Deep Dream is limited to that one kind of trippy experience. It's very good at it, but a competent human artist has a much wider range. [1]

Real AI creativity needs a system that can mimic that much wider range - so you'll never be sure what kind of human experience you'll get from it.

[1] Sometimes. A surprising proportion of professional artists repeat the same theme, style, or imagery over and over. But that's sometimes because of commercial pressure - when you find a formula that sells, experimenting with a formula that might not isn't always wise.

Poetry is the literary equivalent of minimalist painting. The machine is doing just enough, or as little as possible to tick the boxes needed to make the art identifiable as art. But art only exists in context. These poems' only value is that they have come from a machine, just as many paintings derive most of their value from the person who paints them. They are interesting, but as art they are next to worthless because the process of their creation is so easily replicated.

This is art: https://en.wikipedia.org/wiki/File:IKB_191.jpg

I find poems given at http://botpoet.com by bots to be a bit better. Above one are completely incoherent. Haiku would be probably more easier to generate.

If a machine writes 1000 poems and a human chooses the best one did the machine make a poem with emotion or a human?

I see a future where as the "rise of the machines" takes hold in things like News Items, Stories, Poetry, Paintings, Movies etc, there will be a commensurate rise in Detection software. The detection software would flag things as human-generated vs machine-generated.

Pretty soon anything created by Humans will be intrinsically more valuable just because it was human-created and hence somehow better than "machine-generated".

We live in interesting times...

I read only the ending rhyme of every sonnet => 6/6.

I really doubt if you only read this sentence (last sentence of sonnet #2) you'd know it's a machine:

    A thousand pictures on the kitchen floor,
    Talked about a hundred years or more.

If you don't know what's written in the preceding sections, this could very well be written by a human.

Oh you're right,

I read upwards at that poem and looked at the ending words of the sentences. Decided that the combination "tripping" / "dripping" wasn't very poetic. -> machine.

Maybe we should have a machine learning algorithm do this ? :)

But if there aren't human examples that machine couldn't write poems. They are just mimicking within the samples and structures.

Reply on news.ycombinator.com