Comment by _delirium | Hacker News Reader

Comment by _delirium | Hacker News Reader

Parent Full thread

Yeah, I could believe it "aces" homework in a really open-ended writing assignment. An assignment like: write an essay explaining a personal experience and what it meant to you. The people that's the biggest issue for at the moment are probably writing instructors, since the goal of those classes is to just practice writing something/anything. In computer science though, the writing I've had turned in in my classes that I suspect is LLM-generated usually gets an F. It tends to just ramble about the subject in general and not hit any of the specific points that I'm asking for.

Last year I had a take-home exam in an operating systems class that I suspect one student fed entirely as prompts to an LLM, and it was... odd. The answer to every question was a paragraph or two of text, even in cases where the expected answer was true/false, or a number. And even when I did want text as the answer, it was way off, e.g. in one I asked them to explain one strength and one weakness of a specific scheduling algorithm on a given scenario. The submitted answer was just general rambling about scheduling algorithms. Some of this is probably within the reach of an expert using clever prompting strategies, but students who can do that could probably also answer the original question. :-)

To be fair, I have seen the "ramble generically on the subject of the question" strategy manually implemented by humans too, in the hopes that if you throw enough BS at the question you might get partial credit by luck. Maybe designing assessments to be LLM-resistant will have the nice side benefit of reducing the viability of BSing as a strategy.

I used to have students who would write answers like that on in-class exams.

Every answer was at least one full, complete sentence, even for yes/no or true/false. And the “short answer” responses filled all available space when one sentence would do.

My only conclusion is that some undergraduate institutions around the world must be intentionally drilling it into their students to do this.

I suspect it starts in high school. A lot of AP subjects with written portions like AP biology or history are really hard to grade at scale so they have a relatively naive scoring system. The answer can be a total rambling mess but as long as the answer is self consistent (it doesn’t contradict itself) it gets points for any relevant information it gets right.

For example, if the question is about respiration a rambling answer that mentions “oxygen transport chain”, “Krebs cycle”, and “ATP” might get 3/5 points even if it doesn’t make much sense otherwise as long as the answer doesn’t confuse the Calvin and Krebs cycle or otherwise contradict like saying that glucose is a byproduct.

I was told by multiple teachers/professors that its never acceptable to write anything other than a full sentence on a test (unless it's a scantron, obviously). Not sure how common this is, but they could have been trained by other instructors.

I think students also believe they can hedge. If they just put down "yes" or "no" then their answer might be completely wrong, but if they drop a bunch of things in the answer then some of those things might be true and you might give partial credit, or, at least, they can argue about it later.

In the entirety of my K12 education any answer that wasn't a full complete sentence was a zero.

Its possible. I've had proffesors who always gave true/false questions with instructions to either "justify your answer", or "if false, justify your answer".

Practically speaking, there is fairly little downside to putting in extra in your answer, as tests are normally scored by how many points in the grading rubric you hit.

> To be fair, I have seen the "ramble generically on the subject of the question" strategy manually implemented by humans too, in the hopes that if you throw enough BS at the question you might get partial credit by luck.

This is the basic speech strategy of politicians. Don't answer the question asked, just talk about something related that you want to talk about.

I don't think it'd do well even for an open-ended assignment. The best language models I've seen are still easy to detect as bots if you read multiple paragraphs of output.

> To be fair, I have seen the "ramble generically on the subject of the question" strategy manually implemented by humans too, in the hopes that if you throw enough BS at the question you might get partial credit by luck.

I had a college professor that knew to recognize this and actively warned against it during the mid-term and final.

He said that every question will be answerable in 2 or 3 sentences, and that if you write 2-3 paragraphs instead, he would mark you down even if the answer was correct because you're wasting his time and may have dropped in correct statements that answered the question by luck.

So often in school, we'd be getting quizzes/tests back, and I'd peek over at someone else's paper as it was being handed back and notice they wrote an entire paragraph to answer, whereas I answered it in a single sentence and got full credit for a correct answer, and I was always left wondering what the hell they wrote about.

Reply on news.ycombinator.com