an interesting question is how to filter out the folks who just feed it through chatgpt, because obviously the people who just lie to get out of any mild annoyance are the ones you least want as coworkers (though obviously people who can effectively use large language models will be incredibly valuable)
i asked gpt-4 some programming questions today, and although it gave clear and convincing explanations in english, it made a lot of really dumb mistakes in writing the code. but this was in a mode where it couldn't call out to an external interpreter; maybe a lot of those will go away once it can test the code
most programmers would have done worse, but is there a pattern to the dumb mistakes chatgpt makes that is clearly different from the dumb mistakes junior programmers make?
like for example i gave it this
#!/usr/bin/perl
# See also cifs.py. Edited GPT-4 output; see <https://bin.gy/clignitter>
# Try watch -n .1 ./wing.pl.
for(1..9999){$t=0|rand 3;($y,$x)=$t?($y/2,$x/2+($t-1?40:10)):(.3*$x-.4*$y+12,
.4*$x+(.2+abs($$%99/99-.5))*$y);$a[$x][$y]=1}map{$r=$_*2;print map({substr
" .':",2*$_->[$r]+$_->[$r+1],1}@a[0..79]),"\n"}(0..18);
and asked it to translate it to lua, getting the following nonsense in response
math.randomseed(os.time())
local a = {}
for i=1,9999 do
local t = math.floor(math.random() * 3)
local x,y
if t == 0 then
x,y = .3*a[1][2]-.4*a[2][1]+12,
.4*a[1][2]+(.2+math.abs(os.time()%99/99-.5))*a[2][1]
else
x,y = a[1][2]/2, a[2][1]/2+(t-1 and 40 or 10)
end
if not a[x] then a[x] = {} end
a[x][y] = true
end
local str = " .':"
for r=0,18 do
for c=0,79 do
local val = (a[c*2] and (a[c*2][r*2] or 0) or 0) + (a[c*2+1] and
(a[c*2+1][r*2+1] or 0) or 0)
io.write(string.sub(str,val,val))
end
print()
end
of course, i've seen worse from interviewees, and understanding the original program is a high (and arguably counterproductive) bar, but perhaps the particular pattern of bonehead errors here could identify this as llm output? maybe the very fact of being so spectacular at understanding the original code is a red flag, at least combined with being so incompetent at writing the lua version?