I guess beauty is in the eye of the beholder. A simpler way to analyze this problem is noticing the f(n-3) term implies your function has to memorize up to 3 previous results. Then just use the coefficients from the formula to cycle the next result into memory. Using algebra and generating new coefficients as per the OP's solution is unnecessary.
function f(n)
if n<3 then return n end
local a, b, c = 0, 1, 2
for i=3,n do
a, b, c = b, c, c+2*b+3*a
end
return c
end