Thank you. I have just been working on something a this

> awk '{ if (!($0 in seen)) print $0; seen[$0] = 1; }'

fits right in.