A gopher client would be easier and no AI needed as the protocol it's trivial to parse.
You have sites from gopher://magical.fish to gopher://floodgap.org and gopher://sdf.org
(and, as a plus, you can read finger services).
Using gopher client to load http data.