The hilarious bit is that they dropped Java for Python, probably due to the massive levels of frustration encountered when trying to do simple things like this.
I know you saw the title and thought "ahahaha easy time to bash Java again", but it's really ignorant to do so.
"Most of Google is implemented in C or C++ for efficiency and can run in either Solaris or Linux."
and also
"Both the URLserver and the crawlers are implemented in Python."
So at some point they decided to try out Python for certain tasks.
connection.setRequestProperty ("User-agent", "GoogleBot/0.01");
I had never done this before and found it out by looking at the javadoc. I don't see how switching to Python would have made this easier.
Wikipedia's history of Java verions says 1.0, as does the request string in the article. [http://en.wikipedia.org/wiki/Java_version_history]
Was URLConnection available back then?
According to the URLConnection docs it's been around since JDK 1.0. [http://java.sun.com/j2se/1.3/docs/api/java/net/URLConnection...]
Did URLConnection.SetRequestProperty exist back in JDK 1.0?
The closest I could find were the docs for JDK 1.1.8 in a downloadable zip file, and yes SetRequestProperty existed back in JDK 1.1.8 at least.
Looking at the actual response, and the JDK 1.1.8 docs, he would probably have been using HTTPURLConnection (could not find HttpClient anywhere in the jdk1.1.8 docs) and even HTTPURLConnection in JDK 1.1.8 I could not find the string 'agent' anywhere on the page.
So yea, if the settings were there they were buried and not readily accessible in the documentation of the time.
setRequestProperty is there.