ESR announces ForgePlucker: solving the data-jail problems of OSS hosting sites

Your reply doesn't address why you do the things I highlighted in my original comment. For example, in the middle of your generic table parser you have code specific to Berlios making it non-generic. That's there because you haven't actually parsed the HTML you are trying to hack around it with regexps.

In your post you say:

So instead, I walk through the page looking for anything that looks like a hotlink wrapping literal text of the form #[0-9]. Actually, that oversimplifies; I look for a hotlink with a specific pattern of URL in a hotlink that I know points into the site bugtracker.

I don't have any problem with you using regexps to identify the particular fragments you are looking for, my criticism is that you use them for HTML parsing. For example, in your code you use a regexp like this

  <A HREF="/bugs/\?func=detailbug&bug_id=([0-9]+)&group_id=%s">

This mixes structure and meaning to use your terms. You assume that there's an HREF attribute after the A. Your code is brittle when it comes to a change in the HTML (e.g. suppose the page author adds a CLASS= attribute).

This would not happen if you parsed the HTML into a DOM tree and then did queries against it. You could quickly extract all the <A> tags in the path with a //A query (or even all those that have an HREF) and get the actual HREF robustly. Or you could not use XPath and use a parser that does calls backs with robust lists of attributes.

Doing that would be both robust against changes in page structure, and robust against changes in the attributes or placement of attributes in the <A>.

PS Is your name calling really necessary? In your post you refer to me as a 'snotty kid' and a 'fool'

PPS Also you say:

If I did what my unwise critic says is right, I’d take a parse tree of the entire page and then write a path query down to the table cell and the embedded link.

I never said that, I said that you could use XPath queries to extract the data. I never implied that you use a complete rooted path to get where you want in the document. You've twisted what I actually said to fit into your 'structure and meaning' blog post.

And that response isn't very conducive to creating a community of people willing to help you on your open source project.

Too many egos getting in the way here guys. This is an all right project. Don't let disagreements on how to do things make it all fail. There are always better ways to do things and there are prototypes that do a lot of wrong things.

Now's the time to make the bad parts good. The code was just released, okay, now make it right.

He's certainly a talented self-publicist, but I'd go no further than that. He's the Michael Moore of software. Or the Paris Hilton.

Which large scale OSS projects?