If you are into java then I will suggest webharvest...if you want something broader(read generic) and brave enough then you can even try nutch.