Ask HN: Can I automate document gathering for research?
I find that I often need to repeat searches on the same individuals at different websites and news archives, only to repeat the entire process again with a new name.
I haven't a programming background, so forgive me if I'm not using the right terms. I speculate that the process can be accomplished automatically by a webcrawler... though I'm not certain if that is the right solution, and I don't have a clear picture of what the risks in attempting to automate this process might be.
At the risk of being long-winded, here's a typical case. I'm given the name of the firm, it's principal management, and maybe a few articles from the company website. My task is to find out if anyone at the company has been accused of a crime, or whether the company has been sued or engaged in fraud. I open tabs in a browser for each of the main websites/databases I will search (chambers of commerce, trade registries, government websites, etc.). Then, one by one, I put each name into each tab and look for results, all the while saving copies of any relevant documents I find. At the end of it all, I piece together the story of the company or person, using the documents as support. This is the mundane part of the job – gathering the documents.
Is there a way to automate that process? Since I'm learning to code anyway, I'd like to try exploring a solution myself, but I'm not certain what language or process would be best for that. I'd love to hear the advice of the HN community.
Cheers, -Sean