I was considering paying someone to build something like this at some point. With two jobs, I eventually had no time to even organize what I find. It's just piles of links in text files.

Can I give your software a huge list of URL's to index? Or do I need to use browser automation to open them a few at a time with it caching and indexing them?