At Feedity (https://feedity.com), we "index" webpages to generate custom feeds. Over the years, we've designed our system to use a mix of technologies like .NET (C#) and node.js, and implemented a bunch of tweaks and optimizations for seamless & scalable access to public content.