That was one of the challenges when building this. We ended up using Firecrawl. But even so there are a few pages we aren't able to read. In which case we use the Algolia API to get the HN comments and generate the video about the HN reaction on the article instead.
Instead of using HN comments which can be a gross distortion of an article, it is better to ask the user to upload a PDF of the full text of the article. It is easy for the user to create such a PDF. Thereafter, subsequent users should have an opportunity to upload a new PDF if they so desire.