Llama-3 70B on my local MacBook works wonderfully for these tasks.
How's the Pipeline? Do you pass all the html to the LLM? Isn't the context window a problem?
There are phenomenal web scraping tools to crudely "preprocess" the document a bit, slashing outer HTML fluff while preserving the small subset of actual data. From there, 8k tokens (or whatever) goes really far.
At a very generous 50 tokens per second doesn't that still leave you with more than two and a half minutes (160s) processing time per document?
GPT-3.5/GPT-4 ain't the only LLMs available. A Flan-T5/T5 or Llama2/3 8B models may be finetuning for this use case and used for much cheaper.
How do you handle the context window limit? If you push the entire Dom to the LLM it will exceed the context window by far in most cases, isn't it?
My guess is you do some preprocessing on the DOM to get it down to text but still retains some structure.