GPT 4 Vision coupled with Iteration X automatically recommends improvements on HN's live website and adds comments as pins, directly on the html elements.
Neat! Can you describe how the integration works and what APIs you implemented?
As far as APIs we only used the regular chat completion api and promoted the model to output a list of issues with a title and XPath.
The rest is leveraged by our existing chrome extension to capture the screenshot, show the pins, etc.
Are you passing the entire context of the page code to the LLM to get back the XPath?
Yes, the whole document via document.documentElement.outerHTML. Surely there's a lot of optimizations we could look into but this was more to try if the model could localize elements via XPath by using screenshot + HTML. It's doing a decent job for the most part, I definitely have seen some XPath hallucinations though.