Hacker News Reader

Comment by kazinator | Hacker News Reader

Very confusing presentation here. For one thing, the author seems to be referring to links/anchors in a web page as "requests"!

A request is the dynamic, transient action which occurs when a client such as a browser initiates a connection to a server and presents a command like GET or POST.

I suggest an opening paragraph along the following lines:

"WebXray" is a sort of web crawler which analyzes a given cluster of pages for their relationships with each other, as well as external pages which they link to.

It provides information about pages which direct a user's browser to various sites for the purposes of tracking, using tricks like hidden images.

(Do I have that approximately right?)

View on HN

No, he's not talking about links/anchors. He's talking about requests made by the browser caused by scripts which load content from third-party servers. This sounds similar to Mozilla's Lightbeam https://www.mozilla.org/en-US/lightbeam/

it monitors third-party HTTP requests, not links: "webXray is a tool for detecting third-party HTTP requests on large numbers of web pages and matching them to the companies which receive user data...The core of webXray is a python program which ingests addresses of webpages, passes them to the headless web browser PhantomJS, and parses requests in order to determine those which go to domains which are exogenous to the primary (or first-party) domain of the site. This data is then stored in MySQL for later analysis."

So in other words PhantomJS is used as an API to have some pages crawled, and the links emanating from those pages are captured.

sorry, it doesn't monitor links, it captures network-level requests, that's what I analyze.

I think the requests PhantomJS makes while loading and executing the page are recorded and their URLs stored.

So it wouldn't work if PhantomJS informed about this with, say, a callback function?

I use phantomjs because it can run headless and is low resource. the request data is processed in python though.

Not links. Requests. PhantomJS is a scriptable browser, as a second of googling will tell you.

yes! from the api docs:

onResourceRequested

Introduced: PhantomJS 1.2

This callback is invoked when the page requests a resource. The first argument to the callback is the requestData metadata object. The second argument is the networkRequest object itself.

The requestData metadata object contains these properties:

id : the number of the requested resource method : http method url : the URL of the requested resource time : Date object containing the date of the request headers : list of http headers The networkRequest object contains these functions:

abort() : aborts the current network request. Aborting the current network request will invoke onResourceError callback. changeUrl(newUrl) : changes the current URL of the network request. By calling networkRequest.changeUrl(newUrl), we can change the request url to the new url. This is an excellent and only way to provide alternative implementation of a remote resource. (see Example-2) setHeader(key, value)

Reply on news.ycombinator.com