Monitor web page changes with Go | Hacker News Reader

Monitor web page changes with Go

silviosimunic.com

93 pointsssimunic43 comments

Open articleView on HN

When making HTTP requests with Go (or any language) you should at least consider the following:

- Setting a User Agent (req.Header.Set(Key, Value))

- Setting Timeouts (https://blog.cloudflare.com/the-complete-guide-to-golang-net...)

- Avoiding ioutil.ReadAll() and checking body length (or using https://golang.org/pkg/io/#LimitReader)

- Caching DNS lookups (https://github.com/viki-org/dnscache)

+ respecting expires or cache-control headers

Quick question: Since Go's html parsing requires utf-8 encoding is there a way to convert the response to utf-8 without ioutil.ReadAll()? (Still pretty new to Go).

html.Parse takes a io.Reader https://github.com/golang/net/blob/master/html/parse.go#L201...

an http Response.Body is a io.ReadCloser so you can pass it directly with `html.Parse(response.Body)`

But don't forget to defer io.Close the body.

I guess I wasn't specific enough. I'd like to stream the response so as to use less memory. I also need to ensure the body is utf-8 encoded because that makes Go happy.

https://mionskowski.pl/html-golang-stream-processing

If only there were libraries that encapsulate best-practice plumbing and glue logic and offer a high-level API with sane defaults!

A perusal of the awesome-go list [1] (allegedly, a "curated list", which may merit a debate) names a few:

https://github.com/parnurzeal/gorequest

https://github.com/smallnest/goreq

https://github.com/franela/goreq

https://github.com/sethgrid/pester

https://github.com/mozillazg/request

https://github.com/levigross/grequests

https://github.com/go-resty/resty

https://github.com/dghubble/sling

[1] https://github.com/avelino/awesome-go

I would also add retries to the list (https://github.com/sethgrid/pester)

Thank you very much for your comment - I've realized that caching DNS should be considered in any "connected" service.

I would add my two cents:

- Calculate checksum and compare it with the latest entry (storing single 32/40-bytes long checksum takes lower amount of memory).

- Respect various HTTP headers like Expires, Cache-control, Accept-Encoding (accepting compressed content may relieve strain on bandwidth), Content-length, Cookie-length etc.

- If you were downloading a web-page and sending it via e-mail (for later usage?) you may gzip it - it may reduce costs of attachments or S3.

- Attach header "User-Agent" and "From" (with an e-mail address to contact developer) - if you don't mind.

- As you wrote, avoiding ioutil.ReadAll should be a good practice - Datadog published a blogpost which is worth reading - https://www.datadoghq.com/blog/crossing-streams-love-letter-...

- Use logrotate or implement it on your own with i.e. Mutex. Eventually you use fluentd, Kafka, Pub/Sub pipeline or some SaaS offer specialized in storing logs.

Most (or all) of Go packages with "request" in the name that I know doesn't cover aforementioned scenarios automatically - it is up to developer.

Do you mean

> Avoiding ioutil.ReadAll

AND

> checking body length

or do you mean,

> Avoid ioutil.ReadAll()

> and [instead you should be] checking the body length

Do you have an example of that, or an example of LimitReader?

You made me curious, so I've prepared a simple benchmark - https://gist.github.com/piotrkubisa/8ccc308086378da7d2f6c0d5...

That's super awesome, thank you

Wouldn't it be more unixy to run a caching DNS resolver such as nscd locally on the machine, instead of building it into every app? (I would accept an answer such as "there's no caching resolver out there that properly handles thundering herd situations".)

To avoid accidentally fetching a huge resource into RAM and exploding in a rain of fire, automatically spooling large bodies to disk is a good idea (as opposed to relying on swap, which is rarely a good idea). One could use a fixed threshold to avoid spooling smaller files.

>we will need to define two global variables

Or just pass them to the request functions. And why define the global LOOP_EVERY_SECONDS in terms of seconds, rather than a time.Duration?

This code also doesn't handle errors at all. And it can exit any time from within checkChanges. And it compares data by converting the []byte to a string. Use bytes.Equal instead.

Finally, wouldn't this be a bit more useful if it printed a diff of the old-vs-new?

"wouldn't this be a bit more useful if it printed a diff of the old-vs-new?"

I like the idea. If I had to do this quickly, I'd be tempted to use cron, diff and wget as a script. I can't see any handling of HTTP response codes [0] so lots of things could happen that need handling. Does Go have equivalent to the fabulous python ^Requests^ by Kenneth Reitz? [1] Page diffs is a nice feature I might add to a page size tool I hacked up. [2]

Reference

[0] https://tools.ietf.org/html/rfc2616#section-6

[1] https://github.com/kennethreitz/requests

[2] https://github.com/peterrenshaw/pageglut

Handling errors is 90% of the job for a page monitor like this. Have fun getting spammed when the site goes offline briefly, or when your internet connection goes down.

Good point. Added error checking.

This is an interesting submission for educational purposes but I want to warn folks who might be inspired to "roll their own" website change monitoring tool. I started writing a similar tool in C# and since I was hoping to create a generalized tool for any website, I had to deal with lots of tricky edge cases. (E.g. raw HTML that changes vs visible text that doesn't, copyright dates on footers that get changed but the real meat of the page is still exactly the same triggering false alarms, etc.)

I gave up and simply used Aignes Website-Watcher.[1] If you're still itching to program your own webpage change detector, I recommend reading through their 16 years of release history[2] to get an idea of the various bugs they had to solve. The scope of edge cases can be overwhelming if you want to write something very robust.

[1] http://aignes.com/index.htm

[2] http://aignes.com/wsw_history.htm

+1 Read the release logs of existing projects. I wish more projects actually took the time to talk about the problems they solved since those issues usually outlive their projects as new alternatives pop up.

How would one use this with pages that have dynamic ads on them?

Every production site should have content (at least for any page that gets more than 5% total traffic), uptime / load time, application performance, server health, and security monitoring.

Can't count the number of times content monitoring bailed me out... a CMS change went live to early, or someone released the wrong branch to prod, or a DNS subscription I didn't control expired... I usually just string-check the Title or H1 on the page but would be great to have more advanced tools.

I have always wanted a program/script to check for changes in the NIH era commons site, specifically a page that indicates for a particular grant application, what its score is. Problem is, to get to this magic page (which is, I think, dynamically created), you have to:

(1) log into the main era commons site with a username and pw

(2) then a new page opens up, on which you have to click on a particular link to get a list of your grants

(3) then click on the grant of interest

(4) then click on the competition date

(5) then the magic page opens up. It's this page I want to check for changes

I've done the requisite google searching and found lots of generics suggestions about cookies, etc, but I don't have the skills, obviously.

Is there a tool whereby I can click "record" and go through my various steps, and then the tool will keep track of the programmatic aspects of this surf-and-click dance, such that I can execute that script later (e.g. as a cron job)?

Thanks!

An incredibly crude solution: http://sikulix.com/

I like the idea of something that records your actions and creates a script, but for your particular case this would be super easy to throw together with Python and the Requests library. You just create a Session object and tell it to perform the series of posts and gets that take you to the desired page, and then search the text of that page for whatever info you want.

Another option could be to use mechanize (https://pypi.python.org/pypi/mechanize). Originally written in Perl, then ported to Python as well as Ruby.

Got it done with Python/Mechanize! Thanks for the suggestion!

are you a researcher? PM and I'll do it for you. I did this ever day all day for the last 6 months so it won't take me more than an hour.

thanks for this!

I think though I'll try Python/Mechanize first.

Also I'll try not being so neurotic about it and just wait like a human being for the email to be sent to me :)

A cryptographic hash is overkill, murmur or cityhash would be much more appopriate.Furtheron, it might make sense when using one of the mentioned hashes to accept a minimum of change, like a timestamp which has changed on the site , but that varies of course on the use case.

This will break in a lot of pages as the html will be different on each request in most pages although the content will be the same. Perhaps it would be better to compare visible text rather than the raw html.

Neat. Do you have a recommendation for a headless browser library that would simplify this? The difference between checking the raw HTML and visible HTML can be pretty big.

In client-side Javascript, document.documentElement.innerText yields only the rendered text, without markup, CSS or JavaScript source.

But the theory was that the source would be different, and only the rendered content would be the same at the sample place, right? How would you find the 'same place' without fully rendering the page?

One way is for the source to use cache control headers so you can just do a quick HEAD instead of a GET on the whole contents.

lxml provides a module to perform html diffs mainly looking at the content rather than markup. It's not perfect but most of the times works. http://lxml.de/api/lxml.html.diff-module.html

For Go I don't know any.

When I was working on a project like this another issue that I found regularly is that there are regions of the page(counters, advertising, etc.) that change all the time too but they are part of the content. For this I would make several requests to the page and find the regions that would change in every request and disregard them when doing the diff.

Does anyone have insights on how to refine this to only monitor specific elements of the page rather than the entire page?

I may not care if they change a template but would care if they change a price or a warrant canary etc?

You could use an XPath or similar to extract the part of the HTML you care about.

Similarly with a library like Beautiful Soup you can traverse the HTML a lot like a jQuery selector.

I suspect you'd want to search for a specific attribute or tag combination, I used the scrape package [0] recently and found it a pleasure to use. Beyond the example provided in the README the documentation [1] is also good.

[0] https://github.com/yhat/scrape

[1] https://godoc.org/github.com/yhat/scrape

Tangent: why is the Go mascot always so deformed[1]? It makes me sad because I do love Go but having such an awkward mascot makes me feel like the language is deformed, too (which some people would say is true, I admit)

[1] Yes, I'm familiar with the history of its creation, I just wish it weren't so

Do you judge a product from its logo/mascot ?

And books by their covers!

Your parent clearly mentions it's a tangent, and likely not a big deal. I'll go a bit further and admit that the choice of logo or mascot does indicate something about the project, whether it be a sense of style or respect for tradition or history. It's by no means the most important criteria, but it's a datum nonetheless.

Language creator hairstyles are also important signals, by the way.

Thanks for understanding. I think that "don't judge a book by its cover" isn't an absolute maxim to live by. In fact, yes, I do judge a book by its cover. Not ENTIRELY by its cover, but as you point out, it's not irrelevant.

I think this phrase is so infected in common speech and mind that we don't question it with books, but how many would disagree that with e.g. SaaS, having a nice looking landing/marketing page is important. Again, there are still successful SaaS apps that have awkward marketing or even UIs, as it's not the only determining factor, but it is indeed important.

In this case Go's logo has much less to do with the language than a SaaS app's landing page, but as another comment below shares, it still shows some level of influence in the overall project.

Just a note and reply to the comments made so far:

I agree that there are flaws with this as zlagen said (where the page is different in some other parts of HTML).

I've made this according to my need, and that was to check a page for only one number change, where everything else wasn't changing. This is meant to demonstrate general idea behind making something like this.

nemo1618, I agree that code should handle errors and you should definitely do that. For printing out old vs new version, as I said, for my needs, that wasn't needed since I just wanted to get emailed once one number on page changes.

Reply on news.ycombinator.com