Hacker News Reader

When talking about checks on the order of twice a minute, curl is probably the right approach. You can/should still do a full check, but that can be done at a lower frequency.

To be fair, my page load checks aren't just is the site up and responding 200 but also page performance anomaly detection and stuff. So I want/need to see the performance minutely to be able to detectly in a quickish amount of time if the performance has degraded for the entire pageload. You could have a minute where it's slow and that is ok but if you have 5 minutes of it being slower than normal one after another then you have an issue. I feel if you're doing checks every few minutes your data won't be as good as doing it every 30 seconds. It is way more resources but honestly, I think it's the future of monitoring. Also, having multiple types of ways of calling a site via http to monitor it is way more complex.

The main offer is for order monitoring but I am in the middle of creating a just page load monitoring offer for others since I think that service by itself is super useful.

Not sure why you're getting downvoted, personally I agree.

For example:

- a frequent/simple check dealing directly (on the internal network) with the webserver ("does it work well yes/no, what's the raw response time, etc..."). Here is where I would definitely use "curl".

- another less frequent test involving as well the DNS and the external network.

- another end-to-end test (e.g. once every 10 minutes?) involving as well one or more real browsers (this would test as well for example revoked SSL certs).

=> all these infos/metrics should be quite helpful to identify problems, or at least to shrink the potential area that is causing it.

Yep, for sure. Fastmail still uses a once-every-10-minutes frequency for the full end-to-end "can log into the website, compose an email, receive the email, trigger an automated background fetch, receive that email too" tests, though the "can connect to service" tests run much more often.

I wonder if there are page monitoring utilities that can download the full html payload, render the page via a headless chrome browser, and then perform a diff against the content? Most landing pages are quite static, and like you said, it's about full page load, so measuring how long the entire page took to load, along with each asset's network request and subsequent content paints, you could get a pretty good idea of your holistic page load health.

I've seen automated testing tools for that. I was thinking about it for the future once I get other monitoring I want. I'm literally building this to be a monitoring tool I want to use. But you would have to make the montioring system aware of deployments or design changes. But those are the things that generally break things.

But the key thing for me is the "goal" montioring. So for ecommerce it's orders but for other systems it's different. That's the thing I really want to monitor. If other things break they'll affect those so you can detect lots of failures. The only issue is finding out what the cause is. But first I'll improve the anomaly detection a bunch before looking into root cause detection.

It's certainly ok for things to break if it is a deliberate design change. Visual regression testing in your build/deployment pipeline already can flag a change to you as part of a github merge, I don't see why we can't have similar tooling in that pipeline that focuses on efficiency.

Why not run a fully functional monitoring job i.e. upload file with a monitoring account and check the results to validate that it is working end to end. Doing this even once a minute shouldn't put any load and is a much more reliable test.

It would have needed to be done through Chromium in non-headless (likely full Xvfb) mode, with step screenshots, and screenshot comparison (always flaky!), for the Safe Browsing interstitial to have generated an alert.

How about headless mode with validating response by parsing expected output? I don't have a lot of experience so not sure if what I'm saying is feasible.

FWIW, the Safe Browsing interstitials are managed by a slightly disorientating bunch of browser/renderer interactions mostly initiated by C++ that basically delivers chrome://interstitials/safebrowsing?type=malware instead of the page in question, IIRC with the renderer setup so you can't right-click and view source etc.

From a pure perspective, yes, you should be able to use headless mode and run some sort of validation on the HTML. That would make things really straightforward.

In practice... the easiest explanation is that I just spent about 35 minutes trying to find a live test/example safe-browsing trigger so I could check whether right-click is disabled, and I couldn't find one. Even after firing up a separate Chrome profile and verifying Safe Browsing was enabled at chrome://safe-browsing/, modifying /etc/hosts to point http://malware.testing.google.test/testing/malware/* anywhere did not work, none of the links https://testsafebrowsing.appspot.com/ generate any scary red warnings (and now my Downloads folder is full of EXEs), and I'm kinda sitting dazedly scratching my head a bit. Maybe all this requires some magic sauce I'm not aware of, or maybe it's just quietly bitrotted (doesn't seem to be the case?)... but the overwhelming lack of determinism is what makes trying to automate this next to impossible.

You just have to shove the website into a browser and screenshot the output. Because the chain between "load URL" and "user sees red" is that rickety, that there's no clean way to test it. D:

It's funny you read it that way, you may understand it correctly but I came away with a different interpretation, that they allow-listed the developer's IP and returned good non-phishing-warning responses to the monitoring check, but not to end-users.

They said their test scripts worked but people using Chrome got an error. So I take that as in their scripts weren't using Chrome at all.

To be fair, I've not had this happen yet so I am going to try and find a site that chrome won't let me visit and see what happens when I visit it programmatically.

When that warning page is thrown, is a 200 returned? It could “load” ok, but be blocked by a flag for chrome that isn’t http flag. Total guess. Anyone have any insight on that page showing up?

After a bunch of searching I found a test URL. https://testsafebrowsing.appspot.com/s/malware.html via my Chrome script I get a 500. And when ignoring it in chrome manually it returns a 200 so the web url works.

This is totally an edge case I didn't even think of until I read that blog. Super happy that my monitoring approach picks up on it.

For others wanting to do the same I'm using chromedp. It does take up way more resources tho. I worked out I can do 90 per minute per 8-core 16gb server.

[1] https://github.com/chromedp/chromedp

How would Google know that a site is curling another site?

Why would they flag that as a phishing site?

I’m just having a difficult time determining how this situation is not the fault of the site/app; we don’t even know that any of this is true and it looks more scripted than an offended rant.

The curl'ing is not necessarily bad (and can continue just fine), but rather its blind to these types of problems. A false positive in your testing framework (especially something like this) is the worst case scenario.

Google wouldn't know that someone is curling it which their script said everything was ok. While the website was basically down because Chrome and Firefox will both block a site based on Google's safe browsing list.

They could use Google's safe browsing api to check if they're on that list as well as curl.

Frankly, Google's Safe Browsing list is one of those things which should be broken out of Google and ran by an independent entity, much like the Let's Encrypt model.

I don't know how to help make this happen.

I'd suggest:

1. Mozilla and the Chrome clones (Edge, Brave, etc.) partner to make an (open as possible) standard for blocking and reviewing, and start maintaining their own list upstream from Google.

2. When Google adds to their blocklist, independently check it according to the consortium's own standards.

3. Cut a deal with Bing or Yandex to scan for malware as part of their crawls, to get technology independence.

4. Put pressure on Google to get onboard.

Step 4 is the hardest.

I still don’t understand what the problem is here.” With Google doing this.

We don’t know that this whole thing is not invented by the author or that they were indeed not doing anything malicious.

All of this, as credible as it may seem, could just be invented bad PR against Google, which if true should make us take a hard look at whomever is behind this.

Seriously, read the Lessons Learned again and tell me for certain this really happened. How in the world would it look so staged, with the bold words and thought-out structure. I’m not going to jump on the bandwagon everytime someone makes claims.

My point is: How do we know what the sit author is saying is accurate?