A sysadmin's rant about feed readers and crawlers (2022)
rachelbythebay.com
rachelbythebay.com
The one that annoys me most is the accept-language header which is almost entirely ignored in favour of GeoIP lookups to figure out regionality... which I find super odd; as if people are walking around using a browser in a language they don't speak. (or, an operating system configured for a language they don't speak).
ETAG's though, are a bit fraught- if you're a company, a security scan will fire if an etag is detected because you might be able to figure out the inode on the filesystem based on it... which, idk why that's a security problem eitherway[0], but it's common for there to be false-positives[1]... which makes people not respect the header.
Last-Modified should work though, I love the idea of checking headers and not content.
I think people don't care to imagine the computer doing as little as possible to get the job done, and instead use the near unlimited computing power to just avoid thinking about consequences.
[0]: https://www.pentestpartners.com/security-blog/vulnerabilitie...
Well, yes, they are! Computers translated in my native language sound dumb. That's how a whole generation of my world learned better English than native speakers, ffs!
Half of the time it's just translated wrong. You think anyone has any incentive to translate any technology to a language with a couple million speakers, all of whom are obligate pirates?
And it seems like you might be surprised to hear that people speak more than one language. Then where's my global setting to tell the browser what languages I speak, so it'd know what header to send? Same place that lets me configure what ads I'm actually interested in. Nowhere.
>I think people don't care to imagine the computer doing as little as possible to get the job done, and instead use the near unlimited computing power to just avoid thinking about consequences.
This, friend, is what computers are for in the XXI century. "Bicycle for the mind", ha...
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Ac...
Firefox: https://support.mozilla.org/en-US/kb/choose-display-language...
Chrome insists that the first language be the UI language, and Safari insists that the first language be the _system_ language.
So I suppose GGP is mostly right, in the sense that most browsers get this wrong (except Firefox).
As someone who lives between 3 languages what I'd really like is a browser setting for language per-site. E.g. I want my Swedish bank's site in Swedish, not the English translation, I want Google Maps in Japanese so I can see the Kanji for the station names, but I want the AWS console in English. Each of these sites have their own toggles but they are very inconsistent and keep resetting, the browser would have done a much better job.
I wonder if this can be done with a browser extension.
In Chrome: chrome://settings/languages
In Firefox: https://support.mozilla.org/en-US/kb/choose-display-language...
I tried it out then reverted to the default.
Because I keep forgetting it's not the 90s and "we" have also invented such brilliant things as browser fingerprinting.
Look again. Or switch browser. It is a basic feature and the issue is indeed websites ignoring it.
Don't care either way, but it does make you think...
I don't know OTOH whether "target audience's spoken language" is one of the signals an advertiser can key into in targeting an ad (at a glance, it looks like it might be). But (a) advertisers don't always have that signal and (b) advertisers themselves aren't always savvy enough to set it (how many American advertisers targeting Iowa actually tag their ads as "in English?"), so you'll end up with region targeting as a proxy for language targeting.
In your case, it's probably that the ad engine doesn't have enough info on you so it's falling back to geotargeting and hoping for the best (are you running with JavaScript disabled? Clearing cookies frequently? Avoiding logins? If so, these are all things known to decrease ad signal quality).
They literally have first class data.
Back in the day, I had a front-row-seat to this process and I observed how often advertisers simply misconfigure a campaign and under-target it. If you don't set a targeting preference for a given indicator, the default can be to target everyone regardless of what that indicator says about them.
It might be the case that advertisers are saying they want to target you anyway (or failing to say one way or the other) even though they should have enough signal to know it's a wasted impression.
It's the "Your site's broken if IE won't load it" problem.
FWIW Outlook does accept the "Accept-Language" header and I don't think anyone is saying that outlook is wrong for doing that or claiming it to be broken?
Are you totally sure that this isn't a backwards myth?
I think the most likely situation is that locale information for English speaking countries would be incorrect if the default (en_US) was used to install the operating system, which happens on occasion.
I'm talking more like https://www.buerklin.com/. If that site comes up in the wrong language and the only way to fix it is to change the user agent's Accept-Language header, the user isn't going to just figure it out; they're going to navigate elsewhere. So the site has a bug in the top-left to toggle English or German.
Something you mentioned up-thread that I should have commented on but overlooked:
> which I find super odd; as if people are walking around using a browser in a language they don't speak
... yes, all the time. In libraries and Internet cafes, schools, and other shared spaces.
Or your library has somehow misconfigured their PCs when setting them up?
A cookie based override already exists, forcing geoip is strictly worse as a default, except for localising currency? I guess.
https://www.reddit.com/r/webdev/comments/7a2cfe/comment/dp77... for details: there are a lot of reasons speakers more comfortable with another language will have their OS locale (and therefore the accept-language header) set to English.
I lived in Flanders, with my accept-language set to en-US, en.
Ads would pop up in Dutch, Flemish, French and sometimes German. When you think about it, from a brick-and-mortar point of view, it makes sense. I'm more likely to buy <physical product advertised> at the <local chain grocery store> vs buying it anywhere in the USA, based on my IP.
Next to that, imagine you browsing Reuters.com in with a Berlin IP and accept-language set to en-US, en.
What SHOULD they show you? Local news in German, auto translated? Local news in German? Or redirect you to the US page?
Personally I would prefer, for example, Reuters.com to be a "hub", and all the regional variants on de.reuters.com. Then just let the user choose what they want.
[1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Ac...
Especially as the cost to serve this content approaches zero.
I find the take in the blog to be relatively hostile. It's a "technically correct" rant. Not wrong, but mostly missing the point, and being a bit of a dick in the process.
Sure - block the readers that make a request every 10 seconds. It's perfectly reasonable to block clients if they hit a limit like 20 to 50 requests in a day.
It's damn hostile to block for 24 hours after a single request. If the 10MB of traffic for 20 requests is going to break the bank... maybe don't host an atom or RSS feed at all?
---
That said - weirdos can weird on their own sites as they like. It's not a public service.
But I bucket this into the same category of weird as posting a whole bunch of threatening "no trespassing", "beware of dog", "homeowner is armed", "Solicitors not welcome", etc style signs all over their property.
Like - point out on the doll where the rss client hurt you. Because something's up.
Seems like it'd be straightforward to implement a backoff strategy based on how frequently the feed content changed into most readers. For a regular, periodic fetch, if the content has proven it doesn't update frequently, just back off the period for that endpoint.
> Besides that, a well-behaved feed will have the same content as what you will get on the actual web site. The HTML might be slightly different to account for any number of failings in stupid feed readers in order to save the people using those programs from themselves, but the actual content should be the same. Given that, there's an important thing to take away from this: there is no reason to request every single $(&^$(&^@#* post that's mentioned in the feed.
> If you pull the feed, don't pull the posts. If you pull the posts, don't pull the feed. If you pull both, you're missing the whole point of having an aggregated feed!
Unfortunately there are too many feeds that don't include the full content for this to work. And a reader won't know if the feed has the full content before fetching the HTML page. This can also change from post to post so it can't just determine this when subscribing.
> Then there are the user-agents who lie about who they are or where they are coming from because they think it's going to get them special treatment somehow.
These exist because of misbehaved web servers that block based on user agen't or send different content. And since you are complaining about faked user agents that probably includes you.
> Sending referrers which make no sense is just bad manners.
HTTP Referer should not exist. And has been abused by spammers for ages.
That's a niche. It's about 1 million percent more likely a fake request is coming from an overzealous AI scraper nowadays. I have blocked hundreds of them and I'm on the verge of giving up and handing over money to Cloudflare just for their AI scraping protection.
People probably do this because some sites only give you a preview in the feed, to force you to go to the site and view the ads.
So if you want the full post in the feed reader, you need to pull the post as well.
This person isn't thinking as a user.
This adds a nice publish-subscribe model to RSS. Ping the WebSub server when there are changes; subscribing services are easily notified; nobody has to worry about excessive polling. Hooray.
EDIT: There is the `Cache-Control` header, it seems ideal for this use-case
- https://docs.github.com/en/enterprise-cloud@latest/rest/acti... - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Re... - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Ca...
That said, I think 3 or 4 days seems reasonable.
It's easy to just curl a feed every second, but should you? (Of course not)
Take it as a challenge to make your reader as fancy as possible, use every trick in the book to optimise how it fetches content. Analyse the patterns of releasing new content per feed and adjust the fetch frequency based on that.
And if you're building a reader for distribution, don't let the user set a refresh interval that doesn't make sense.
But I bet you can get 95% of the benefit with a simple exponential backoff scheme.
Exponential back off is great for a lot of problems, but irregularly updated blogs doesn’t seem like one of them.
In some cases the reader should fetch both the feed and the pages. Unfortunately, none do