Reclaiming the web with a personal reader
olano.dev
olano.dev
There's an amazing power to just monitor websites with no sweat and skim in the morning
- new job openings for companies you like
- new job openings/closings from your current company
- products you're waiting to go on sale/back in stock/available refurbished (got 70% some nice headphones)
- covid sewage stats, if you want to know about spikes
- apartment listings
- github releases you care a lot about (<3 yabai)
- legal-ese for critical websites
Personally I rent a little digital ocean droplet for $5 since I also self host a RSS reader, personal telegram bot, etc (and it's very useful to set up little http site for experimentation) but could do it on your laptop since it doesn't have to run every day at the same time
[1] Feed me up, Scotty! https://feed-me-up-scotty.vincenttunru.com/
its not the first time i hear about feeling healthier after moving to the feediverse. i have my own set of scripts and mini apps running on top puppeteer with a local llamacpp for summaries and recommendations, its not perfect but im planning to put more effort on this and maybe look for OSS projects that are aiming at that (ie. (*arr suites, nostr, activitypub, veilid).) and offer that to friends and family members to see if they like the idea, also i have a name for all those scripts: "not a browser" the web without HTML CSS and JS served along with the data, just provide the data, how you displayed, its the user concern.
This sort of thing is very achievable if you work in IT and can run your own server. But what about everyone else?
Your idea of a personal “IT person”, like your own personal barber or tailor, is very intriguing. I wonder if its feasible to provide a service like this to people who are of a similar mindset about disconnecting from huge tech companies/algorithms and using something more personal, but don’t have the technical means to achieve this?
I’ve also been thinking about the personal data aspect of healthcare. I hate that my medical records are stored in MyChart and a dozen other proprietary systems that I have no control over. Yet, I have a super computer in my pocket. Why can’t I maintain my own copy of my records and selectively share data with my doctors when I arrive for an appointment? Why do I still need to have one doctor’s office fax something to another’s? I should be able to own my data and tap a button on my phone to do this. Only Apple’s Health app seems to come anywhere close to providing a fraction of this functionality, but there seems to be 0 adoption of this within the US. Even then, this only benefits Apple users. Something like health data should not be locked in a propriety system, even one that runs locally like Apple Health. There should be some open protocol and an ecosystem of implementations.
We had something close to good for a brief window before marketing and greed took over the internet.
I think if someone bully a synology nas-like product with apps designed to benefit the end user and people (“your IT person”) to support you which wasn’t aimed at fleecing all personal data and dollars out of customers it may very well work and would be utopia for the end user but the economics probably makes it a low margin business with lots of risks (e.g. liability over losing peoples files)
Wow, I haven't thought about this idea before. I really resonate with this and can see it being an actual reality in the next 5-10 years. That being said, I'm sure somebody has tried the approach. What's preventing it from happening now?
Just improvising with the idea and trashing more on big tech: I would say, nothing, if something like that gets to happen, my guess, it will be silent, away from the noise, would not be a sexy tech headline, kind of what is happening with mastodon, there is no big names, personalities or numbers with tons of 0s behind it, just an interoperable protocol, you know, real tech. it does not need marketing or infinite scale, dependent on FOMO or other social phenomenons, its just pure value for its users, also, as Cory Doctorow said somewhere, as instances of mastodon, ideas like that are going to come and go, and that's fine, that's the process of finding the next valuable thing that will stick with us, it should be organic, the next big thing will not come from the silicon valley casino like esque. An "IT person" would be just a job. pretty much as any other craft, what happens its that some of the most noisy parts of our industry are sick, feverish, delusional and full of their own bullshit, and we have in our subconsciousness that everything needs to be flashy, glorious, amazing, disruptive, big, competition proof and fast (in terms of success), not sustainable at all.
Also worth checking more about Jenny Odell's other projects, like The Bureau of Suspended Objects (https://www.jennyodell.com/bso-cjm.html)
In that vein, I'd also recommend another of hers, "Saving Time," which is (quietly?) incredibly radical. Personally, I found it the better of the two, with more of a focused narrative.
But I want to go further to not only be a personal feed but also time limited and distraction free.
I want to build a feed of all the written content I follow. And every day it selects the combination of items that together make for about 30 minutes of reading. This should include blog posts, articles, tweets, everything.
It can use chatgpt to filter what is the most "nutritious" content, or whatever other tool, but it should give priority to valuable content over flame wars.
Then this should be delivered to my Kindle or my remarkable tablet. Away from colors and flashes and away from fast internet.
Finally, for step two, is I want to be able to subscribe to my friend's feed and get "guest" content from their feed every now and then.
Now it’s just a simple JavaScript app I use locally but maybe I should make it cooler… this idea sounds cooler than what I was thinking
What's absolutely necessary to get anything done in a large mature project with many developers and a sprawling code base may be cumbersome in a small single-developer project.
In a small project where bugs are easy to identify and tend to be an easy 10 minutes fix, foregoing the correctness tax and fixing bugs as they become apparent is the more economic choice.
So I do think the "always test" mentality is a reasonable default for the non-prototype type of work. There's a tipping point where going on without testing can get the project out of control and it's hard to tell when that is, so in contexts where you care to avoid that risk it makes sense to be strict about it. I didn't care for that risk in this case, so it made sense not to bother and focus on building momentum. I'd probably add some integration tests now if I wanted to try a significant new feature or refactor, or if I had to consider contributions from other developers.
For example, testing access permissions. Your UI isn't gonna display data or operations you don't have access to.
But that doesn't mean your back end is honoring permissions.
The other thing to take into account is how noticable the bug is. You should inch more towards writing tests for bugs that are less likely to be noticed by someone.
A couple years ago, I was working on a personal project where I'd built a respiratory gas analyzer and needed to write some software that could interface with it over Bluetooth and display the data in real time. Unit testing for various functions that needed to accurately perform scientific calculations was very beneficial, but in retrospect, writing tests for other aspects of the interface was pretty much a waste of time. I ended up getting rid of the tests entirely once I had confidence my functions wouldn't be changing by that point. It turns out that when you're a solo developer, manually testing your application can be perfectly adequate.
My jaw is on the floor because this post seems like it could have been written by my future self. I can’t believe how much in common I have with the author.
I’ve realized I’m burnt out and have been planning to quit my job early next year (wanting to state this is the reason for the anon account).
But there’s a lot of people who are probably feeling this same way. What’s astonishing to me is what the author did is almost exactly what I’ve been daydreaming about doing in my time off.
I’ve also been thinking about the open/IndieWeb and how I’d like to engage with it. I was planning to build some sort of app to experiment in this space.
There are similarities even down to one of the specific problems in this space: how to prevent infrequent posts from being lost in the flood. And technical considerations: what languages and technologies to use. I had also been considering building something using modern web technologies, where I’m about 10 years out of date on web development.
In some ways, I’m delighted there’s another person out there to provide some validation to what I’ve been thinking and feeling recently. It makes me feel like I’m going to head down the right path.
In other ways, I’m upset and jealous the author got there first.
If it makes you feel better (or worse?) I wrote something very similar to this 20 years ago and still use it every day. I wanted my own RSS reader back then, hated all the ones I saw and wanted one that looked like a normal blog (not inbox, similar to OP's requirements) that I could design however I wanted. So I basically wrote an RSS feed parser to give me everything and designed it to look like my normal blog does.
That got modified to going to grab the full article if the RSS feed only had blurbs. I didn't want to have to click outside my reader to view the full post, I wanted everything in my feed reader. Since I had a basic page scraper, I used that for other sites that didn't have RSS feeds, this helped a lot when social media became popular so I never had to actually go to any social media sites at all. I could just stay in my own feed with the content I wanted (I was never really into social media anyway).
And as you can imagine, it being 20 years old means it's written in old tech, PHP and XSLT, because that's what I was writing 20 years ago. And it's still written in that.
In any case I highly suggest making your own. It's a fun project. Sure mine is crufty and old and sometimes doesn't generate the right content I want (scraping is fairly imperfect), but it's mine and it's been my daily reader for 20 years and I love it.
Don't be! the whole point of this was to build something for myself, and use the process to reflect. There's no reason why trying something similar shouldn't work for you, I certainly wasn't the first to implement a personal reader.
Fun fact: I just skimmed through one of the indie web posts I linked (which I had read months ago) and it struck me how much of their ideas I just replicated almost verbatim in my post:
> Firstly, don't try and make your software work for everyone, or just for a specific set of people you think may be interested. Make it for you.
> By making it generic and possible for others to work with it, you'll make tradeoffs that may make things worse for your own usage, or may even design for an imaginary user that may not even exist, or build a system that with 17 different configuration items you could have a completely different system. Be selfish and make it more useful for yourself.
Thank you for the encouragement! I do intend to still do something in this space if only for no other reason than to go through this process myself with the hopes of rekindling my interest in technology and software development.
And thank you for sharing your experience with us! The validation and motivation I’m feeling after reading this definitely overshadows the jealousy. :-)
The other main benefit for me is also being able to play and use any "unconventional" tech I care for. Building new-fangled single-binary PHP executables, using sqlite in prod, deploying WITHOUT Docker: these make it enjoyable for me.
What I've noticed is that this also has a knock-on effect on the main work for me - personal use repos often have new techniques & optimisations I can add in.
Its funny, how many technologists play with stuff on the side, derive learnings from these experiences and end up bring something helpful, even valuable to their professional jobs...but employers sometimes - at least in some of my cases - disparages such efforts, or often block such enthusiasm. Then again, maybe i'm just working for the wrong kinds of orgs. ;-)
I'm glad that you have been able to reap the benefits of the knock-on effect; kudos to you!
my experience matches https://news.ycombinator.com/item?id=38642092
(see https://news.ycombinator.com/item?id=38369946 for more details. Newsboat stores its state in a regular sqlite db, so I've since queried the db for stats and evened out my alphabetic splits)
I use rss2email to send all my rss subscription entries as emails. So I have one centralized location for everything (personal emails, mailing list messages, rss feeds). Combined with filtering to individual spools and a powerful email client (mutt), it is a very pleasant unified experience.
Would it instead be possible/easier to throw it on a VPN and make the VPN accessible from anywhere?
The reason I ask is that I want to be able to access personal web apps securely and I’m trying to figure out the easiest approach. Every time I look at authentication, it’s a labyrinth of concepts, protocols, and libraries. I don’t want to maintain that!
I really cannot recommend tailscale enough for how easy it is to set up a secure network of your own devices.
Edit: any pointers on how to set this up would be appreciated! Maybe I’m using the wrong search engine, but I haven’t found this scenario laid out clearly, yet.
In practice though - you don't have to worry about any of this! These are the steps:
* Create a tailscale account (there's a good free plan)
* Set up server. Give it a hostname (I'll use "mypi" in this example)
* Set up your web service on server - check you can get to it locally (e.g. connected by wire or just on http://localhost on that computer)
* Install tailscale on your server, and log in to your tailscale account (tailscale login and follow prompts)
* Install tailscale on your phone/laptop. Log into to tailscale
* On your phone/laptop go to http://mypi and it should Just Work!
When the VPN is on, the phone directs any traffic for 10.200.200.0/24 through the VPN and the rest of the traffic through the normal network stack. This is often called split vpn or split tunnel.
The other end of the VPN needs to be running wireguard and accessible from the internet. I have a VPS for this because my desktop is behind a firewall. But I can connect from my desktop to the VPS over wireguard (same setup) with keepalive, and they can all talk to each other over that private network.
I don't usually have this on, but occasionally if I want to ssh back home from my kid's soccer practice, I'll use it. I ssh to the 10-net address for my laptop after bringing up the split vpn.
For "securing" my application which hosts text snippets I've clipped from other websites and nothing else, it is sufficient.
version: "3.7"
x-common-variables: &common-variables
PGID: 1000
PUID: 1000
TZ: America/New_York
services:
caddy:
container_name: caddy
image: caddy:2.6.4
restart: unless-stopped
environment:
<<: *common-variables
HOST: "redacted"
LOCAL_IP: 192.168.1.2
ports:
- "80:80"
- "443:443"
- "443:443/udp"
volumes:
- ./appdata/caddy/Caddyfile:/etc/caddy/Caddyfile
- ./appdata/caddy/site:/srv
- ./appdata/caddy/data:/data
- ./appdata/caddy/config:/config
wireguard:
image: lscr.io/linuxserver/wireguard:latest
container_name: wireguard
cap_add:
- NET_ADMIN
- SYS_MODULE #optional
environment:
<<: *common-variables
PEERS: myPhone,myLaptop
ALLOWEDIPS: 0.0.0.0/0,::/0
volumes:
- ./appdata/wireguard:/config
- /lib/modules:/lib/modules #optional
ports:
- 51820:51820/udp
sysctls:
- net.ipv4.conf.all.src_valid_mark=1
restart: unless-stopped
(config) {
@internal {
remote_ip 192.168.1.0/24
}
handle @internal {
reverse_proxy {args.0}
}
respond 404
}
mySecretService.{$HOST} {
import config "{$LOCAL_IP}:5678"
}
- You may want a VLAN or separate guest network depending on if you allow guests on your network, what type of services you're running, etc.
- Many of the things I run at home have password authentication and I use them in addition to the VPN restriction.
- This was the first thing I thought of and may be insecure for reasons outside of my expertise.
- The nice thing about this is that I run pihole in the same compose file so when my phone is on my VPN I get remote ad-blocking "for free".
- Tailscale is easier and nicer (UI-wise) to set up, but I stopped using it because it's a battery hog on iOS. The "trusting someone else's server" thing is also an issue, but if not for the battery issue, I would probably still be trading the added risk for the convenience. This was not too bad to set up, though, and I'm happy with it for my simple needs. The Tailscale app also doesn't have a convenience feature that the Wireguard app does: I can tell Wireguard specific networks that I don't want it to run on (i.e. when I'm home) so that it enables automatically when I leave and turns off when I'm home.
* Being able to say “sync now” when we have a moment of connectivity (for example, passing an island with LTE)
* Being able to Readability and local cache all content (including images) by default so one can read when offline
miniflux as a self hosted option worked out perfectly for me (I used pikapods to host them).
It almost feels like reading newspaper, checking off one page at a time and when done - I move on to my other work/daily duties. No more random/aimless scrolling or visiting multiple pages.
The one thing that drives me crazy is that there is still no way to get posts from facebook groups into any sort of rss feed. Does anyone else have this problem or know a solution? The rss-bridge for FB groups has been broken for years b/c the FB redesign in 2020 made scraping harder.
The main project is on Ruby on Rails, but I have a microservice on node just for the "readability". I also extracted another service that would need a lot of memory and run only occasionally.
Those microservices are only needed from time to time, and I call them from a background job, so I let them autoscale to 0 on Fly.
I have lots of ideas I want to implement inclcuding storing all this data using embedding layers and the ability to deconstruct the html in use.
My last thought was that I want to take the 'articles', pull out the content, redisplay using django or a smaller web server, and replace the ads with positive affirmations such as "you are doing great!", "great job on your excercise". I figure I can even use deep learning to generate fake ads that look like normal ones but only serve to uplift me!
I enjoyed your take on using the app how you imagine it and developing around it (frequency buckets, etc). Those problems sometimes go unnoticed during the design stage but are crucial to really getting the benefit of the app - the REAL reason why you're building it and and eventually excited to use it. I have gone through similar career burnouts and it's projects like these that can re light the flame... Until the next burnout lol
> I treasure my attention and so I've spent some time to opt out of ads. 5 years ago I couldn't tell what DNS was, now I've got an OPNsense router at home running ZenArmor/Unbound and I can can link to it from my phone over wireguard.
> Using ublock I've made a point to prune the pages I visit, so classes like "header" "breadcrumbs" "recommended" "sponsor" &c. are hidden.
Google started dying for me with Reader, then Inbox, and now the last blows were the recent abuses in Chrome, YouTube and the absolute garbage the search frontpage has become.
I will definitely take a look a feedi. You just gave me some hope.
I did the same thing with my feed reader (where I'm subscribed), which sits nicely combined with my timeline (where I publish) in the same website system that I built.
Although the option to immediately reply to posts in my reader (by means of webmention) is very attractive, for now I don't mind clicking and visiting external websites and maybe leaving a comment over there.
For the rest, this article really hits home for me. I'm also dogfooding and trying to make it work for myself first, although my pub/sub functionality is part of a larger website/homepage system.
I'm really happy that more people are building this kind of stuff to be able to ignore algorithmic timelines, ads and other enshittification.
edit: Some kind of automatic reader is next on the list. I want to have that filter on keywords I entered upfront. In the background, this automatic reader should collect interesting articles/posts and create a sort of Newspaper or Magazine, which periodically presents itself to me as a sort of surprise.
If you have not figured it out yet, they would make more money having you do what you love to do, but the sadists in HR want to destroy your soul so they push you to do things you hate.
Mmm, no you don't. In fact, even if you install it with NPM—which you don't have to do, of course—Readability was designed to run in the browser, not on Node.
The "JavaScript" → "Node" logical leap that people make (even in starkly inappropriate circumstances like this one) shows how much damage the warped traditions of the package.json cult have done to good, clear-thinking reasoning.
Having this logic available in the backend was convenient for me, anyway. I'm using it also for the "send to kindle" feature, which I couldn't have if content cleaning was done browser side. Having it in the backend also opens the option to save the content in the db to skip loading/parsing latency and preserving it long term.
I don't know what you mean by "ad hoc". Again, Readability was written to run in the browser. It operates on the DOM, not HTML. If there's anything ad hoc going on, it would be (a) the fake DOM that Gijs wrote[1] so you can run it when all you have is HTML instead of an object graph, and (b) logic involved in shelling out to a separate NodeJS process from Python. These are hacks on top of hacks.
> which I guess is what you suggest
I wasn't suggesting anything. I'm making an observation about how illogical the JS,-therefore-Node cliché is. If I were going to suggest anything, it would be "don't use Readability" since it isn't a good fit for this use case. If "use Readability" were a requirement, then I would suggest, for the benefit of yourself and your brethren, rewriting Readability in Python or creating a binary Python module using either QuickJS and Readability plus Gijs's fake DOM, or Haxe.
1. <https://github.com/mozilla/readability/blob/main/JSDOMParser...>
It is a perfect fit for the user experience I was going for, even if it adds development and operational complexity --both of which were a lower priority for this project, as I stressed in the blog post.
That's not what I said. You're moving the goalposts.