One Year of TILs | Hacker News Reader

One Year of TILs

simonwillison.net

228 pointstosh53 comments

Now I want to mark this site (in Google, say) with a "high quality" measure.

Then I want my search results to prefer this page, and other pages marked high for quality by people who [some algorithm] to this page.

Then Google would be far more useful to me than it is now, when screeds of blogspam always seem to occupy the top organic results for any "how to" tech questions.

In turn I would be a happier and more loyal Google customer (not a high bar).

And then less reputable website will hire consultants and influencers to mark their website as high quality. And then click farms will appear in India and China to mark website's as high quality at a price. And that is why we cannot have nice stuff / nice search results...

I think what they're going for is more personalized results -- i.e., don't aggregate my preferences with those of a bunch of click farms.

Let everyone mark sites as high or low quality, and then count other peoples ranking in proportion to how well the correlate with your ranking.

(A vague idea I've wanted to implement for awhile now for both search engines, and for HN/reddit like sites... but the amount of effort to do it well would be really high)

I would guess that, apart from the immense effort of building it, delivering personalised search results like this would be enormously expensive in storage for the search engine. Much more than sorting people into a few cohorts/buckets.

But FFS, it's 2021, we deserve some decent search engine results.

I doubt Google would do it unless they absolutely had to, so I hope you or someone else forces their hand and shows them that it's time.

> I would guess that, apart from the immense effort of building it, delivering personalised search results like this would be enormously expensive in storage for the search engine.

How expensive would it really be?

You have O(the_internet) in pages and metadata, and you have O(world_population) in user preferences. So long as your index structure allows those to be mostly decoupled (if I had to take a first crack at it I'd probably try to embed preferences and pages into a vector space and build a projection index -- exact matches are hard in that system, but decent personalized results are easy), I don't think it'd be all that much more space than a non-personalized search engine, especially given that the world population is kind of small compared to the size of the internet.

For that matter, the web isn't thaaat big (ignoring images and video). The entire common crawl can fit on a single $3k-$5k disk uncompressed.

It's not about who else marked this as high quality. I want my signal to count (disproportionally).

You're contradicting your first comment:

> I want my search results to prefer [.....] other pages marked high for quality by people who [some algorithm] to this page.

Facebook/YouTube already do this and it is a disaster, which maybe even has indirectly contributed to rise of antivaxx, Q, Trump election, Capitol attack etc.

Before personalized recommendation systems, when an antivaxxer made a video, hardly anyone saw it, when they liked it, the like resulted in nothing.

When their likes started affecting their personal search results, liking one antivaxx video made search produce hundreds more of them. It finds so much of the same stuff they no longer have free time to check an opposing opinion.

You're operating under an assumption that your judgement is more correct or better for you, and the entire problem is about finding gems in an ocean of trash. The reality is more complicated, for you antivaxx content is trash, for them it is your content that is trash.

I see what you mean, but like the OP, I would rather have google search preference resources that I like over resources that everyone in the internet.

Like if I think Wikipedia is a better resource than crunch base, then let me make that decision. I don’t care how good their SEO is.

This is not really whataboutism, with two sides.

And in any case, I don't care. I know what I feel is good content and it's stuff like this, and I want more of it, and less blogspam.

So yes, my assumption is that my judgement is more correct for me - because it is.

I will subscribe to a site I like in an rss reader. Then you can search your feeds. If that fails, you can broaden your search (e.g. Google).

> my judgement is more correct for me - because it is

That is plain arrogant. Not only you refuse to remember that you may be wrong, you also want search engines to protect you from ever discovering that.

What if your neighbor likes something that will eventually hurt you. For example, burning the nearby 5G tower that serves your phone, because they like the theory that will improve everyone's health around. Is their judgement correct for them? Do you want them to get more of stuff they like? Should their community of single-minded people be assisted by search engines in avoiding (what they think is) spam?

I understand everybody wants less SEO spam, I simply point that the solution you're thinking about has already been tried and found to have consequences no one expected. "Give me more stuff that I like" is the old problem. The new problem is called "How to find more stuff that I like without creating an echo chamber".

You seem to be all about straw man situations affecting other people - I'm (perhaps selfishly) only interested in the quality of my own search results, and how much better they could be if I could give feedback to the search engine.

If my neighbour burns down the cell tower, if another neighbour has trapped themselves in an antivax chamber on Facebook - well, those are some hypotheticals that have literally nothing with my point, which is simply that I want better search results, driven my my own selections.

Have you tried making your own google search? You can pick what sites it will search. I'm not sure if you can say "Prioritize these sites, but still give me results from other sites too".

I do wish there was something next to each search result though that was like "I like this site, include it more often." or "This site is garbage, ignore it forever"

I think it was this thing:

https://programmablesearchengine.google.com/about/

How do you rate the quality of things on Facebook?

My app only allows me to "like" things and I like some stuff that is low quality because I like to see it in my feed.

This is something that would be really cool to have and I think it would work if you could set up a proper incentive structure so it doesn't get gamed.

Some comments I had on this a little while back: https://news.ycombinator.com/item?id=20282606

YouTube has what you want and it sucks. You end up in an echo chamber of your own preferences. I know what you mean but sometimes your wrong about the world and knowledge so it makes sense to make your preference compete with others

Youtube can be house-trained to occasionally bring you something interesting. It does take effort, but it's not as bleak as I've seen people complaining about it.

A web-of-trust approach might help with that.

Select your own trusted reviewers, and unsubscribe from any that start recommending garbage.

Something like this would feel like a small first step, maybe the other way around though.

I'd like to put a timeout on any blogspam web sites I come across, and not have them show up in my search results again for one year. By that time they might have switched to having decent content, or maybe just gone down the toilet anyway or been pushed down by Google's algorithms. Or if not I could ban hammer them again.

Maybe you want to set up a programmable search engine? https://support.google.com/programmable-search/answer/451388...

Haven't tried it myself; but remember using a predecessor to this maybe 15 years ago.

"You may want to augment your results with general Web Search results. This includes results from anywhere on the web, but places emphasis on your personalized results"

That's interesting, but it lost me at " all you have to do is choose which sites to search".

I was hoping you would just need to add urls to it.

I'm guessing the right way to handle this problem is to search your browser history or bookmarks but those are additional searches.

You know about https://www.google.com/save aka https://www.google.com/collections right?

Some details: https://support.google.com/websearch/answer/9217379?co=GENIE...

This instinct is so right, but the multitude of comments show how hard of a problem this is. I'd be so elated to work on something that solves this but I think the answer may lie in self-regulation.

We software people are generally mini-philosophers and it can be easy to lose sight of ourselves. But can you or I solve Search?

I guess what I've learned before replying is that it is often a high bar whether we recognize it as such.

I'd love something like this.

I've managed to get partway there by using ArchiveBox and feeding it my bookmarks, and then using the sonic to search through them.

Would be great to have this more streamlined into a normal Google/DuckDuckGo/whatever search workflow, with an option to also search your friends' bookmarks.

Link to actual TILs: https://til.simonwillison.net/

In case the author reads this, what might be useful is to group them by some topic. A framework is only interesting if I also use it, and as it is, I'm not into front-end dev so none of the frameworks are interesting to me, but I have to read the 41 categories individually because they're spread throughout. Same for Mac stuff (I have no mac, so I don't use homebrew), database engines, python: they're all over the place.

---

This one sounded interesting: "Search across all loaded resources in Firefox" but unfortunately it's just about the debugger (the debugger also gives you that help right when you open it). What I'd love to know is how to ctrl+f across all tabs' page contents, or even just a stable search field that doesn't change every time I switch tab (so I can just Ctrl+Tab,F3 across them).

I have a simple search engine for them here: https://til.simonwillison.net/tils/search?q=debugger

I haven't put a great deal of thought into the https://til.simonwillison.net/ page design!

Hey Simon!

I love the concept of "learning in public" and I absolutely love what you've done here. So much so that I am now creating my own TIL page and have also decided to create a webpage listing links to TIL pages, with TILs sortable by category, interest, topic, person, date etc.

Your page will of course be the first on the list. If anyone else has a TIL page (or is planning to create one, or knows of any other great TIL pages) please reply with a link and/or follow me on Twitter @TILpages and I'll follow you back and find your page! Or you can use the email address in my bio. I'm excited! I think this will be a fun project.

P.S. And I just realised, Simon, that you are the co-creator of Django, the framework I am currently learning. Ha! What a beautiful coincidence. Only on HN could this happen.

Anyway, thanks for Django and for providing the inspiration for this project :)

I have one! Here's my (proud :)) TIL repository: https://github.com/saveriomiroddi/personal_notes.

Thanks, Saverio! Your page is now on the list! I had a look through and your notes are great as well.

Inspired by Simon, I created my own TIL site a few months ago. https://til.codeinthehole.com/

It's built using Hugo and hosted on Github Pages. Hugo is good fit for collecting TILs as it makes it easy to create tag pages and link to related content. Source: https://github.com/codeinthehole/til/

As Simon notes, writing TILs drastically lowers the barrier to publishing something online. I see publishing in increasing sizes of granularity as a pipeline (notes → tweets → TILs → blog posts) where each prior step helps inform the next step. To paraphrase a British proverb: look after the TILs and the blog posts look after themselves.

Hey David, I'm adding your TIL page to the list/site/project I mentioned upstream in this thread (a sortable collection of TIL pages and notes).

> I see publishing in increasing sizes of granularity as a pipeline (notes → tweets → TILs → blog posts) where each prior step helps inform the next step. To paraphrase a British proverb: look after the TILs and the blog posts look after themselves.

I really like the way you've expressed this point and I'm hoping you won't mind if I quote you somewhere on this new site, either on the homepage or about page?

This mindset is closer to those who keep personal wikis in public - not simply broadcasting updates to an audience, but writing and working in public. It gets even better when conversations start to emerge with others who are also blogging this way. You can start to do longform group messaging that is open-ended - anyone can join in if they want.

TIL is actually a foundation of my knowledge, although I have a different approach - I keep Markdown files categorized by subject (e.g. `git`, `ruby`), essentially, notebooks.

The ultimate goal is both to memorize and to keep references to useful snippets or quirks I may need in the future - I actually use them very frequently (it's a format that tends not to be useful to the general public, though).

In the long term, some subjects become small books. Amusingly, by far, the largest notebook is Rust - around 15k words, and I am still an advanced beginner (!).

I really miss a program that is very good for both editing and viewing Markdown! I currently use an always-open VSC; reading Markdown is ok, but not super convenient.

Assuming you know about stuff similar to obsidian.md (markdown left, rendered right) and Typora (rendered where you edit), what is missing for you in terms of editing/viewing MDs? What makes current stuff bad?

I like haroopad for editing markdown, especially the option to export styled html is nice. Another nice editor (with a different scope, it's a note taking app) is Joplin. You can even paste an image and it will get inserted into the markdown as file.

inkdrop might fit the bill for editing/viewing markdown. It's paid, but i get value from it.

https://www.inkdrop.app/

I do the same with a snippet manager. The one I use [1] is also a decent markdown editor / viewer.

[1] https://www.renfei.org/snippets-lab/

> I really miss a program that is very good for both editing and viewing Markdown!

I like Typora for this, because by default it's WYSIWYG but you can switch to Markdown whenever you need to. https://typora.io/

As a former coworker with Simon, he is endless fascinated with learning and helping others gain knowledge. The fact that this page exists is not surprising to me in the slightest. It represents the high quality research and documentation he consistently provided. He is not with my company any more, but I am happy he continues this tradition and shares it with the internet as a whole.

Thanks! This bash string escape trick will be really useful for me(1). It is one of the biggest pain points in using cli especially with nested commands like su -c where the actual command is supposed to go inside quotes and becomes hell to quote if it too has nested quotes inside it.

Oh how i wish that bash and every other language would support the Perl's brilliant `q()` and `qq()` for quoting strings. That simple fn really made life very easy and code cleaner.

(1) https://til.simonwillison.net/bash/escaping-a-string

There's an easier way to escape in bash (that I didn't know about for very many hours of frustration):

     $ echo $'foo"\'"bar' $"foo'\"'bar"
     foo"'"bar $foo'"'bar

Yes I learned the same too yesterday from Hacker news. What a coincidence! There was also a bashquote function which you can put in your .bashrc file but this approach really beats everything!

I do the same thing, except my bar is "could this help someone else?"

I don't collect visitor statistics nowadays, but those posts tend to bring hundreds of visitors per month, so they're certainly worth something to someone.

This is somewhat similar to Zettelkasten[0] which is a way to take note.

Here's a quote from the book "How to Take Smart Notes" by S. Ahrens[3]:

Notes build up while you think, read, understand and generate ideas, because you have to have a pen in your hand if you want to think, read, understand and generate ideas properly anyway. If you want to learn something for the long run, you have to write it down. If you want to really understand something, you have to translate it into your own words. Thinking takes place as much on paper as in your own head.

One of the differences is perhaps Zettelkasten notes does not necessarily linked by topics. Instead, notes in zettelkasten are linked by tags[1]. Another diference is separating bibligraphical notes and permanent notes[2]. Explaining it in your own words is the best to make it stick in your head. Linking ideas by topics and objects may lead into your own new ideas.

[0] - https://en.wikipedia.org/wiki/Zettelkasten

[1] - https://web.archive.org/web/20210126003640/https://zettelkas...

[2] - https://web.archive.org/web/20210423224947/https://zettelkas...

[3] - https://books.google.com.ph/books/about/How_to_Take_Smart_No...

I've been doing something similar, albeit less organized and, more automated. I have a page in my blog [1] where I showcase all my TILs, which are mostly interesting links I encounter. This page is updated through a Discord Bot [2]. The bot updates a JSON and using the JSON, I use a Jinja template to reconstruct the TIL.md file. I want to figure out better ways to display this information, both for my own purposes and for sharing with others. And I also want to automatically archive links I record this way. I am considering using Firefox Bookmarks to manage them so that I don't need the bot, just a cron job that updates the page using bookmarks. And I can reorganize at will using the bookmarks. I'm mostly concerned about the privacy problem, and I've never been such a great user of bookmarks in any browser.

[1] https://stonecharioteer.com/til.html [2] https://stonecharioteer.com/sarathi.html

Oddly I wonder where I came across the idea of a TIL page. @pradyunsg had a status up, but I wonder if he was linking to OP's database. Either way, this is something I enjoy so much that I also maintain my interview and tech learning notes publicly. [3]

[3] https://notes.stonecharioteer.com

Public learning should be more of a thing than it is, currently.

I think a lot about how we amass 'knowledge' all our life, then at one point we grow old/die and most of it is 'lost' unless we've written a book etc.. passed it down w/ someone else who has done the 'work' to have it transferred etc..

I wonder in say.. year 2155, the learning 'curriculum' for whatever tech thats going to exist then.... will be super streamlined and powerful.

I follow Simon on Twitter and discovered his TIL site a few weeks ago -- it has to be one of the best things I've found on the internet in a long time. Refreshingly useful and personal, if you're reading this: thank you Simon!

https://simonwillison.net/2002/Jun/26/useRealLinks

I started keeping track of useful tidbits i found or cobbled together from stuff on the web, on my blog, years ago , to keep them easy to find. This is mainly for my own benefit though there’s some added value in publishing it all in my public page, as opposed to keeping it in my local notes (easy to refer to and search) - if search engines index my content and it ends up helping someone, that’s an extra win :)

I don't know what I was expecting, but I'm a bit disappointed they're all so programming focused. Though it seems like that may be the point.

I think I came across something similar related to TILs and that motivated me to start my own TIL and that has really helped

If you are interested here’s the link: https://til-mraza007.vercel.app/

Reply on news.ycombinator.com