Hacker News Reader

Hacker News

Comment by verall | Hacker News Reader

Full thread

verall

I guess cppreference.com isn't even a part of the list?

I tried a couple test queries:

> lambda decay to function pointer c++

I get some FSF pages and the wikipedia for Helium?

> std function

I get... tons of Rust docs?

> std function c++

All rust docs? The wikipedia page for C++??

Interesting idea, but this seems like it would be the primary failure mode for an idea like this: as soon as you are researching outside of the curator's specializations, it doesn't have what you're looking for. Yet these results would both be fixed simply by adding cppreference.com to the index. Let's try and give it a real challenge:

> How to define systemverilog interface

And as I might expect, I get wikipedia pages. For "Verilog", for "System on a Chip" and for "Mixin".

1st google result:

> An Interface is a way to encapsulate signals into a block...

Working as expected

View on HN

I added cppreference.com now and kicked off a crawl. It'll be a while. The list of domains is pretty small right now -- it was intended to be bigger before the announcement was made. Will also add RFCs and man pages soon.

There will (soon) be a form to request new domains are added to the index, so if there are any sites you want indexed which are outside of my personal expertise, then you'll be able to request them.

You probably are already thought about it, but just in case feature idea: adding moderation support for collaboration. Somewhat trusted persons vetting niche subjects.

I guess missing content can be explained by...

> Notice! This product is experimental and incomplete. User beware!

But in reality, if you know where is the answer that you are looking for, why would you use that search engine.

I use DDG and if I want to search on the Scala docs, I use "!scala whatever I'm searching" instead of just search with DDG.

GitHub doesn't seem to be either. I get that it's a competitor but not being able to search GitHub is probably a deal breaker for most devs that aren't Drew.

I'm not opposed to indexing GitHub, but the signal to noise ratio on GitHub is poor. Nearly all GitHub repositories are useless, so we'd have to filter most of it out. I think instead I'll have to set it up where people can request that specific interesting repositories are added to the index, and maybe crawl /explore to fill in a decent base set.

Perhaps all repo's that have a published package is a good heuristic. Then you'll at least get all the repos of npm, python and other packages.

Some interesting repos have no published packages. A combination of number of commits, stars and forks would be probably more relevant.

And likewise, some uninteresting repos do have published packages.

GitHub is hella tricky to crawl too due to its sheer size and single entry point (meaning slow crawl speed). I've been looking at the problem as well, and so far just ignored it as un-crawlable, but I might do something like crawl only the about pages for repos that are linked to externally some time in the future.

There's an asterisk to that: they serve the underlying content through two different APIs so one can side-step the HTML wrapper around the bytes: the discovery phase has a formal API (both REST and GraphQL) for finding repos, and then the in-repo content can be git-cloned and one can locally index every branch, commit, and blob, without issuing hundreds of thousands of http requests to GH. One would still need to hit GH for the issues, if that's in scope, but it'd be way less http requests unless your repo is named kubernetes or terraform.

We're still talking about git clone:ing a hundred thousand github repos. Git Repos get big very fast. That's a lot of data when you're realistically only interested in is a few markdown files per repo.

Reply on news.ycombinator.com