350 karma · joined May 7, 2017
No, for retrieving the Hacker News Posts we were using the public Hacker News API, which returns the posts in JSON format: https://github.com/HackerNews/API
The crawling speed of 100...1000 pages per second refers to crawling the external pages linked from Hacker news posts. As they are from different domains we can achieve a high crawling speed while being a polite crawler with a low crawling rate per domain.
We mainly surface historical content, which receives additional traffic from DeepHN, instead of taking value away from the website.
"Well-defined": Just a guess: We are doing key text extraction, i.e. we try not to index boilerplate stuff an and menu items. As "well-defined" is within a short list item, it might be accidentally skipped.
So, that is not yet perfect, and considering the diversity in web page structure it probably never will. But we will try to improve.
The original HN link is opened in a new browser page/tab. The search results should remain unchanged in the previous tab. So you don't use the back button, but the previous browser tab.
>> Clicking on what I understand are tags (hashtags) in a given post has no effect.
Search for "google", go to the first result. there is a hashtag "oracle", click and the search results are filtered by that hashtags. Please post an example where its doesn't work, so that we can fix it.
The date is shown in the preview panel on the right hand side. But wa are thinking adding the time also to the result list.
We are deriving the tags from the terms and bigrams in title, text and parsed html of linked web pages. Top frequent terms per post, but only if terms are within top 65k tags per index. Stopwords are excluded.
Sometimes this is legitimate, sometimes spam.
The pricing for an Index like DeepHN would be $99/month: While we are indexing 30 million Hacker news posts, for DeepHN we are combining a single HN story with all its comments and its linked webpage into a single SeekSorm document. So that we index just 4 million SeekStorm documents.
Yes, it would be possible to expand index with embeddings (vectors) and perform semantic search. This would we an auxiliary step between crawling and indexing.
If you scroll long enough within the search results you will finally reach results where the search term is not in the title, but only within the linked web page.
It is more obvious if you search for queries which are less popular than "friendship"
SeekStorm is intended that the user can index and search their own private data.
But if there is demand we could also allow to search in a central public index for web data or other public data.