daoudc

This is really cool! Please consider joining forces with us at mwmbl.org, would love to incorporate some of these ideas.

I have also been struggling with this, both in storytelling for my kids, and in trying to make interesting YouTube videos.

Lots of people on YouTube seem to recommend Save the Cat by Blake Snyder. My experience of it is that there's some useful ideas in it, but it's overly formulaic.

We have a distributed crawler that volunteers can install as a Firefox extension to crawl the web ;)

Thanks, the organisation is a large part of what we're working on.

We do have other plans for financing, e.g. premium versions of the webmaster tools.

Tech stack is mainly python for back end and vanilla JS for front end right now. However we'd like to move a lot of the back end to Rust eventually, and for the front end we will probably adopt some framework.

This is great! If you're interested in search engine UX, I'd be interested in incorporating some of these ideas into https://mwmbl.org - would love to get your input - join us on Matrix https://matrix.to/#/#mwmbl:matrix.org

Not at all, feedback is very welcome!

Fascinating, thanks for this! Looks like it became controversial because of a lack of transparency.

Our index is still very small. Help make it bigger!

Feel free to read the source code if you're concerned about what it does!

1% of 1%

Roughly this https://github.com/mwmbl/mwmbl/wiki/Crawler-Phase-2-Design

It changed a bit in the implementation.

The data is available! Not currently documented but check out the code for the API.

Wikipedia has to deal with similar issues. It's not easy, but it should be possible.

Good point, this is not going to be easy to get right.

Yeah the search quality has a way to go!

That word (Mwmbl) is actually a link, but I think the link colour is not different enough from the text. Let me see what I can do.

I'm hoping that having community moderated search results will limit this problem. The problem will be SEO people trying to infiltrate the community, which may be tricky to solve. But Wikipedia has had to grapple with similar problems, so I think it is solvable.

Do people use YaCy? I found it very slow the last time I tried it

That's pretty cool, thanks! The loading page between each section seems unnecessary though.

This looks cool, but I'm not familiar with the concept of a "tour" for a project. Can you point me to an example?

I'm currently trying to promote my open source project Mwmbl (in particular the crawler extension https://github.com/mwmbl/crawler-extension). I'm wondering what a tour would look like for it.

It is an actual web crawler. I should have put a link to the server side code.

Thanks for the feedback! Yup, a delay is on the to-do list.

Oh that's interesting, I may try and get in touch with the author.

Ah good point, we haven't done any stemming or anything like that yet.

While search funds most of Google, most employees at Google don't work on search. A small proportion of Google's revenue would be plenty for good quality search.

Thanks for the detailed feedback! I think most of most of the problems here are because we have a really small index right now. Increasing the number of documents is our top priority. I agree that some kind of feedback when there are no results would be a good idea.

Update: there's been interest from a few people so I've started a Matrix chat here for anyone that wants to help out or provide feedback: https://matrix.to/#/#mwmbl:matrix.org

Good suggestion, thanks.

Impressive! Does that include the startup costs of the server you bought, and maintaining it?