Does anyone here have experience trying to build a search engine, even a "boutique" one, for the web? Is it something an individual could conceivably operate on their own?
Yes, I've built a curated boutique search engine, such as that described in the article. Short summary is that yes, it is possible for one person to build a useful search engine nowadays, if they don't attempt to index the whole of the internet (the niche mine covers is personal and independent websites). I've plenty of details about building it in the blog at https://blog.searchmysite.net/ , but key points you might find useful:
- It is now indexing around 6.5M pages, which is around quarter of what Google indexed when it was launched in 1998.
- Estimated running costs are now looking to exceed US$1000 a year. (I could change hosting provider to reduce this.)
- In January 2021 I estimated I'd spent around 350 hours building it (evenings and weekends over the preceding year). I haven't estimated how long I have spent this year, but it won't be quite as much as last year.
Do you make money with this project?
Hardware investment is about $3-4k as a one-time cost and then I estimate I'll need a 1 Tb SSD per every couple of years as the server does kind of chew through them with great appetite.
My monthly operational costs are $15 in power, and $20 for cloudflare because I kept getting DDoS:ed by botnets.
As for development time, dunno, I've been working on it in my spare time since this spring some time, generously estimated 30h/week x 30 weeks, so the upper bound may be 900 hours, but probably closer to something like 600 hours as I have other projects as well, and I'm not always feeling it.
I don't think off the shelf search solutions or databases are viable, they are too flexible which means they can't be fast and space-efficient enough to keep cost down. They're meant to run in a data center, not on a single computer. That means your operational costs will be prohibitive.
It's required a lot of old-fashioned wizardry to build though, bit-twiddling and demoscene-esque hacks to coax a lot of data into a minimal amount of space, the type of microoptimization stuff that usually is a waste of time except the data set is so large that saving single bytes in object encoding often translates to saving multiple gigabytes. If you aren't at least fairly comfortable with building custom compression algorithms, memory mapped hash tables, things like that, it's gonna be a rough project. If I didn't have a background in low level programming, this would have been nearly impossible.
Beyond that, most of this stuff you can pick up along the way. I didn't really know shit about building search engines before I started. I just threw together a design that made sense and built... something, and iterated upon that. With every iteration it's gotten faster, smaller, better, smarter. I think the upcoming release is gonna be yet another huge improvement.
Honestly though the big issue is crawling. Not only is it a bandwidth monster many sites are hostile to non google bots, along with cdn’s and cloudflare.
Indexing is the bigger challenge. An efficient fast service for an index of billions of pages is a different level to that for millions of pages.
I do my own crawling and don't agree with any of these statements. Bandwidth is not a bottleneck, and blocking is mostly only a problem if your bot is too aggressive.
https://blog.mojeek.com/2021/03/to-track-or-not-to-track.htm...
It runs off one somewhat well-specced consumer grade PC running in the living room of some person living in Sweden as far as I understand.
Already I find it totally delightful to use compared to both Google and DDG for anything that fits its index.