Hacker News Reader

Comment by bambax | Hacker News Reader

All true. Taxonomy is indeed hard. But, does it actually matter?

It seems what matters is not how files are stored/organized, but how one can find the files one is looking for. Taxonomy is mostly a search problem.

Yet, although one is usually capable of remembering specific or unique details about a file, it's still incredibly hard to search for a file or its contents effectively.

Dropbox, to pick just one example, ditched all its advanced search functionalities a couple of years ago and only allows to search for any keyword (not all of them), with forced stemming, and no filtering of the results.

I seem to remember there were startups a decade ago trying to address the search problem on the desktop but they all got acquired or folded; I don't understand why. The problem is real and seems quite solvable; yet it seems there's no actual market for it. It's a bit of a mystery.

View on HN

> it's still incredibly hard to search for a file or its contents effectively.

Underappreciated aspect of this is trust. As much as "taxonomy is mostly a search problem", search is in large part a trust problem - trust that the search was done exhaustively, and if it returned no results, it means there aren't any.

This is a problem that IMO most offerings are blind to. One big offender for me is Windows Explorer. It has a search tool that can search both in names, metadata and file content. Yet in the past, it frequently failed to find files I knew were there. Truth is, it may not have been even looking in the right place - it depends on what's been indexed and how, which is information you can find somewhere in the system, but notably not in the search interface itself. I've stopped using it long ago, as I don't trust it at emotional level. I prefer to literally walk the filesystem structure by hand.

And, on Windows, I at least get that option. This is the fallback, the baseline: even if I don't trust a search engine to be exhaustive, I can cope with it as long as I can perform an exhaustive search manually. I.e. as long as I have a way to list everything. But this, then, is almost universally missing from cloud offerings.

(There's a whole rant to be written about the incredibly dumb idea of hiding the filesystem / database from the end-user, but I'll skip it today.)

> there were startups a decade ago trying to address the search problem on the desktop but they all got acquired or folded; I don't understand why. The problem is real and seems quite solvable; yet it seems there's no actual market for it. It's a bit of a mystery.

Nobody really wants to solve this problem anymore, because it conflicts with the major thing vendors want: for you to move your data into their cloud. The desktop, and the ability to own your data, survives only because most cloud systems are still shit. This might eventually change, and in the meantime, the market definitely isn't interested in helping you own your data.

> because it conflicts with the major thing vendors want

Maybe, but that doesn't explain why there are no startups on the case, that wouldn't have this conflict of interest.

Who would fund those, and where's the potential for outsized growth?

I think the problem is there's no demand, not that incumbents are fighting it. But I also think it's weird and a mystery that nobody wants good search.

The trust issue you mention may be a clue.

There isn't even an open source product with traction.

ripgrep solves the "precise full-text search" problem quite nicely.

There is that, and locate.

Most people have Word Documents, PDF files, and other things that need a more complex indexing strategy. Also a lot of people have lots of image and audio files which pose their own challenges, namely indexing textual metadata and possibly some indexing of the content.

> I think the problem is there's no demand

Computing is a supply-driven market. The demand is there, it's just partially latent, partially ignored. Vast majority of technology users have no choice but to choose from what's being offered, and the minority of tech-savvy users with opinions are increasingly too small a niche to support "power user" tools.

You'd think it wouldn't be that hard to build a personal search engine on top of Lucene or Elasticsearch and on one level it isn't. But there are two very hard problems.

(1) Performance. Back in the day people frequently turned full text indexing on Windows off because it would slow down their computer too much. People won't be happy with the overheard of a search engine that is always scanning 10's of GBs of document.

(2) Search quality. People are familiar with Google being an effective search engine but they've certainly tried a number with terrible relevance scoring and probably learnt that it is not worth trying the search on a web site or the search on the help of an application. In the case of Elasticsearch the default similarity is BM25 but that has two tunable parameters. There are other similarities you could use but most of them have tunable parameters. It makes a real difference what you choice and there is a methodology for tuning them that is now built into ElasticSearch.

https://www.elastic.co/guide/en/elasticsearch/reference/curr...

I talked to about 20 vendors of full text search products and found that only 2 out of the list regularly evaluated the quality of the results, 1 of them just did it so they could get some advertising by being on the TREC leaderboard. They told over and over again that customers didn't care about search quality, they just wanted to see a list of 350+ data sources that the product could index.

> They told over and over again that customers didn't care about search quality, they just wanted to see a list of 350+ data sources that the product could index.

This seems to describe corporate buyers ticking boxes on a form, more than actual users.

Sometimes those box tickers are your customers, as the ones paying you. That's how you get Enterprise Software

I think what you're describing is database query, not search. Heuristic search is going to be non-exhaustive by definition.

Desktop-focused databases are of course widely available (Paradox, Access, LibreOffice Base etc.) but the volume of data you could manage with them is of course limited. I.e. non-toy datasets will naturally be hosted on some sort of "cloud".

> I think what you're describing is database query, not search.

What's the difference?

> Heuristic search is going to be non-exhaustive by definition.

To me, a non-exhaustive search is broken by design, because it will miss results. Making data accessible only through such search means there will be a hole in the system into which some of the data falls, never to be seen again.

Elasticsearch in particular can do the kind of queries most people expect from a database, see

https://www.elastic.co/kibana/

At work we have a search interface powered by Postgres that uses the full text index but also uses GIN indexes on arrays to index things based on categories/tags.

That argument is much more convincing in the "open world" of internet searches, where enumerating all results is simply not going to happen.

In the scenario where you're searching for a file in a group of low-hundreds of files, it's nothing but a bad cop-out excuse.

> startups a decade ago trying to address the search problem on the desktop but they all got acquired or folded; I don't understand why.

Work migrated to the cloud.

True if you know what you're looking for. But apparently discovery is highly underrated by many.

Yes! I think this is a key concept that is often overlooked. Taxonomies -- and ontologies for that matter -- can often serve as a guide to the corpus, particularly when a user has less familiarity with the knowledge space.

And yes, any taxonomy is by definition always a subjective, biased construct. But this is often useful in helping to reveal the preferences and motivations behind the creation and curation of the corpus itself.

I'd guess only a small portion of people actually care for that feature. People are maybe better at remembering what's in their files by name alone than we give them credit, or willing to just look through things to find them, or are better organized via folders than we might assume.

Important for technical users, not so much regular people.

For Windows there's Everything a pretty good search tool.

https://www.voidtools.com/support/everything/

MacOS and Windows got built in search around 15 years ago. Anything more advanced would be very niche, so there's probably no market left.

Windows search only finds things sporadically though. I turned off search indexing on my Mac because otherwise I'd have to go out for coffee when I open any application.

More like 25 years ago. https://en.wikipedia.org/wiki/Mac_OS_8 had it in 1998 (possibly earlier. It isn’t clear to me whether Sherlock became PowerPC native in 8.5 or was new and PowerPC native in 8.5), Windows NT in 1996/2000 (https://devblogs.microsoft.com/windows-search-platform/the-e...)

Reply on news.ycombinator.com