Hacker News Reader

Comment by ChuckMcM | Hacker News Reader

I totally feel your pain. I have been slowly building a local search engine for navigating the ebooks, PDFs, and other documents I've got stored on my NAS device.

I ended up replacing my NetApp StorVault (6TB (4TB usable), RAID6) with a FreeNAS box from iXSystems (24TB (20TB usable), RAID6) which cost me a bit more than $2500. (it is also much quieter than the StorVault was :-)). And on that 20TB I've stored a bit more than 200 book volumes that I had digitized at 1DollarScan, probably close to 1000 magazines (scanned generally manually with my ScanSnap 1500), and perhaps as many PDF documents that were never printed to begin with (like data sheets). I also have my music collection, and I have recently looked at putting movies on there as well since the streaming services are letting me down here as well.

At the end of the day it is the 21st century version of one's personal library.

View on HN

I have had a FreeNAS mini for 5 years now. Best 2K on tech I have ever spent. So much better then rolling my own (which I used to do). 4x4TB (2 striped 4TB mirrors). Thinking about replacing the drives with 8TB ones. I had one drive fail in 5 years.

Currently I can only really do metadata search, but I'd really love to set up fulltext at some point. I have this dream where someday I can do things when the Internet is down.

The other thing I'm interested in, and wonder if you have thoughts on: When you pass, your personal library is inherited. Will people know what to do with your digital library? Will it be useful? Will people want it?

My goal is exactly that, usability without me present :-).

The tools for processing PDFs into searchable text have a lot of warts. For a while IBM was offering a free Watson service to do this (now its part of Watson Discovery) which has some warts. I did manage a set of perl scripts that would post process the statements that I downloaded from the bank into CSV files, but I would still like to pull tabular data out of PDF book scans to make the data they provide more useful.

I have a simple frontend based on the perl Mojolicious module which Blekko had developed as part of another project but my indexing tools are still quite primitive. Simple bi-gram and tri-grams, and a growing synonym index. I don't give it enough queries to use my own traffic for ranking feedback. So basically everything is nearly equal rank. Basically I am about to the AltaVista level of search capability :-).

The vision is it just runs as a server and anyone on the same network can access it like a web service an pull up documents (and in the future media) of interest.

Did you get very far with other enterprise stuff like Sharepoint?

I didn't try Sharepoint, although I looked at Elastic Search on AWS briefly. The goal though was to have all of this stuff on premises both for latency reasons and to maintain a credible defense should someone come after me for copyright infringement.

For my eBooks, I use a java app called DocFetcher. It's indexing and word search capabilities are pretty nice, and I've had great luck with it, especially as I can set it up portably.

A bit of a tangent, but if you use a Mac, I've found that Dash is really great for getting things done without internet access. It's main purpose is to pull in local copies of docsets and search those, but it can also pull in sections of stack overflow (such as all Python questions, or all Pandas questions), and it integrates with things like Alfred, so you can do scoped searches without any GUI interaction.

So do you have ~19TB still free? It doesn't seem like your items would take too much space.

I do still have a lot free, when I replaced the StorVault it was about 75% full (3TB out of 4TB), I'm up to about 5TB now. But there are more things on there as well. The Postgres data for my local gitlab instance for example, and time machine backups of my Macbook pro. So it isn't all library stuff.

Reply on news.ycombinator.com