Hacker News Reader

Comment by Mister_Snuggles | Hacker News Reader

I've started saving electronic copies of manuals, assembly instructions, etc for anything that I purchase. If there's no electronic copy available, I'll scan the paper manual. In all cases, I'm putting the documents into an instance of Mayan EDMS[0]. Mayan also automatically does OCR on everything that comes in, so even if the PDFs are non-OCR'd scans they're still searchable.

This is part of a larger project to significantly reduce the amount of paper that I'm keeping, which is why I'm using a document management system as opposed to a Dropbox folder. My goal is to divide the mounds of paper into things I need to keep for a long time (e.g., tax documents), and things that I can shred after a year (e.g., bills, receipts, etc). In all cases, I want the documents searchable and backed up.

[0] http://mayan-edms.org/

View on HN

Which scanner are you using? I've been trying to eliminate paper for a while now. I keep digital copies of everything I can download, but using a traditional flatbed/all-in-one scanner to digitize things that I on have on paper would take forever.

I'm terrible at organizing paper, and it's like a plague.

I'm just starting out, so I'm using an all-in-one flatbed scanner because I have one available. It's really slow, especially when I up the resolution enough that OCR works usefully.

My plan is to step up to something like a Brother ADS-1000W[0] or ADS-1500W[1]. I also hear good things about Fujitsu ScanSnap scanners, but those are USB only. If you step up to Fujitsu's fi series you can get a network interface, but I think they're more expensive than the Brother models I'm looking at.

Regardless of the brand and model, the required features are that it has a document feeder, scans both sides of the page, and that it can save the output to a server without needing a computer to mediate it. The Brother scanners have WiFi and can save directly to an FTP server - Mayan can pick up PDFs from a filesystem directory, so it would be pretty easy to have stuff go right from the scanner into Mayan.

[0] https://www.brother.ca/en-CA/Scanners/11/ProductDetail/ADS10...

[1] https://www.brother.ca/en-CA/Scanners/11/ProductDetail/ADS15...

Thanks, those were the models I was thinking about as well. Double-sided seems like a necessity, and a relatively fast scan seems important, but the only other thing I know I care about is durability.

I hadn't thought about using WiFi --> FTP. That's a pretty handy feature. I run Linux, and there is a lot of great hardware that is barely functional on Linux, especially from the likes of Brother.

I've thought about bringing paper to a staples to scan, or paying a scanning service to scan everything. The benefit of the latter is that they'll preserve your filing system and name your files for you. We went paperless at work, and they can be pretty expensive per document unless you have a ton of paper for them to scan.

Is 600x600 enough resolution?

Thanks for that though - I'm hoping it's available in Australia. I was previously looking at the Fuji Xerox DM4440 series which are much more expensive.

I use Fujitsu ScanSnap ix500. It's great: fast, reliable, just works. It has nice features like automatically orienting the document and built-in OCR.

That's a project I have too. A few things I struggled with, would be curious to know how you solved them?

- backups. I plan on using it to go paperless and keep tax returns etc. I can't afford to lose that info, but I can't come up with a robust enough backup plan. How are you doing it?

- concurrency. It runs on a 8 years old home server, and when I scan multiple pages documents, the ocr slows down to a crawl. Have you run into this?

It runs on a 8 years old home server, and when I scan multiple pages documents, the ocr slows down to a crawl. Have you run into this?

OCR is going to be CPU bound, so the solution seems obvious.

Indeed, I wasn't too clear in my comment. It slows down to a crawl because rather than queueing OCR, it does it all at once. 100 documents means 100 tesseracts running in parallel.

I would look into implementing a workflow.

Backups are one of those things that I don't do well at the moment. Fortunately, Mayan doesn't have any special requirements - basically back up the database and back up its media directory[0].

My ideal solution is to have everything backed up "in the cloud" plus have a backup at home. So far I've got the home backup happening - every machine uses 'attic' to back up onto my NAS. I'm still figuring out what the "in the cloud" side looks like.

One thing you can do is keep copies of important documents elsewhere - like Dropbox, a USB drive in a safe deposit box, etc. It all depends on what technologies you trust and how much you trust them.

For OCR, I can't say I've had much trouble with it. Mayan breaks the PDFs into pages, then feeds each page to tesseract. All of that happens in a worker process and I haven't found that it has a significant impact on the use of Mayan. The OCR section for the document will be incomplete for a while though.

[0] http://mayan.readthedocs.io/en/v2.1.4/topics/backups.html

I don't trust dropbox, at all. I'd like to store encrypted backups online for the Mayan db and every document.

For the OCR, my issue is that if I import several documents Mayan will feed them all to tesseract in parallel. Then I have two available cores, but 40 tesseract workers and the system comes down because it runs out of RAM and CPU. How did you set up your OCR?

I just used the instructions here[0], with some minor changes (pathnames, db server, etc). I've never seen more than two tesseract processes running on my two-core VM, no matter how many documents I give it.

Maybe something changed at some point to limit the active workers to the number of cores.

[0] http://mayan.readthedocs.io/en/stable/topics/deploying.html

Mayan EDMS issue to add rate limit support to the OCR backend. https://gitlab.com/mayan-edms/mayan-edms/issues/287

I wonder if there was a change in celery that limits the number of active workers to the number of cores. This is the behavior I see on my instance.

That could explain it, it would be a nice addition

Reply on news.ycombinator.com