Comment by NL807 | Hacker News Reader

Comment by NL807 | Hacker News Reader

I just finished importing and sorting thousands of PDF research papers, about 50 GB I've accumulated over the decades. Originally, I was sorting them in file system directly, but the whole thing got out of hand at some point, and thus the Documents folder became a dumping ground for PDFs. So I decided to take control of the situation and use Zotero to manage all this stuff. First I used Claude Code to generate all the BibTex files for the PDFs, then sort them into categories (still on the filesystem). Then created a master bibtex file that lists all the docs in each category. I used that to bulk import in Zotero, which then normalised all the file names once imported. What's really cool is that you can use Zotero built in web APIs to get an external tool to manipulate your collection directly. I've also setup WebDAV to sync the entire collection across multiple machines. Can you do all these without Zotero? Absolutely. And I did, but can't say I'll be looking back any time soon.

At least with Zotero Version 6, you could have just selected all your PDFs and drag-n-dropped them into the interface.

Like another poster here, trusting an LLM with my reference database -- the ultimate source of truth -- is not a step I'd be willing to take. All it takes is a single hallucinated reference, and your career would be forever tainted. It's not worth the risk.

Sadly, Zotero seems to have removed this killer import feature in later versions, which is the reason I keep using version 6. It feels like later versions have been a route to dumbing-down the interface, prioritising simplicity (and an ultra-low-contrast interface) at the expense of functionality. (If you can still drag-n-drop PDFs straight in with the new versions, someone please let me know?)

I still drag and drop PDFs in Zotero 7. It's less finicky then v6, even. The trick is to drop them to root of the tree on the left. Not to the middle of the interface.

I drag-drop PDFs and ePubs, read PDFs and ePubs, and sync mobile with laptop, etc.

Drag-and-drop of the PDF file worked for me in Zotero 9 (latest version). I never used this feature before. This would be greatly preferred to my earlier suggestion to get a LLM to generate a list of DOIs.

>At least with Zotero Version 6, you could have just selected all your PDFs and drag-n-dropped them into the interface.

I'm on version 9. I did the drag and drop for some of the files, but Zotero has only like 50% hit rate with finding the correct bibtex, and those are not that high quality either. People talk about hallucinations using LLMs, but I found them to be more reliable and produces richer metadata than Zotero's system. You can even use the Beaver plugin to correct the metadata, if you find them dubious or low quality.

>All it takes is a single hallucinated reference, and your career would be forever tainted.

Of any references I use, I always read the PDFs. The entire point of me using Zotero is to manage PDFs and being able to read them. If the title and author data in the item view don't match the PDFs then it's easy to notice. And even if i get the reference wrong, i'm not worried about my career, because it wouldn't matter.

> First I used Claude Code to generate all the BibTex files for the PDFs

I think this has an unnecessary risk of hallucinated bibliographic data. For anyone doing something similar in the future, it would be more reliable to make a LLM generate a list of DOIs and have Zotero import the DOIs.

Using a non-deterministic machine on something which needs to be pretty accurate and precise looks like playing Russian roulette with one's academic integrity.

It's not my cup of vodka, to be sure.

And moreover, something already mostly solved algorithmically. Eg in zotero you can drag and drop files and it generates a reference based on the metadata as long as that is enough (somebody mentions it is not possible in newer versions, I use the latest version and it works fine).

Each one can choose their own tools of course, but we often see llms shown as solutions to something presented as unsolvable before, while it already was pretty amenable to normal tools.

> but we often see llms shown as solutions to something presented as unsolvable before, while it already was pretty amenable to normal tools.

I see that as a symptom of being intellectually lazy, or not knowing the value in mastering a tool via deliberate use over time.

Once one invests the required time to use a tool properly, they can reap way more benefits from said tool when compared to jumping tools at minute's notice.

> And moreover, something already mostly solved algorithmically. Eg in zotero you can drag and drop files and it generates a reference based on the metadata as long as that is enough

As mentoned somewhere else, zotero's metadata retrieval is like 50% hit/miss in my experience - as in for the papers i've imported it was able to ID about half of them. An the quality of that data is not that great either. Lots of OCR errors, sparse bibtext fields, author names wrong, etc.

You can always get the missing data with a DOI, which is always present on the paper if it's recent enough.

I have tried filling some missing fields in my books list with help of an LLM, and none of the books were rare. I allowed it to search the internet and it hallucinated so bad that I nuked that version of the list and reverted all the changes.

It's more accurate with all the holes in the data.

> I allowed it to search the internet and it hallucinated so bad that I nuked that version of the list and reverted all the changes.

Really? This does not reflect my experience at all. I've been using the Beaver plugin and also Claude Code, and they have been spot on.

Probably due to language barriers. The books were not written in English and not authored by authors living in English speaking countries. Mostly were non-fiction, too. Mostly history and neighboring genres.

None of these issues are relevant to me.

I'm a graphics programmer, i keep a lot of journals for research, and i needed the least painless method to get them sorted. Even if, say 1% is misclassified or has wrong metadata, it will not impact what I do the least bit. And those can be corrected with ease.

That is a sensible way of turning chaos into order, but let a contrarian voice tell an alternative story. I think filesystem pdf organizing is underrated. I have been curating a tree along years and now I know where to park new pdfs by heart in there. Wrote an arxiv button adder with grease-monkey to save pdfs with title, year and authors as file-name. Windows have a nice abstract property system that can use pdf (bib) metadata by installing a shell extension, that you can use in Explorer, where you can customize displayed fields. Classic Windows search offers a good background indexed database of content and metadata for local files. You just need an "IFilter" component to let it see inside the pdf text layer (suamtraPDF cames with one). It is the kind of functionality one would expect of a commercial product, just for the windows license. Beyond File Explorer, You can query the index from scripting or dot-net. I wrote a flow-launcher plugin to making searching faster than from Explorer search text input. Search supports "saved searches", saving queries in xml. I use a lot two, listing the saved pdfs of the last week and the last month. Windows is weird in having that cute infra built-in, but just having left unfinished the last glue code needed to give some good user experiences managing docs.

Reply on news.ycombinator.com