Zotero Better Notes – Knowledge management inside Zotero | Hacker News Reader

Zotero Better Notes – Knowledge management inside Zotero

github.com

263 pointsmickelsen99 comments

Obsidian has replaced all (work, home, mobile) note taking solutions everywhere for me. The last want from my side is removing the friction related to editing markdown tables (e.g. multi-line content in cells) - Obsidian has some plugins that makes things better but it is miles behind what, say, confluence wiki offers for tables.

Biggest issue I have with Obsidian is sharing notes with others. Conversely the crimes I am able to commit with the dataview plugin make it all worthwhile.

You can use https://github.com/KosmosisDire/obsidian-webpage-export to turn Obsidian into a static site generator, and deploy the html to Vercel or github pages for free

When tables get too complex (e.g. generated columns using dataview) I switch to embedding subnotes using the ![[]] syntax and "clean embeds" CSS [1].

[1]: https://kool.casa/notes/dft1156nwrunrsma9dmyegh/

This is a good tip but I'm referring to the inadequacy of markdown tables + Obsidian plugins that deal with that. I'm yet to see a solution that works as well as tables in wiki. I think Obsidian should focus on improving that as part of the core software.

Vrite was featured recently, I just mention it because I like how the editor does tables:

https://editor.vrite.io/

https://news.ycombinator.com/item?id=36446045

Obsidian should definitely take a page from that. I'm currently with a plugin they won't add to the community marketplace:

https://github.com/Stardusten/ob-table-enhancer

Vrite doesn't support mobile though, which seems important for OC's usage.

Thanks for the link to the plug-in for Obsidian! This solves so many of the problems I have had switching to Obsidian completely.

If only there were a way to enable a three pane layout in Obsidian, I’d be in heaven.

I think you can as it has both left and right sidebars. All settings, plugins, themes get saved in a dot subfolder from your starting point, so you can replicate that layout for other vaults if you want.

I like highly minimal interfaces, so I modded mine with the Border theme and few CSS snippets so it looks like this: https://cldup.com/LAeiG4q2ED.jpeg

Is it ok for you to pay 8$ monthly for sync between devices? Or did you use another way of sync or don't use it at all?

That’s my only gripe with obsidian.. I’d like sync between my devices that avoids central servers.

SyncThing can accomplish that.

Thank you - I’ll check it out

You can use SyncThing

Thank you I’ll check it out

Charging for the sync is their primary revenue model. Zotero is doing the same.

I don’t bemoan this approach that basically gives everything local away for free.

Having an easy way to sync is kind of a core feature of a note taking app.

I'm not sure how profitable Notion is though.

It’s a tricky one in this age of “free”. I’d ironically pay for them to allow my agency and privacy, I’m probably not _most_ people though.

Sorry whete are you missing agency and privacy? It’s desktop first software you can back it up and pretty much sync it as you like.

Specifically regarding Obsidian it’s just a folder full of plain text markdown files, doesn’t get much more portable than that.

I just store my vault in a backed up iCloud folder. Syncs to Apple devices automatically for free.

Does it work on iOS devices?

yes. This is the configuration I use.

I just use syncthing. Regarding keeping configurations in sync, I have a "template" vault where I install and test all my plugins first and have a script that copies the entire .obsidian folder (minus some specific files) from the template vault to all my other vaults which results in syncing of all plugins/settings. I did buy their Catalyst license though since I wanted to pay back something for all the good work they did.

$8 USD ($12 AUD) seems incredibly expensive just to sync your markdown notes around. That’s three times the price of my entire iCloud services (which apps other than Obsidian sync with fine, but Obsidian has all sorts of issues with it).

I use Resilio sync to sync the Obsidian vault on all my devices and NAS.

There's a plugin called Remotely Save to sync files in a s3-like service (I use the 10gb free backblaze offer for it).

Setting it up was a bit of a hassle but now it just works.

I mentioned my workflow in a separate comment here but I’d say for personal notes yes you can use Obsidian for everything.

Where Zotero has come in useful for me is to gather and sort academic journal articles and to be able to feed them into Obsidian.

I found the Obsidian mobile app to be really flaky. Unreliable sync and it didn’t feel at all like a native app (on iOS at least).

I'm not an Electron hater but I really want notes apps to feel as responsive as possible.

Ive been using Bear (now Bear 2 beta) for years for that reason, but I really wish it had a few features such as shared notebooks and a plug-in system.

I use synchthing so syncing is quite prompt. The android version works well.

This is first I've heard of Obsidian, and looking at the web page I sort of maybe get what it does. But to help me out, can you describe what you use it does for you. I didn't see any references to citations or the like that zotero is mainly for. Is it just a note app? If so, what makes it so good?

What it does for me:

- everything is markdown which I'm comfortable in

- I can use conventional structure of folders and notes within folders (not for everyone but it works for me)

- extensive support for cross-linking/embedding/bookmarks along with connection graphs/search (out of these, I had use cases for cross-linking and embedding which worked very well)

- nice responsive UI with support for themes/customizations (I particularly like the Nord theme). Allows lots of control like custom CSS etc for more fine-tuning

- allows customizing keyboard shortcut for practically any action

- quality of desktop and mobile (Android) application is pretty good and the look and feel is consistent

- extensive plugin system: I mainly use plugins out of the box but with stuff like Data tables plugins, you can automate/implement your own workflow (I am too lazy to use all that). Even with out-of-box stuff, things like the git plugin allows version control with minimal effort, smart links allows regex based rules to convert strings to URLs (like entering JIRA ticket no. which converts to clickable URL), MathJax support for entering formulas which render beautifully etc; there are lots of such quality of life benefits via the plugins which adds up.

- allows easy syncing either using their paid service or using syncthing (which is what I use)

Thanks for this. Would you recommend this to someone who is not already a markdown user or that a key prerequisite for this to be useful?

The only concern I have is that the content that happens to match markdown rules will get interpreted and the subsequent highlighting etc might surprise/annoy a user not familiar with markdown. E.g. adding a dash at the beginning of the line will get interpreted as a list item, typing asterisk will trigger the editor to put another asterisk and put the cursor in the middle with the expectation that you want to type bold text - you can press backspace to remove the extra asterisk and continue as per normal but if you are in the business of dealing with loads of asterisks/dashes at the beginning of the line, and don't care about markdown, you might be pissed.

All the approaches to handle this kind of thing are not as simple as turning off a checkbox. If that is a deal breaker, then this may not be the app for you. OTOH, if you don't mind occasionally dealing with that (or better still, picking up the bare minimum amount of markdown), it'll provide a pretty rich note taking experience. For typical content, I don't think this will be a frequent issue but then again, what is typical for someone may be atypical for me and vice versa.

Personally I think it is totally worth it but YMMV.

The iOS app is similarly well done.

Do you use syncthing to sync notes with your phone too? How do you set that up?

Obsidian has an optional paid sync plugin that works out of the box. You could use Syncthing but I don't think it does text file diffing? Obsidian Sync keeps a version history of each file

I have tried to use a plugin called Obsidian Git that automatically runs git commits and pushes on an interval, but I have found it frequently had conflicts and the visual for resolving those conflicts wasn't super helfpul. I ended up just paying for Obsidian Sync. I wish I had a solution that kept my data just to myself, but I don't care that much of Obsidian has my notes

Great writeup of the tradeoffs. Also, Obsidian Sync offers end-to-end encryption.

For my notes, it is only me who edits, so I don't have conflicts as such. If I edit notes on the phone while on the move, syncthing syncs to PC once I'm back home; similarly any changes done at home on the PC flow to the phone. I also use the Obsidian Git plugin (though I have disabled automatic commits and commit as and when I feel the need) - haven't had any conflicts so far.

It's also only me editing my notes and I have Obsidian git conflicts all the time.

Also, it freezes my typing every time it tries to push/pull. So if you put it on an interval, to make it more automatic, your typing will just be frozen every however many seconds

I do - Syncthing runs on both Linux (my PC) and Android (my phone). The configuration is same as any syncthing setup to share a folder (the top-level folder containing all the Obsidian vaults in this case) between two nodes (my PC and phone in this case) - their UI[0] is relatively straightforward.

[0]: https://docs.syncthing.net/intro/getting-started.html

What are you doing to sync your files? I'm current using the git integration and while it works it's a bit too manual for my tastes. I think I'm ok with last write wins as long as the sync stores history. The whole commiting changes, pushing, pulling, dealing with merge conflicts is a bit too much workflow for my notes.

Not parent, but I have had a good experience using SyncThing to keep my obsidian notes synchronized across devices.

I have a server at home where I self host a few services and access over tailscale so I have a device that is “always on” and syncing, but prior I used a $5 droplet for much of the same. Can recommend.

I second SyncThing in combination with Obsidian. Works like a charm.

I just setup Obsidian a couple days ago and have been using https://github.com/vrtmrz/obsidian-livesync which seems to work ok. I'm curious if SyncThing would be preferable. How does it handle conflicts? How often does it sync? Does it drain the battery on mobile if it syncs too often?

I cant speak to mobile personally as it is not a use-case for me but as far as conflicts SyncThing works great.

I have been running it for a little over a year now across three primary devices and the server and I don’t think I’ve seen more than a handful of conflicts. Don’t remember a single one with my obsidian directory which is currently ~500 files (markdown/pdf etc).

I’ve found that the official Obsidian Sync solution performs well and has no issues with conflicts, unlike iCloud Drive which I used prior. It does feel pricey for a single feature, but I just frame it as valuable support for an amazing product that I’d otherwise be paying nothing for. After futzing around with sync for a while I was happy to throw some money at the problem and move on.

I use iCloud to sync. Just locate the vault in an iCloud drive folder called Obsidian. Works beautifully to sync between my different Macs and iOS devices.

Drop the folder into Nextcloud.

I don't think people should be making tables, they should be making data, which they put into a table via a query.

I ran with this for my own note taking system, mdld. https://github.com/vid/mdld , which works with hedgedoc.

A Table is data, don't make things unnecessary complicated. Some data are better managed directly in a table, if you have a good table interface, others are more natural managed in separate files. Each case their own.

And how does mdld differ from dataview?

Huh! I hadn't seen dataview before (I think mdld predates it), I'm more than happy to use another work if it's a supported superset. mdld is sort of inspired by Semantic Mediawiki, dataview uses SMW's :: notation, which is a good sign. However, I don't see a few features that mdld has, like concepts, which are very useful, and the hedgedoc application of mdld has some nice dynamic views, for example, creating a gantt chart out of data dispersed across markdown files. In general, its emphasis is creating things out of data to support features like round-tripping, rather than creating self-contained black box information views. But its more of a proof of concept, so I'm going to look at adopting dataview.

The problem I have with obsidian is I cant use it on my corporate machine which I'm forced to use all the time.

I don't think both serve the same use cases. Or can you use obsidian or does a plugin exist to annotate and read pdfs?

I use this one: https://github.com/elias-sundqvist/obsidian-annotator

Native PDF search and frictionless adding of metadata to non-markdown files are lacking also.

Zotero is great, but only if enhanced with plugins and a few settings. I don't use Zotero Better Notes so far, but here's my list:

    - [Opinionated decision] disable sync
    - [Opinionated decision] set “Base directory” (Preferences > Advanced > Files and Folders) to local literature folder
        - every time you download literature PDFs, create a new subfolder with the motivation for search,
        - prepend folder names by date-string (yyyy-mm-dd_look_for_albert_einstein_lit)
    - set PDF View to “System default” (Preferences > General > “Open PDFs using..”)
    - Enable recursive quick search in folders:
        go to Preferences > Advanced > Config Editor, search for `recursiveCollections`, double click (set to True)
    - use CTRL-SHIFT-DRAG to drop files into Zotero as Links, see [#77](https://github.com/zotero/zotero/issues/77)
    - use CTRL-Shift-C to copy bibliography to clipboard
    - Dark Theme [1]:
        - Go to `%AppData%\Zotero\Zotero\Profiles\` (`XXXXXXXX.default`)
        - Create `chrome` folder
        - Place the `userChrome.css`
        - Start Zotero
    - Add-Ons:
        - zotero-pdfkit [2]
            - allows to modify/select a “default” PDF attachment to be opened
        - ZoteroDuplicatesMerger [3]
            - easier merging of duplicates
        - zotero-folder-import [4]
            - bulk import PDFs from a folder
        - zotero-tag [5]
        - allows to add stars to items (Num Key `1`, `2`, `3` etc.)
            - example see [6]

    [1]: https://github.com/Rosmaninho/Zotero-Dark-Theme
    [2]: https://github.com/sharpevo/zotero-pdfkit/
    [3]: https://github.com/frangoud/ZoteroDuplicatesMerger
    [4]: https://github.com/retorquere/zotero-folder-import
    [5]: https://github.com/windingwind/zotero-tag
    [6]: https://ad.vgiscience.org/links/imgs/zotero_stars.webp

Thanks for sharing these. I already use a number of these on your list. Though I don't use copy-by-link for files. For me one of the key purposes of zotero is that its the place where I keep these files so I don't have to keep them elsewhere where they aren't as easily searchable.

I'm excited to try the duplicates merger tool, I've been forever putting off this because the normal way is such a slog. The Tag one also looks good.

To add to the list, my key add-ons are:

- better bibtex, this one is essential for me as a latex user

- zotfile. This one is cool because if you add annotations to a pdf, like highlighting or comments while you read it, then zotfile will pull those out into notes in the zotero entry. This makes it really easy to see both that I've read this document, and what parts I thought were important.

[1] https://retorque.re/zotero-better-bibtex/) [2] http://zotfile.com/

Yes, I think this needs to be decided based on preference. I like to be independent in my file/pdf organization, and also search without Zotero through these files. I further consider original file names important and finally sync my literature folder to my Nextcloud, with full-text search. This is why I prefer linking, instead of file management by Zotero.

Zotero was key to finish my long delayed thesis, reducing the toil of managing a lot of references on a topic that I disliked more and more by the minute. Unfortunately back in the day they did not have yet the embedded PDF reader and highlighting features, but still was super helpful (I decided to stay away from Mendeley for license and affiliation reasons)

I still keep Zotero around for some projects that require more serious bibliography management. Glad to see advances in the note management direction, even if it is a add-on. While serviceable, the default note options always felt a bit rough to me.

> Zotero was key to finish my long delayed thesis, reducing the toil of managing a lot of references on a topic that I disliked more and more by the minute.

Exactly the same for me, I'm forever grateful.

Same. Hearing Zotero brought back memories of undergrad for me. lol

I’m using Zotero + Obsidian for my academic work and PhD.

If anyone is trying to work out a Zotero to Obsidian to Word/Libre Office workflow, give it a look.

I’ve created a GitHub repo README with instructions on linking everything together.

Hopefully someone finds this useful, happy for more contributions also (hence why used a repo to do this)

https://github.com/evolve2k/obsidian-pandoc-academic-word-do...

It looks like a powerful combo. Back in the day I went the LaTeX path, nowadays I’ll be tempted to follow your approach.

I have tried Obsidian a couple of times, but it never stuck with me. For some reason I cannot go back to “regular editors” after years of vim (and now neovim), it is not only the modal editing (IIRC Obsidian has a decent vi mode), I guess it is more about having all your editing in the same tool (and the terminal).

Nowadays I stick to neovim plus Telekasten plugin, not as polished as Obsidian, but with enough features.

+1 for Obsidian, though until now I haven't felt the need to write actual papers in. Zotero + latexmk + vim + Skim/SumatraPDF mostly scratches that itch for me.

any examples of generated pdf ?

Oh good suggestion. I can’t get to it straight away but happy to add one. Feel free to add an issue in the repo, then I can report in once it’s added.

In practice the markdown to pdf conversion is performed by pandoc. Pandoc has functionality to recognise academic formatting styles (like APA, Harvard etc)

Pandoc also has functionality to turn a table of contents on based off of the semantic headers.

My guide is pulling together issues that arise across Zotero <> Obsidian <> Pandoc <> Word

I’ve found most forums are concerned with one or two of these steps depending which forums you’re visiting.

Nice! This seems to actually solve a problem in my workflow with Zotero: manage the _personal_ metadata around the references. How did I find the reference? What parts are relevant? Etc. Definately going to give it a try.

Yeah, I just stumbled upon this project and wanted to share, I'm currently using Obsidian for my personal wiki, but I use Zotero a lot as a paper repo and reader, the organization and metadata tools are great, and extending it to a more powerful note-taking tool seems like a no-brainer.

Now it just needs an EPUB reader to replace Calibre, then it'd just be the perfect all-in-one personal library. For now I'm using this plugin that exports and keeps in sync the calibre library to Zotero:

https://www.mobileread.com/forums/showthread.php?p=3339191

Very grateful that this open source project stays alive, I've seen attempts over the years from startups and other projects to tackle on spaces like pkm, research, paperless office, to then be abandoned yet Zotero keeps getting updates.

There's also Tropy, from the same organization that develops Zotero, for organizing digital assets:

https://tropy.org/

Getting a bit off-topic, but this thing could use some sort of Moodboard designer to visually sort the assets in a canvas, kind of what you can do with Miro, Notion, Mural or locally with Obsidian Canvas/Excalidraw. On that note, there's also this project that takes on that space, previously featured on HN but has made significant progress lately:

https://github.com/toeverything/AFFiNE

I also think an epub reader in zotero would make it a complete library manager. I used Calibre just for reading epubs, so I want to ask how and why do you need to sync its metadata with zotero?

I just like having the entire collection of files in one place, some of them are textbooks or reference books that don't sit well in either app entirely, and while I also keep novels there in a separate subcollection, most of my ebooks are non-fiction, where topics are the same or related to the rest of material I already keep in Zotero.

Some are non-academic PDFs, blueprints, guides, whitepapers or manuals that I refer to for a given project, and while this is not the main purpose of Zotero, as it is a research-oriented tool, it still proves more useful than other tools, tidier than classical folders+files, and less of a hassle than coming up with my own system using something like Obsidian's dataview plugin.

Calibre I prefer it to fix the metadata of my ebooks, but the local reading story isn't ideal for me. I've tried serving the collection over self-hosted web apps, to keep the page progress like calibre-web, but it wasn't ideal for me or conducive to what I look for: https://github.com/janeczku/calibre-web

IDK if it helps you, but a simple epub reader is Bookworm.

It's lovely but I'm on a Mac. For now I've been switching between Google's reader and now trying Yomu (https://www.yomu-reader.com/) but ideally I'd use something open source.

I’ve just started using zotero again after 10 or so years, being back at uni for some further learning. I’m so impressed that zotero has remained useful and these plugins look super handy, will be giving them a red-hot go. Thanks for sharing!

For those considering a commerical solution, DEVONthink handles personal knowledge as well as PDF and other files, split into separate "databases" that can sync across local WiFi to other devices running DEVONthink.

I recently discovered it can handle Markdown files and link across entries.

Search is powerful and fast. There are very few sync issues.

Downsides: MacOS/iOS/iPadOS only, and not free or open-source (but it's a one-time purchase).

Not affiliated, just a happy user.

The strong point for me is that DEVONthink uses RTF (and RTFD) as the native format. RTF is, along with Markdown, a document format that I expect to be widely supported for the next 40 years, but has the advantage of being WYSIWYG and supporting more advanced formatting. RFTD supports embedded images - it is less widely supported, but there are third party editors. The "databases" are MacOS packages, and they document how to get the files out if DEVONthink becomes unavailable. Hence for me it gives a good balance of functionality and longevity.

I like to annotate pdfs by hand, so import from Zotero into Notability on my ipad. Does anyone know of a good way to automatically reintegrate hand annotated pdfs back into Zotero? Ideally a given Zotero reference entry would have two pdfs, original and annotated, as well as a .note file. (The various plug in options seem only to manage text annotations.)

I realize that below I'm not answering your direct question (and instead just talking about the text-based annotations), but I wanted to document it because I went from handwritten to highlighting and typing (clumsily on the ipad)

I used to do this myself (though with GoodNotes), but I'm switched the Zotero in-built PDF viewer.

The annotations you make in the in-built PDF reader are separate from the PDF file itself to: 1. Easier syncing - the annotations are light, rather than syncing the entire PDF > "We'll always try to support external workflows as efficiently as possible, but it will never match the seamless experience we're able to provide when everything is done within the app." [1] 2. The internal (text-based) annotations show up as notes, which has a neat effect that when I started using BetterNote this morning, it already has every one of my annotations ready to be used. I was surprised and a little delighted (there's a tiny button on the tab bar with a magnifying glass[2], it show all your notes/annotations from EVERY file and lets you link them to your workspace note). 3. You can export your annotations onto the PDF if you want to store them on the PDF itself.

The bad: handwritten annotations don't show up in the notes list (in the Zotero 7 beta build I'm running, at least)[3].

[1]: https://www.zotero.org/support/kb/annotations_in_database [2]: https://app.screencast.com/gv9JBZNpJyoSF [3]: https://app.screencast.com/0YQjZSV54R1Ha

Thanks for this information — I’ll give the built in Zotero annotation took a closer look.

I just updated Zotero from the AUR and, during the build process, the pdf.js dependency reported 25 vulnerabilities, of which 5 being critical. Given that we sometimes pull pdf from questionable sources/websites, should that not be a priority to fix? Or is there a standard mitigation process that I am not aware of?

Zotero is awful. It's heavyweight - in fact, a whole separate browser only to manage my bibligraphy (ever tried to use it with ~10k items?). Provides no CLI interface (to automate workflows with scripts). It's data model is weird - e.g., item types include such thing as "instant message", but more academia-relevant types (e.g., "dataset") are absent ("preprint" was incorporated only recently); ISSNs are treated as properties of articles rather than journals. And it boasts it has included its own PDF viewer, whereas I see it as down-side rather than improvement (more bloatware included - why not just follow the division of labor and delegate this functionality to external viewers... ahhh, yes, there is no annotating standard among PDF viewers...).

But all other alternatives for bibliography management I know of are even worse.

I had written a script to manage my pdf literature using crossref and arxiv to fetch metadata and store them in pdf tags, and just renaming the pdf in place. No hidden complexity, no gui.

I hacked it into a python package a couple of months ago in an ADHD hyperfocus induced frenzy. Sounds like you might like it. Still a long way to go, but it's out there and FOSS. Contributions are welcome.

https://github.com/jayghoshter/prem

> ever tried to use it with ~10k items?

Yes, around 13k currently. Startup is slow, but I keep it open so that doesn't matter much. With Zotero 6, speed isn't great for general usage, but workable. Zotero 7 (beta) however feels very fast. I've switched to it already -- most plugins haven't been upgraded yet, but (at least currently) you can switch between v6 and 7 since there's no database updates involved.

Seems pretty similar to Joplin, but Joplin has tons of sync options and an Android version

I just recently discovered Zotero and it makes my life so much easier. It finally makes sense to read PDFs and work with their content, when I know where to store the notes and find them, when I need them.

This seems like the best place to ask, does anyone have any recommendations for a very simple notes tool that can be easily shared amongst a family or team?

Preferably mobile friendly, Even if just in browser.

I like Joplin, and the sharing is done via Dropbox or a number of other backends. Have a Dropbox shared folder that everyone has access to, and you can edit notes. Joplin is a great note taker in general, as well, and the mobile app is great.

Joplin isn't that easy to setup for family. It also doesn't work like that for Dropbox, it links to a specific app folder on your own account. I'm also not sure about distributed editing it would result in a lot of conflict notes that you manually must fix.

Just use Notion or Google Drive folder with docs if you don't have too many notes.

Assuming you're well aware of something like a shared Google Doc and you need something a little more fully featured...

For "easily shared" it's hard to beat Notion. They have a really good mobile experience and the ability to control permissions and sharing at a detailed level.

The only drawback with that recommendation is you also asked for "very simple," which Notion is definitely not. But, you don't have to use all the features available--you can use the complicated tool in a simple way if you like.

for Apple ecosystem, the Notes application works well for "simple" shared notes

Does it give you a way to link a fragment of text in a PDF to notes about it somewhere else and navigate between the two? This is the biggest pain point that I basically solved manually in my PhD with a kind of made up system with hash tags embedded in PDF notes. It's such a painful process maintaining a collection of citations around a particular topic, and being able to map what papers support what pieces of knowledge along with context around that. Still seems relatively unsolved problem to me.

That's what it does, see its readme

I love Zotero. It's the only tool I use that is an .xpi.

What are some learning resources for this programming system?

I used to use and recommend qiqqa for full text search, highlighting, and annotating pdfs and managing bibliographies but it seems like zotero has stayed the course and qiqqa wound down. Glad to see a solution still available!

How well will this work when accessing Zotero notes via the Mobile App. Will the notes still be viewable from iOS Zotero App, even with more limited functionality?

I've only ever used Citavi during my BSc and MSc.

Installation does not work, unfortunately

Reply on news.ycombinator.com