Mendeley encrypts users' database after Zotero provides an importer
zotero.org
zotero.org
Mendeley used to be quite a good program, but recently you can not export annotated PDFs meaningfully. For example, sending a folder of annotated PDFs to a co-author during a literature review is impossible. This is obviously the case since Elsevier does not want you to trade research papers, whether you have lawfull access or not.
The updates that took away features were silent. What happened to me some time ago was these updates occured during a high-stress phase with a short deadline until conference submissions (if you are a researcher, you know what I mean).
I had used Mendeley for years to annotate and categorize literature. I was now in need to send my categorized PDFs to a central repository for my co-authors to evaluate and add to. After some update, without me noticing it, it was no longer possible to export folders of PDFs or PDFs in general!
I had everything in Mendeley, weeks of work. I was completely f'ed - deadline approaching. I had to re-aquire all PDFs and go through all annotations by hand.
ELSEVIER IS SIMPLY ANTI SCIENCE. Collaboration is a key in science. Sharing results, research and literature is crucial.
Mendeley makes this impossible. It does NOT allow you to fully access your own work!
So in conclusion, USE ZOTERO. It's good now, better than before. You can use a PDF reader with annotations to open and save the PDF and Zotero will keep those annotations. You can export Bibliographies, including notes AND files. You can not do that with Mendeley.
So again, as a researcher, I emplore you to drop Mendeley completely, as I have done.
Thank you.
However, as someone who worked in a large IT organization for a huge company and wrote software for license compliance tracking, I completely understand. Grab me a pitch fork, I'll march.
This is exactly why you should not depend on proprietary software for anything even remotely important. This is not the first nor will it be the last time that something like that happens. Letting a company (or an individual) dictate how and if I can access my work is unacceptable to me.
Thankfully Zotero is FOSS, I will be staying with BibTeX though.
In any case, the Zotero plugin works great and just saves Metadata and PDFs, as it should. For the usual preprint or paper providers, it works perfectly.
So even here, using Mendeley is just bad.
Mendeley's advantage was that you could just drag and drop a pdf in and it would add meta-data.
I am happy to report that Zotero does this as well now. And it works quite well, sometimes better then Mendeley.
So if that was your reason, like it was for me before I was hit with the "no export disaster", go ahead and switch.
Let Mendeley die the lonely death it deserves.
- The way that it still doesn't play nice with cloud services (syncing the directory and its just a matter of time until you get database corruption. It takes a lot of wonky setting up to get it to kind of work, which just shouldn't be the case
- The lack of developers and thus slow pace of improvement. I'm a researcher not a programmer - which I think describes most people using it. That means unfortunately we are reliant on one or two volunteers to improve the product. The pace of improvement is slow, and theres also no way to meaningfully advance it - be that through offering bounties for someone to implement certain features, just inputing lists of bugs/feature requests (the list is already v long, and doesn't move much), or anything else.
It's a really good bit of software (and I don't want to sound ungrateful), I just know it still has a lot of quirks. This means it can't always do what you want, and it isn't an easy obvious choice for new researchers - Mendeley is certainly more familiar and easier to use.
Of course Zoteros development is less funded and less agile. Given that, I think they have worked on many shortcomings. The interface is now good, the group-based sharing works, PDFs are read and meta-data is added well. The import plugin is better than that of Mendeley.
Mendeley has an advantage in that it has a great PDF viewer and editor. But since you can not do anything with these PDFs and annotations, like export them or send them anywhere, it's now pretty much useless.
Switching is not easy, but in the long run I don't think you'd be faced with much issues going from Mendeley to Zotero. It was certainly worth it for me.
As it stands, you can still import from Mendeley to Zotero, so I'd at least do that now, until Elsevier finds a way to shut this down completely.
Having never set it up with Google Drive, I'm not sure of anything else that could help your situation. In that sense your original point is quite valid then - this must be a quirk that still needs ironing out.
My system does seem to work pretty well, I just prefer it when things are very much plug & play. It stops you screwing anythign up, and makes it easy for non power-users to get things done.
I'm using Zotero's $20/year for 2GB space membership and it was quick and easy to set up. Works great too.
I'm not sure why you have that impression. Zotero has amazing, invaluable volunteers, but there's a paid, full-time dev team working on Zotero every day. In the last year, we've added:
- Google Docs integration [1]
- Unpaywall integration [2]
- A new, greatly improved PDF recognition system [3]
- Faster citing in large documents [3]
- A much more powerful saving interface [4]
- Mendeley import...
- ZoteroBib, a free web service for generating bibliographies [5]
- A barcode scanner for iOS [6]
- Regular updates and bug fixes [7]
[1] https://www.zotero.org/blog/google-docs-integration/
[2] https://www.zotero.org/blog/improved-pdf-retrieval-with-unpa...
[3] https://www.zotero.org/blog/zotero-5-0-36/
[4] https://twitter.com/zotero/status/991052142717886464
[5] https://www.zotero.org/blog/introducing-zoterobib/
[6] https://www.zotero.org/blog/scan-books-into-zotero-from-your...
[7] https://www.zotero.org/support/changelog
(Disclosure: Zotero developer)
It is fair to say though that it isn’t as well resourced as others, and is also starting from behind. As long as that momentum continues it should eventually be the de facto solution (iff Mendeley are going to make user unfriendly choices), but as yet I don't think it is comparable to the behemoths like R that have maturity and continuous development and thus are superior in every possible way to the paid alternatives (Stata, SPSS).
It's a bit of an anti-feature. We can't really control the data store if you're monkeying around with it under the hood.
Polar already supports cloud sync so we encourage users to use that. Same with Zotero I imagine.
- The lack of developers and thus slow pace of improvement.
Do you mean the dev pace of Zotero is slow? I wonder if their legacy infra is slowing them down.
FWIW, I think they brought back exporting annotated PDFs in the most recent version, 1.19.3
At least maybe there is hope for exporting your work to Zotero.
Can Zotero respect my filesystem organization (à la Lightroom)? I have thousands of papers organized, and don't want to move them all in (too much work, what if I don't like it?), but I also don't want to have two copies of some subset of the papers, which might not be in sync.
Respect the filesystem? Erm, again we're in the world of the murky where it probably is possible with Zotfile, but out of the box, it wants to rearrange them in to folders like 'MN8YD'. I've made my peace with that, but if you are very touchy about it, try it with a copy first and see how you get on.
+1. I worked in publicly-funded research labs for 15 years and there is no single organization I despise as much as Elsevier - only Springer-NPG comes close. If the company went bankrupt tomorrow it would be a great day for science.
It looks like they're "encrypting" by saying that there is some sort of GDPR requirement when in reality it's more plausible that they're trying to lock in users.
Zotero = Your personal research assistant. Zotero is a free, easy-to-use tool to help you collect, organize, cite, and share research. https://www.zotero.org/
Mendeley = Reference Management Software, produced by Elsevier who also happens to be the publisher of many peer-reviewed journals. Elsevier come under fire for it's high costs and gateway actions to restrict access to information they've published in journals and host in archives. This most recent action of making the database of references in Mendeley difficult to export is a continuation of their attempt to protect what they, and some legal systems, would see as their IP. Others disagree.
The battle continues...
I read a scientific paper or look up a citation. I add that with a click or two to my reference manager. It also stores the PDF for me.
In the future, I can easily re-read the PDF. I can annotate it and the annotations will be stored.
Critically, when writing my own paper, I can import those citations and trivially change the format to whatever the publisher wants without any effort.
To me, across most features, these two programs do exactly the same thing. When picking a citation manager, it's more about which one I trust will be around for the long haul and will not interfere with my research.
Mendeley opens PDFs externally, but many PDF readers can create and save annotations.
I have nothing but good things to say about about Papers for Mac though.
Mendeley and Zotero basically do the same thing, which is categorizing literature - specifically research papers. Mendeley offers a built-in PDF viewer with annotations tools, Zotero relies on external viewers for that. Both offer cloud storage.
Researchers usually have to categorize and scan over huge amount of prior literature. If you are working on a project, you usually want to have a good library or folder structure detailing relevant literature. When writing your paper, you also want to be able to cite from this library quickly, via BibTex for example.
Researchers maintain a personal library with literature they read, plan to read or have cited. When researchers collaborate, they need to merge these libraries. Since tools differ, the lowest common demoninator is often a dropbox somewhere. Sometimes everyone works on Zotero or Mendeley, such that files can be shared within the system. Usually not.
Mendeley and Zotero used to be equally open to sharing and collaborating. Mendeley was in a sense superior as it was polished, had great tools to annotate and great tools to work with Meta-data. Zotero, while FOSS, was always behind.
Then, Mendeley was bought by Elsevier, the largest publisher. Elsevier does not want people to share PDFs, because Elsevier wants everyone to pay for the priviledge to download those pdfs. Thus, Mendeley started to make it more and more difficult to collaborate. Now it is even difficult to share files within the Mendeley eco-system!
Perhaps you are using Calibre for you Ebooks. Now imagine Calibre would deactivate any way for you to view, send, export or use your files (the files that YOU put into it) outside of the Calibre Ebook viewer, and it would encrypt its database so that you would not even try. That's what happened.
Sharing research is the lifeblood of science, and Elsevier wants to destroy it.
Elsevier has done many other things that has harmed scientific progress - the majority of this undertaking you fund with you tax dollars! The way this is done is almost comically blatant. Elsevier acts like a comicbook villain. They are literally evil.
There aren't that many SQLite encryption libraries around (generalising), so it's probably using either SQLCipher or SEE.
https://www.zetetic.net/sqlcipher
https://www.sqlite.org/see/doc/trunk/www/index.wiki
There's a reasonable chance the page size and passphrase is stored or cached on the machine. If someone (not me) takes the time and effort to trace through the application, it should be feasible to figure out what's going on. :)
I ended up adapting some code that scans an OSX process for AES key schedules and prints any keys that it finds (caveat - sqlite3 rolls its own AES and the key schedule ends up byte-swapped in memory).
I also had to write some Go code to decrypt the sqlite3 database file. There is code to read an encrypted database buried in sqlite3's revision history, but it's an old enough version of sqlite3 that it didn't understand dropbox's schema.
As an alternative it should be quite possible to produce a hook using Frida or similar that would disable database encryption on the Mendeley side so it can be imported through the usual channels. Not pretty though.
The idea was to have Polar also work with Zotfile since a lot of people use Zotfile.
I'm not a programmer however, so this may be very naive question!
My reason for wanting to switch was that Zotero has Google Docs integration. After making the switch, I was pleased to pay a modest amount for storage of my PDFs, which makes me feel like a customer instead of a product.
> It includes a Word plug-in, from which you're able to directly search PubMed [...]
That sounds like it's at least focusing on medicine as a field (and generally fields in which Word documents are an acceptable means of dissemination...)
Regarding open source: Being open source is certainly not core to the functionality of a reference manager. But it does protect you from exactly the kind of behaviour shown by Mendeley here. So recommending a "solution" for the situation that has exactly the same drawback as the Mendeley (i.e. vendor lock-in) is short-sighted in the extreme.
If it helps I have just under 2000 papers in my Zotero. It isn't massive by research standards, but it is substantial.
[1] http://zotfile.com [2] https://retorque.re/zotero-better-bibtex/ [3] https://www.zotero.org/support/google_docs
Asking how that could lead to Elsevier obtaining evidence of paper piracy is like asking how using gmail could lead to google using your email to do targeted advertising. You are giving them everything they need to do it, so of course they might do it! The position that needs defending is the supposition that they can't.
It's probably not that difficult to reverse-engineer and extract the decryption keys, but doing so opens you up to DMCA risks.
I re-un-recommended Mendeley back in June 2018 when this news first broke. (I was initially too sketched out by Elsevier to recommend Mendeley, and this just confirms these suspicions.)
tl;dr Zotero is a pretty good bet for most people, especially since they added support for citation management in Google Docs at the end of last year (very important for academic writers). PaperPile (https://paperpile.com) and Papers (https://www.papersapp.com) are also worth checking out.
Zotero will let you send PDFs to/from a tablet with the ZotFile (http://zotfile.com) plugin. This works well if your workflow is (1) find PDF on your computer; (2) read on tablet. But if you want to do anything else -- even choosing a PDF to read from your tablet without touching your computer -- then Zotero won't work. With that said, I know Zotero is working on mobile apps...not sure how far they have come though.
If you're Mac/iOS only, also check out https://www.sonnysoftware.com/bookends/bookends.html.
Also, Bookends annoyingly checks to see if you are running the same license on multiple devices on your network, and if you have more than once instance running it forces you to close it down. This may seem superficially reasonable, but for someone with both a desktop and a laptop it's quite annoying. I emailed the developer about it and they didn't seem to get why this was a problem.
I like Zotero, but the lack of an ipad client is annoying. The file plugin requires more forethought than I really like. Bookends having a mobile client was one of the attractive features.
What is the work that it takes to get set up?
Looking at their website, it does seem like Citationsy is somewhere in-between a full-blown reference management application and one-off bibliography creation applications.
What Citationsy does not appear to do is manage inline citations within a Word or Google Docs document. Zotero will let you hit a keyboard shortcut, search for a reference, and insert a citation right in a paragraph (e.g. "(Smith, 2001)") . It then takes care of also adding that citation to the bibliography at the bottom of the document, and keeping the numbering in sync (if your citation style uses numbering).
I would only use a reference manager that has a word processor plugin for inline citation management -- and it doesn't look like Citationsy does.
To quickly build a bibliography, or just get a properly formatted citation for a single paper, I like https://zbib.org (also from Zotero).
Edit: clever, clever, parent poster is the creator of Citationsy. Hello! For more backstory on Citationsy see https://blog.prototypr.io/on-citationsy-4e143bbafc04. Sounds like Cenk has good taste (mentions iA Writer as inspiration). I'll be exploring Citationsy more.
sqlite_rekey_v2 is at 00CA1C13 on the Windows version of 1.19.3
Hooking it with Frida or similar should allow you to drop the 3rd parameter and set pKey to NULL allowing you to create a database which can be read by standard sqlite.
Keys however appear to be account-specific - I haven't looked into the full algorithm yet, but it's fairly pointless as you'd need a copy of sqlite3 with SEE support (which costs $2600 or so) to decrypt it using that method anyways, I tried briefly but was unable to come across a stray copy of sqlite3 with SEE enabled, the above solution is probably better for now anyways.
Both Mendeley and Zotero are better than Polar (for now) with bibliography management but I believe we're better at both at document annotation and knowledge management.
We're also Open Source and launched on Hacker News a few months back.
https://news.ycombinator.com/item?id=18219960
We're still rather new so not a lot of people have heard of us yet but moving forward very aggressively.
I'd also like to thank the Zotero guys for posting their notes about the Mendeley encryption issue.
I think this is a completely unacceptable situation and antithetical to what we should expect in the scientific community.
The Polar on disk repo is exactly the opposite.
We store all your data on disk and your annotations are in JSON format so they're easily hackable.
This is part of the design.
We MAY add end to end repository encryption at some point but it will be in the users control. We're not doing it to lock the user out or to prevent export.
The end to end encryption is so that you can store your repo in the cloud and not have to worry about your data being viewed by anyone other than you.
I will try your software soon. What I really like so far is the reading progress, which is actually very, very important for me. So far, I have to do this with manual tags in Zotero.
By the way, since this is a work tool, I (and I imagine others) are fully willing and able to pay you for support or storage or other features.
But since Mendeley, me and people I know have become very sensitive about lock-in, encryption and collaboration barriers associated with pro/non-pro accounts. It seems you are determined to take the right steps, I just want to emphasize the importance for your business.
The backlash against Mendeley isn't random. If I build my pipeline on your product, my lifelihood and hours upon hours of crucial work depend on it. This is not a casual software, where switching is inconvenient but not a big deal. Mendeley has f'ed me in a crucial deadline situation, even though I was paying them lots of money. That company is dead to me, and I will badmouth and try to destroy their business any chance I get.
This company bought Papers, the Mac app, and it looks like they are finally getting ready to release the desktop client based on that acquisition.
The browser and MS Word plugins actually work pretty well, although several of the citation style templates seem out-of-date.
Tried zoetro but prefer using highlights app ad It saves the highlights on the PDF. And you can then export to markdown etc.
Not sure how zoetro saves highlights but I am never going to take that risk again. A years worth of research down the drain.
It is wonderful software. It provides me with my own personal google for my PDFs. I used to use google desktop.
And hey, user security is a pretty good reason to encrypt a database, but if security were the only aim, they could have made the data accessible for export purposes (after all, it's accessible for the user to read).
At this point in history I'm not even sure it's worth going into all of the reasons this is a terrible thing to do on their part. At best, they'll scare off some users from migrating away briefly and buy themselves some time to figure out what they need to do to produce something that's better than the competition they're attempting thwart. More likely, they'll simply fail (quickly or slowly). It's that last point that really matters to me. If I'm buying an application that I'm going to rely on, I want comfort that the company will continue to maintain the product -- or in the case of open-source, confidence that the product is still regularly maintained (or in languages/frameworks that make it practical for me to maintain myself).
Aside from the fact that I wouldn't want that feature 'as it has been designed, today', I wouldn't want to rely on a product that's being designed by individuals who aren't well versed in the perils that DRM-like behavior causes. After all, that's what this is -- it's an attempt to keep the user from transforming their own data for the purposes of locking that customer into the product while attempting to position the lock-in as a security feature. And it failed on both accounts, Zotero worked around the encryption (which means the bad guys can to) and if people looking for a product like this care about that capability, they'll likely refine it to the point that it's as capable encrypted as it is decrypted.
Every time I've been asked to implement a DRM solution I've either outright refused or paired it down so much that the "R" didn't really exist. They're horribly complex to write, off the shelf solutions are already cracked and time would be better spent giving customers something they're willing to pay for. The thing that makes me roll my eyes every time I read one of these is "there are still engineers out there who think this is a good idea?"
I can't wrap my head around what causes this kind of thinking -- it's not a zero-sum game, there's room for more than one product in the marketplace -- even if one of them is free and open-source. Trying to beat the competition by sabotage rather than by making a better product doesn't work ... at least not for very long.
[0] And they also do my least favorite "other marketing thing" -- "Create a Free Account" and no hint of what the thing costs in any obvious place. So maybe this other product is free? I doubt it -- Like most companies, I'm assuming they're perfectly happy collecting your personal information for free (in order to send marketing to you to get you to pay for whatever service is tied to the free account). It's the little deceptive things that drive me crazy -- you didn't trick me, you only managed to make me suspicious of your product.
[1] I've gotten lucky ... usually budget and time doesn't allow for wasting energy on customer-hostile features that won't achieve the ends they're aiming for. The only case where something "DRM-like" was in an application I wrote was to be the opposite of customer-hostile. We had an app that provided information to users based on their permissions, determined by them logging in. I pushed to make the automation (sending us the login information) opt-in, with manual provisioning options. It worked well, since if they failed to true up after months of warnings, the application would deactivate (due to the size of the customers, months of warning was somewhere near a year and could also be disabled with a flag).
I was unaware that this was an Elsevier product (another comment pointed that out).
I do not use Elsevier products (directly, anyway, maybe I do indirectly but I doubt it). I have, however, followed the various controversies surrounding the company for the past several years. Those who are with me will not be surprised that Mendeley added this sort of "feature".
Elsevier is not in the business of providing a helpful, useful, research library and tooling. Their business model is to be the only source of that information. This has been done using a combination of legal, contractual and technical means. I won't get into the whole ethical/legal argument about whether or not they should be allowed to do some of the legal/contractual things that they do to "pseudo-control" that data[0].
This smells like a feature borne out of corporate culture. Their business really is about exerting control over their customers, so it's natural (to them) that tool their customers rely on would similarly contain features related to keeping customers in the walled garden[1].
[0] Especially research done via grants paid for with public funds) -- I suspect I'd enter an echo chamber pretty quickly (and even if not, I'd be unlikely to persuade someone in a medium-length comment.
[1] There's some clever pun here about paper walls and research papers but it's too early to be that clever.
I hate being right about such things. This is exactly what the fear was when they bought what was a fantastic piece of software, and doing really interesting things for researchers.
The issue is that due to Elsevier, a DRM logic was implemented that just doesn't make sense for scientists. The idea is that I can tell you about a paper, but you have to download it yourself because I can not send it to you. I can annotate the PDF, but I can not send you the annotated PDF. We have to look at the same screen to see what I wrote.
That end result means that Mendeley is no longer a good product. It makes scientific collaboration difficult to impossible. Sharing of files, ideas, results and writing is such a basic thing in todays world, that you should really be surprised how a company would think that this is a good idea, especially for scientists who almost always collaborate in loose teams.
But that is was happened, because Elsevier only makes money if the only place you can get any PDFs is their DRM walled garden.
So Mendeley is now bad. And NOW it matters that you can not get your data out of it. And the thing is, these annotations and folder categories, tags and meta-data - that's all specific to one researcher. It can not be replicated. It could be years of work in that database, and Mendeley holds it hostage.
https://gdpr-info.eu/art-20-gdpr/
"Right to data portability"
>The data subject shall have the right to receive the personal data concerning him or her, which he or she has provided to a controller, in a structured, commonly used and machine-readable format and have the right to transmit those data to another controller without hindrance from the controller to which the personal data have been provided, where:
> the processing is based on consent pursuant to point (a) of Article 6(1) or point (a) of Article 9(2) or on a contract pursuant to point (b) of Article 6(1); and the processing is carried out by automated means.
>In exercising his or her right to data portability pursuant to paragraph 1, the data subject shall have the right to have the personal data transmitted directly from one controller to another, where technically feasible.
[0] https://twitter.com/mendeley_com/status/1006915998841221120