Backing Up My Kindle Ebooks (2018)
sonyaellenmann.com
sonyaellenmann.com
By default, converting books to another format will change the layout! Destroying the original formatting by default is unacceptable behaviour to be honest, so user beware.
For Kindle, I suggest using the KindleUnpack plugin instead; which can extract the EPUB file embedded in the AZW/KF8 format file losslessly for use with EPUB based readers without having to worry about Calibre's conversion messing up the formatting.
These settings can be useful in cleaning up badly formatted files, but should not be used carte blanche on all input files, especially those that don't have problems. You'll lose niceties like the first paragraph in a chapter not having text-indent, etc.
Generally it's better to use the "Edit book" feature on EPUB files and manually fix up the CSS if the styling is not to your liking. Converting with Calibre also hard-codes @page margins which overrides most e-Ink reader software's margin settings; so if I do end up using Calibre to convert to EPUB, I always have to edit it afterwards to remove these hard-coded margins in the CSS.
FWIW, I still heed those warnings, and so I still hold on to the de-DRM'd AZW file even after I convert it to EPUB.
There are two reasons I can think of for why you'd get broken paragraph separation. First, the markup in the kindle version (if it's old style mobi, or really badly generated kf8) could be broken, so when calibre tries to split into multiple files to prevent single (x)html files from being too big, it can't tell where paragraphs are so it might split in the middle of a paragraph.
Second, you might be converting a pdf or azw4 (essentially a pdf) to epub, in which case you'll get paragraph breaks across pages. You shouldn't convert those formats to epub without extensive editing.
How do I list books on my Kindle? How do I list books in my account? It's all very byzantine.
- Open Calibre
- 'Add' your EPUB file(s) (top left button)
- Connect your Kindle via USB
- Select the book(s) you want to send
- Click 'Send to device' (it's one of the buttons at the top)
- If you get asked to convert your book, choose AZW3 (it's newer than MOBI)
- Done
----------
To view the books on your Kindle:
- Connect your Kindle via USB
- Click 'Device' (at the top)
Note 1: If you want to be able to put your books on a different Kindle, you need the DeDRM plugin [1]
Note 2: If you want to be able to convert your books to EPUB to use with a non-Kindle e-reader, you need the KFX Input plugin [2]
Note 3: KFX is not a great input format, if you don't have moral reservations it's better to 'obtain' the books you paid for from the internet in EPUB format.
----------
To view the books on your Amazon account:
This is not possible using only Calibre. You need to install Kindle for PC/Mac and get the files from its storage directory. The notes above apply, i.e. it's probably better to just obtain EPUBs from the internet.
----------
In my opinion, it's not really worth it to back up books from your Kindle, you're better off obtaining the EPUBs off the internet. If you want to put those EPUBs on your Kindle, that's where Calibre shines.
This is also great if you don't want to support Amazon and their lock-in scheme. Just buy your books in EPUB from any online retailer and use Calibre to put it on your Kindle. Note that some publishers apply Adobe DRM to EPUBs (which you can break using DeDRM [1]), but there are also many EPUBs using only a watermark (aka 'social DRM'), which is not DRM.
[1] DeDRM: https://apprenticealf.wordpress.com/2012/09/10/drm-removal-t...
[2] KFX Input: https://www.mobileread.com/forums/showthread.php?t=291290
Use-case: I have bought books from other vendors, converted them to epub format, copied them via Calibre onto my Kindle reader. - I made lots of annotations in those books on my Kindle reader. Does it not make sense to back those books up?
It's kind of like pandoc, except pandoc only supports epubs.
I mostly agree that you want to avoid "transcoding" an EPUB. But the formatting on some official EPUBs is so bad that I like to use readers that can override the built-in formatting. I like to choose the font, spacing, hyphenation, and justification myself. (Full justification and hyphenation are my bugbears--I think they are never appropriate on smaller screens, as they lead to large rivers and too many words broken up, but are fine on large ones.)
- MobiPocket and early Kindle version 7 or less ebooks are unpacked to the original html 3.2 and images folder that can then be edited and reprocessed by MobiPocketCreator.
- Kindle Print Replica ebook are unpacked to the original PDF and any associated images.
- Kindle KF8 only ebooks (.azw3) are unpacked into an epub-like structure that may or may not be a fully valid epub depending on if a fully valid epub was originally provided to kindlegen as input. NOTE: The generated epub should be validated using an epub validator and should changes be needed, it should load properly into Sigil and Calibre either of which can be used to edit the result to create a fully valid epub.
- Newer Kindle ebooks which have both KF8 and older versions inside are unpacked into two different parts: the first being the older MobiPocket format ebook parts and the second being an epub-like structure that can be edited using Sigil
[1] https://gist.github.com/yangchenyun/a1c123935d82f5e25d57
kf8 and epub are roughly equivalent.
KF8 is a proprietary wrapper around a regular EPUB file.
I'm also interested to know more about why Calibre destructively converts EPUBs instead of making a copy to work on instead. Do you have any sources for how this works?
As an aside: I was helping a friend out who owned a Kindle recently and shocked to learn it can't read mobi or epub. Why would anyone buy such a crap e-reader?
And I use it because it’s got the best hardware experience of the bunch, IMO.
ePubs seem more popular than mobi though, so I think that’s where most of the “converting for kindle” happens.
Kindle does support mobi, but you have to convert epub files (this can't be done through your Kindle email address' convert ability for some reason).
I think the reason is they don't want to make it too easy to buy ebooks elsewhere and read them on the Kindle. Having to use something like Calibre to convert epub to mobi before sending it to a Kindle is just enough hassle and requires enough comfort with technology to make most stick with the easier option of buying ebooks from Amazon.
The point, and imho only point, of Kindle is the tight integration to Amazon store. It is very much dedicated single purpose device for reading Amazon content. I have one and its great; I browse Amazon store on my computer, select a book, and with one click buy the one I want and have it automatically delivered to my Kindle. No need to think about files and formats or copying and converting stuff around.
There is an exemption for breaking DRM on ebooks to enable their use with assistive technologies used by people with disabilities. There are some audiovisual exemptions for educational use that encompass "multimedia ebooks" but only for specific purposes.
https://www.federalregister.gov/documents/2018/10/26/2018-23...
AFAIK, Amazon's latest DRM on KFX files hasn't been broken yet. That came out in 2015. Some people get around it by getting the book in the older format, but then you lose all the advanced typography stuff that KFX supports.
[1] https://apprenticealf.wordpress.com/2019/03/30/dedrm-tools-6...
dvd locked by region? don't worry, more expensive models can be bypassed by those two button presses, only the poor is really impacted.
tracking users and OS security bugs, don't worry, the latest $600+ devices will get a patch. only poor, used device buyers and developing countries models will be affected by not receiving a patch.
the modern world now is divided into the elite and the not-economically-viable-to-port-security-patches. And everytime we, the elite, dismiss something as a nuisance because we are the elite, we are making things irreparably worse.
my sugestion: do not give in to drm because you can work around it. instead pay the extra cents and get a used hardcopy. the convenience of a ebook is lost if you have to work around a drm anyway. and that way their drm sales drop to zero and they have to rethink this whole falacy.
The number of people accepting DRM content is used as a sales pitch by Pearson printing to force universities to use DRM text books that can't be resold! again, screw the poor, right?
PS: sorry for the rant. DRM makes me salty.
I got my first kindle in 2010 and bounced over to Barnes & Noble for a while with a Nook but then came back to the Amazon ecosystem. I've bought hundreds of books over the years.
But just like paper books there are a lot of books that I read once and never look at again. They aren't priceless treasured objects for me.
The thing with Amazon.. people complain a lot, but their ecosystem just works the best in the eBook space. I actually find it a lot less annoying than some of the shenanigans with other digital stores like the iTunes store over the years. I haven't tried Kobo but B&N was always a hot mess compared to Amazon. Tons of annoyances in the store & device software that Amazon had right from day 1. And you were paying more for a worse experience.
P.s. I wasn't sure if you were saying this, but I don't think Kobo has anything to do with B&N.
But I hear enough through the woodwork that I have been under the impression using Kobo would be trading off annoyances with some things better & some worse. Mostly what I hear is Kobo often has some hardware advantages but has some different software annoyances.
What format the book uses is a very minor concern to most of us who just read the books. Epub vs Mobi vs KF8 or whatever has no impact on me 99% of the time. The only time it'd really matter is hopping around between platforms and having to convert stuff.
If Kobo started having a huge advantage I'd be more likely to buy a Kobo and keep my Amazon library with Amazon and start buying Kobo books there. I wouldn't go on a big project to convert all my Amazon bought books to a Kobo compatible format. Cause once done I wouldn't actually need/want to re-read all those books on the Kobo.
On kobo, the dedrm process involves installing the kobo desktop software (windows/Mac only), which syncs the books to your desktop. From there you can snatch the ebook from the file manager and put it in calibre. There’s an “obok” dedrm plug-in.
If you switch from kindle to anything else, you will lose the ability to download your kindle purchases. You need an active kindle account/device to download ebooks from the kindle library (this wasn’t always the case).
That's a little different than photos in that once the photo is gone, you don't have the opportunity to capture it again.
[1] https://www.nytimes.com/2009/07/18/technology/companies/18am...
I've run into this, especially with science or reference books. And was impossible to fix on my own because the original download was to an old Kindle that's now sitting in a box somewhere in the garage, or to the Kindle app on an old phone that's no longer in my possession.
I suppose that could be fixed with a call to Amazon's customer service who could probably de-register those old devices in their system. But who wants to deal with that? Every book I've ever bought from Amazon has been cracked and backed up.
I had been willing to put up with the DRM. Didn't want to go to the hassle of re-downloading and processing hundreds of books. But when I ran into the "This book has reached its license limit and cannot be downloaded", that was the last straw.
Traditionally your books could have been donated, or given to friends and relatives. DRM prevents those things. It really sounds like you might be better off using a library to borrow ebooks than to buy them. You're already basically just paying for a rental.
I just want ebooks. I don't want an 'ecosystem in the eBook space'. If I had a similar paper book ecosystem 'in the paper book space' I'd have some giant foreign corporation insist that before I was permitted to read I must buy a bookshelf from them. Then all their books would have to be on that shelf. Then if there's a book they don't sell, or for which I choose another retailer, I'd need to buy a different shelf. Then every time I want to read a damned book I have to first know which corporation's mandated shelf it's on.
The whole DRM-ebook business stinks. I do buy the odd one from Amazon if I can't get it anywhere else, but the DRM goes immediately on download.
I feel uncomfortable about the legalities but I do not like the social contact being written in this way, against any consumer interest and without discussion.
Buying books is important. Authors deserve their pay. DRM is like book cancer.
From the kindle you can also export all the highlights of a book to your email. It's a lot better to read it on your email. The kindle device reader note reading experience is subpar.
Would you mind sharing your script?
Suggestions welcome!
I’ll test it as soon as I can and will get back to you with feedback if I make changes. From quickly glancing over the code, I do like it!
This closes my last Kindle lock-in. I’m very grateful _/\_
I believe this link will only be available if your account has a real kindle (e.g. eink or fire) registered to it. It encrypts the downloaded file for whichever registered kindle you select from a list of all your registered kindles and apps, with the apps grayed out so you can only select actual kindles.
https://www.downpour.com https://libro.fm
Most titles on audiobooksnow.com are free as well, but not quite all, so make sure there's no "DRM Protected" symbol.
I bounce between these providers (subscribing and cancelling to their various plans as needed) based on book availability and price. I believe I pay slightly less overall than if I used Audible for everything, even if acquiring books takes more work.
In a pinch, you can also remove Audible DRM with various tools (see sibling comment), but we should support proper DRM Free providers wherever possible!
OpenAudible looks like a nice tool that'll automate all of these steps for you.
The second answer (which is the top voted one) is my answer. It's basically using ffmpeg with -codec copy.
Basically you can selectively "patch" ebooks to censor profanity, sex scenes, violence, (and more) depending on your tastes and on what you like (and don't like) to consume.
However, this is extremely unpopular with authors/publishers for the same reason that censoring movies is extremely unpopular with Hollywood (hence why companies like VidAngel are constantly mired in legal issues despite widespread popularity with consumers).
Honestly, I don't understand why since at the end of the day everyone gets paid, and indeed get paid more than they normally would have (I guess it's more of a "you are altering my art so people can consume it in ways I didn't intend" attitude). But in my mind it's no different from using whiteout on a physical book or picking out the mushrooms (or whatever you don't like) in some chef's dish.
Basically, I've already built the tech to do this for myself privately (and have patch files for ASoIaF and a dozen other epubs), but I will never share it (I've even considered FOSS) except with close family and friends even though I know it would be wildly popular in places like Utah because I fear legal pursuit from authors/publishers (and since you must break DRM in order for the tech to work, I don't see a way out).
If you don't want to be exposed to these things, why not just only consume content that does not have them in the first place?
From context! You don't need to explicitly see two characters having sex or read an erotic paragraph about them having sex to figure out they have an incestuous relationship.
And most, if not all religious folks just want the sex and violence to be less explicit (left to the imagination) not completely whitewashed from the story. Otherwise you wouldn't even be able to read the Bible/Quran/Book of Mormon/etc. without encountering sex and violence.
I think you misunderstand what religious people really want. Most people with "puritanical" objections to media content are just wanting it to be less explicit (i.e. left to the imagination) - not completely whitewashed from the story. After all, most religious texts themselves are rife with sex and murder; the acts themselves are usually not explicitly described in great detail, however.
"Lysanderoth's sword tore into Archibald's neck, spraying blood and chunks of windpipe across the battlefield and leaving him loudly gurgling as he slumped to the ground"
vs.
"Lysanderoth slew Archibald with the sword"
You can easily retain the story while eliminating the explicit aspect of the prose.
For example, I edited the entirety of the "A Song of Fire and Ice" series, and you may be interested to learn that eliminating every instance of explicit violence, sex, and profanity altered slightly less than 1% of the total word count of each book.
Whiting out in a book, or in your own ebook copy, is one thing. Sharing, then, those edits would be like releasing a fan-cut of a movie. Doing so wholesale (the entire movie) isn't ok - sharing timestamp swaps that you can plug into VLC is a different story.
That is precisely what I am suggesting sharing - page numbers, paragraph numbers, length, and other metadata needed to filter the book (essentially "timestamps" for books). This metadata would tell the software where to excise sections from the ebook. People would need to own their own copy of the book, and the ewhiteout software just alters it according to the metadata input.
The problem with "you are altering my art so people can consume it in ways I didn't intend" can be a purely economic one, as it's misrepresenting the "brand" that is the author. If you removed the "vulgar" things than you are changing the product in a non-controlled way. It's why some directors or writers remove their names after producers or whatever external controllers get a hold of it. It doesn't represent their creative abilities. They could put out two versions, but I think that the opinions of most authors that if the sex scene could be removed, it didn't need to be there, and it wouldn't have been there in the first place.
It's removing an aspect of the brand.
How is a consumer altering a movie experience or book experience any different from a consumer altering a:
- music experience (remix, etc.)
- food experience (customizing a chef's dish)
- video game experience (with game genie, mods, etc.)
- picture experience (with photoshop, memes, etc.)
?
People create derivative works all the time of all media types to suit their tastes, dreams, and whims - I don't see why that should be illegal as long as they aren't trying to resell it or create a competing brand.
FOMO, mainly? You feel left out when all of your coworkers are talking about GoT everyday... if only there were a way to consume it in a way that met your standards and you could participate in the discussions too...
> Would you feel differently if I wanted to _insert_ sex scenes in to movies that didn't have them?
Not at all! If that's what you want to do, you should be able to do it on your own copy! Or if you wanted to replace the main character with Thomas the Tank Engine, whatever floats your boat, that's great.
Widespread popularity with an incredibly small number of consumers who are generally very religiously oriented. And as seems to be common in the highly-religious groups, VidAngel commits copyright and trademark violations with the justification of "it's for families!" Just like a thousand religious summer camps who make "Reese's Jesus" shirts and shirts with "Jesus Christ" written like the Coca Cola logo because "it's for Jesus!"
The vast majority of people don't feel the need to censor the content they watch because they either understand what's in the content, or simply avoid tainted content altogether. If you're censoring things like ASoIaF, then maybe ASoIaF isn't for you, considering how important incest is to the plot.
Right, but said people number in the millions, and would consume said content if only there were a way to selectively censor what they find objectionable.
> If you're censoring things like ASoIaF, then maybe ASoIaF isn't for you, considering how important incest is to the plot.
Speaking from experience, me and my family/friends agree that ASoIaF is a fantastic series, even when completely censored to remove all instances of explicit violence, sex, and profanity.
The only thing you really lose is explicit detail when it comes to specific sex scenes or violent acts - and that constitutes less than 1% of the total prose. Everything else (intrigue, politics, etc. - the 99%) is still there.
Perhaps I'm not understanding, but if you're whiting it out, you're reading it anyway, right? So, what's the benefit?
If you get to a sex scene, can you just skip it or something?
When you hit the timestamp of a filter, it just skips over the scene.
Books would be the same, but with the added advantage that you can easily alter the text to make it feel more natural. So instead of an explicit sex scene, you can end the chapter with a new transition where the characters enter the bedroom and close the door and it is simply implied what they did in between chapters.
FWIW, congress wrote an exemption to the Copyright Act when a company wanted to do this for movies. https://en.m.wikipedia.org/wiki/Family_Entertainment_and_Cop...
So if your product was actually popular enough in Utah that might happen again. Of course that was 13 years ago.
Ironically GRRM specifically mentioned that he doesn't write "comfort fiction".
I avoid things that I believe are bad for my body, and I avoid things that I believe are bad for my mind. You might argue that it's impossible to consume something "bad for your mind", but I disagree. I think there are certain forms of entertainment (written, visual, and interactive) that absolutely can desensitize you, distract you, addict you, influence you (negatively), etc. - I try to avoid those effects where possible, and self-censorship is just one tool to combat them.
All my project allows me to do is export a set of changes on a book to a JSON-based metadata format, so the changes can be shared. I'm afraid to monetize it because I don't think I could deliver a good UX without breaking DRM (which would get me into legal hot water), but perhaps making it into a context-free Calibre plugin might be safe...
I suspect your argument is that you're only distributing the "patch files," and so that's not intrinsically a copyright violation. (This is distinct from VidAngel, which admitted that they did, in fact, have to have copyright-violating master copies of movies they were "patching" in order to stream the bowdlerized versions.) And...maybe? Yes, in that case, DRM would become more of an issue. But I'm not convinced that a publisher with sufficiently deep pockets who wanted to press the case couldn't successfully argue that the combination of your patch file plus the original work creates an unauthorized "derivative work."
Well, let me rephrase that: that combination absolutely creates an unauthorized derivative work. It's not clear to me that it's illegal, but the ethics strike me as dubious. For instance, using my own book again as an example: the book wouldn't be materially affected if the main character's penchant for swearing when she was stressed got toned down. But it would be materially affected if someone came through and edited out the parts that made it clear that she was pansexual, or that changed the non-binary character to no longer be so, or that changed a gay character to not mention his orientation at all, or that quieted the vocal politics of either the protagonist's socialist sister or her libertarian friend. And that's the real issue you'd run into: it's not just the potential for "you took out the R-rated parts, you prude," it's the potential for "you've made my text mean something it wasn't intended to."
If you made a firm commitment to work with authors and publishers and only release approved patch files, that would probably make this possible (again, setting aside the DRM for a moment). But without it...well, with respect, I'd definitely rather not see that.
Yeah but they purchased one DVD per streamer, so each streamer was "virtually" modifying their copy. Otherwise the FMA is incompatible with modern video technology such as streaming. The real reason VidAngel got in trouble was because studios do not want users to have the ability to remotely buy/sell/stream DVDs - even if they technically owned them at the time of streaming. To do so would circumvent their lucrative streaming licensing.
> You don't get to produce a "derivative work" from Kismet without my explicit approval
Agreed, but that depends on the definition of a "derivative work".
> Well, let me rephrase that: that combination absolutely creates an unauthorized derivative work
Not necessarily. Does using a game genie on a Super Nintendo game create a "derivative work" if you use it to completely modify the game at run time? The courts ruled that it didn't since it didn't create a new copy[1]. Hence if I could modify your book at run time such that I didn't create a new copy, it could be considered the same. This is sometimes known as the kaleidoscope effect (i.e. looking at a proprietary painting through the lens of a kaleidoscope doesn't create a new painting and thus doesn't create a derivative work).
Or, even more generally, what's to stop me from building two physically overlaid screens - one that displays your original work and one that overlays alterations? That doesn't create a derivative work anymore than drawing on my television screen with a sharpie while a movie is playing creates a derivative work.
Again, I understand your desire for users to consume your work as you intend. But extend that desire too far and you turn into a strong DRM proponent. I'm a DRM opponent, and I respectfully assert my right to modify my copy of your work as I please - and I invite you to do the same for my works - be they software, novels, or whatever.
I can't believe you don't see the issue?
It seems like "the issue" boils down to "it offends the author"
"The author and publisher have provided this e-book to you without Digital Rights Management software (DRM) applied so that you can enjoy reading it on your personal devices"...
So I assume either they don't have kindle DRM, or it is ethical (possibly legally allowed) to remove the kindle DRM.
> [...]
> Termination. Your rights under this Agreement will automatically terminate if you fail to comply with any term of this Agreement. [...].
https://www.amazon.com/gp/help/customer/display.html/ref=hp_...
So you may get in trouble if you're caught doing stupid things, yes. Will they bother though? Unlikely.
https://www.law.cornell.edu/uscode/text/17/1201
https://www.eff.org/deeplinks/2015/10/victory-users-libraria...
> The exemptions we requested—ripping DVDs and Blurays for making fair use remixes and analysis; preserving video games and running multiplayer servers after publishers have abandoned them; jailbreaking cell phones, tablets, and other portable computing devices to run third party software; and security research and modification and repairs on cars—have each been accepted, subject to some important caveats.
[^1]: I am not a lawyer.
I have a fairly enormous ebook library, and I don't care to have it exploded due to, e.g., some error in the kindle software api.
Haven't done it often, but it seems to work fine.
https://www.willus.com/k2pdfopt/
to pre-process them in various ways so they are actually readable on a Kindle. (Crop whitespace, flip to landscape etc)
To remove that you need another add on and the serial from the kindle as key
Don’t have the name of it with me now sorry
Last time I checked, there was no crack for Amazon's latest version of their DRM (KF8 files). A workaround exists where you get Amazon to wrap the content in the old DRM, but then you lose all the typography improvements in the KF8 file.
You need to download old Kindle app where it's possible.
"hacker news"