Thinking hard if I ever want to get another Kindle when Amazon can just screw around with what I put on my Kindle ...
I find it easier than converting to Kindle format and then copying over USB.
Convert your book to .azw3 in Calibre
Instead of sending it to the device in Calibre, locate the azw3 file (Right click -> Open book folder).
Copy the file to your Kindle, but not to the "documents" folder (where Calibre usually puts it) but rather into Downloads->Items
This folder is where books go when you buy them from Amazon or receive them after using the Send to Kindle feature. I have only tried this with azw3 so far but it might also work with .mobi format.
At what point do we stop giving the benefit of the doubt that it's a "bug"?
They're perfectly happy to let you email books to the kindle that you bought at other stores (or stole), as well as sync your progress with those books, backup those books to their servers, and generally have the full reading experience with all the benefits of the kindle ecosystem even if you didn't buy the book through kindle. If they didn't want to encourage the use of third-party files, surely they'd make it more difficult than a bug that randomly deletes books off some people's kindles sometimes.
Also by emailing books or loading through their servers, they can still track and get that sweet sweet data/metadata that Amazon thrives on. When you sideload, you don't even have to connect it to the internet, which makes analytics more challenging.
if they want people to buy books from their store, why do they make it so easy to not buy books from their store?
bugs happen. not every bug is part of jeff bezos' nefarious plan.