Hacker News Reader

Comment by Const-me | Hacker News Reader

Please don’t.

When people are sending or sharing a document, they don’t expect to leak edit history of that document. And SQLite inherently leaks that edit history, in database freed pages and unallocated space. You can VACUUM the database after each save, but that’ll negate the performance benefits. Also VACUUM was designed to save space, not sure it reliably cleans up unwanted data.

OTOH, these zipped XMLs are free from unwanted data, they (hopefully) only contain the data user wants to save, i.e. the content of the last version of the document.

View on HN

The "PRAGMA secure_delete=ON;" command will fix that for you. Or you can make secure-delete the default at compile-time using the -DSQLITE_SECURE_DELETE option. There is a (small) performance hit, since extra writes have to occur to overwrite the obsolete data with zeros, but it does work.

See https://www.sqlite.org/pragma.html#pragma_secure_delete for further information.

Not enough. See my next comment: https://news.ycombinator.com/item?id=15607650

I agree. People are vastly underestimating the information already exposed by people not understanding what they are sharing when they hand a modern document format out.

New features need to examined for unintended consequences, and the fact that this feature might entail data leakage through fragmented included data should be a red flag.

People should publish PDF version of a document. But if they send the working copy around, then the history may be helpful for other collaborators. Think of git. They can even merge the changes easily.

Agreed. History should be considered a feature.

And it is. That's fine, as long as the user actively turns that feature on, and by default no data is leaked.

Unfortunately, companies don't do that. They send the doc/docx. They also send the xls/xlsx instead of the csv. Protecting users from accidentally doing a very wrong thing is better than being slightly more helpful as a default. Security should be the default. IIRC there have been cases where confidential information has been found by looking through the history in government released documents.

docx/xlsx are better, they are zipped XMLs too, just like odf.

Older ones, doc & xls are closer to sqlite, for the same reason, performance. I remember finding confidential information in these files received from third parties they probably didn’t want to disclose, e.g. offer price that was before the last edit.

You know you can pop calc with a csv file opened in Excel?

from http://georgemauer.net/2017/10/07/csv-injection.html :

  UserId,BillToDate,ProjectName,Description,DurationMinutes
  1,2017-07-25,Test Project,Flipped the jibbet,60
  2,2017-07-25,Important Client,"Bop, dop, and giglip", 240
  2,2017-07-25,Important Client,"=2+5+cmd|' /C calc'!A0", 240

Yeah - that "=2+5+cmd|' \c calc'!A0" gets Windows to execute code...

> People should publish PDF version of a document.

Oh please no! PDF is quite user-unfriendly (e.g. can't reflow, typically can't cut/paste properly) designed to emulate an ancient medium that has benefits and drawbacks different from screens.

And PDF, while executable, is essentially compiler output, with all the destruction of semantics that implies.

It depends on what your are trying to do.

Short of exporting to an image format, PDF is the best format for preserving and sharing a final rendered output. All of the things you list as drawbacks are virtues if your purpose is static presentation.

I only have issues with ebooks, outside them it is just perfect.

Cut/paste only fails on me, when whoever created the PDF had the brilliant idea of using images instead of actual text, e.g. scans. Something that using another format won't help.

Maybe you should use another PDF reader?

For a resume, PDF is ace. I don't want the reader to see version history or copy/paste (even though its peanuts with a half decent PDF reader). I want them to print it.

P.S. Even fragmentation can leak data. E.g. search for the most fragmented portion of the document in the DB, and you'll likely find out the most heavily edited paragraph of the document.

I believe that you raise excellent concerns. I also believe the solutions are perhaps just a bit of thoughtful UI design away.

The options could be something like:

    ----Performance & Privacy-----
    
    Document Save Mode
    
    [x] Balanced. Document history is not shared. (Default)
    [ ] Fastest. Not recommended for sensitive data 
    [ ] Maximum privacy. Slowest, but most secure

    Click for more information on what these choices mean

The first option would zero out previous edits. This would be the default. This would still be faster than re-zipping the entire archive -- you'd only be zero'ing out a single file. Metadata could still be inferred via fragmentation as you say, but I do not believe this is a concern for most use cases.

The second option would optimize purely for speed.

The third option would optimize purely for privacy. Perhaps the entire document database would be rewritten from scratch on every save. This probably wouldn't be too much slower than the current XML dance.

I understand the value of this kind of "paranoid" mode but I think it's overkill for most situations. Personally, the situations where I'd want this kind of behavior are extremely rare. Clearly some people would need/want this and that's fine; it would be only a click away. And then all future documents could inherit this choice until the app is told otherwise.

This UI solves a problem for the computer, not the user. Whenever you find yourself making a UI like this, you should go back and change your program so the computer doesn't have that problem any more. In this case, there's surely a way to write files quickly enough while still maintaining user privacy -- just do that instead of bothering me with a wall of text every time I save a file!

     just do that instead of bothering me with a wall of text every time I save a file!

I certainly wouldn't want to be hit with this wall of text every time I save a file. I was thinking that this sort of option ought to be a global setting that could perhaps be altered on a per-document basis if (and only if) the user cares to muck about with it.

It could, perhaps, be a choice that is presented to the user upon new document creation.

     This UI solves a problem for the computer, not the user. [...]
     In this case, there's surely a way to write files quickly 
     enough while still maintaining user privacy

Life is full of these choices. There's a gas pedal in my car. I can mash it (to go fast but burn a lot of fuel) or be gentle with it (and save gas, but I won't go as fast). Ideally the car should be crazy fast and crazy safe and crazy fuel efficient no matter how you drive the sucker, but that's a tough engineering challenge and ultimately we (today) leave that choice to the drive.

That said, you are 100% correct. Maybe the privacy/paranoia steps are something that could be done on a background thread. The file could be saved quickly, and then the database could be privacy-optimized in the background immediately thereafter.

That's a terrible UI. First of all, it's an issue that users wouldn't even know exists, so they'll probably never look for the options anyway. And the proposed default leaks data, a terrible choice! Secondly, why are there three options? Two of them leak data, one doesn't. Who in their right mind would pick the "leak some data" option?

Leaking sensitive data that users would expect to be deleted is a bug. People don't want their software to be full of configuration options that amount to "fix this security hole, but run really slowly" or "leave this security hole open, and run faster".

    And the proposed default leaks data

The suggested default ("don't share edit history, but don't go 'full paranoid' and vacuum/defragment the database on every save") wouldn't leak data. It would, at times, leak information from which some metadata could sometimes be gleaned. Most edits would presumably not even cause fragmentation unless they spanned more than a single data page in Sqlite's on-disk storage format.

If you counter that "metadata is data; therefore it does leak data" I suppose you'd be correct in a very strict sense. But, regardless of semantics, I think "sometimes hinting at which parts of the file have been edited" is quite a bit different from "leaking a document's edit history, verbatim" -- in much the sense that a common cold is different from HIV despite the fact that they're both viruses. (Though, I suppose they can both kill you if the conditions are right)

I can certainly think of situations where a user wouldn't want to leak such vague metadata, but that seems to me like an edge case and we as developers should strive for sensible defaults. I can't think of too many instances when I would have cared if somebody could perhaps infer which part of a document had been edited most often.

That said, yeah, offering choices like this is clunky. Maybe the privacy stuff could just happen automatically in the background following a save, or at export time, etc.

> wouldn't leak data

They will leak data, e.g. deleted information. Search “SQLite forensics” for technical details.

> Most edits would presumably not even cause fragmentation

You replace “a” with “the” in the middle of a large table, and generally the table becomes fragmented. The way it’s fragmented indicates not only where it was edited, but also what kind of edits were made, insertions and deletions fragment database differently.

Reply on news.ycombinator.com