Hacker News Reader

Hacker News

Comment by killercup | Hacker News Reader

Full thread

killercup

> Flash / SSD Optimization

I.e., a "unique copy-on-write design"

> Space Sharing

Basically, ZFS datasets.

> Snapshots

If those can be sent: Finally Time Machine done right.

> The AFP protocol is deprecated and cannot be used to share APFS formatted volumes.

Interesting.

> An open source implementation is not available at this time. Apple plans to document and publish the APFS volume format when Apple File System is released in 2017.

Yay!

Limitations: https://developer.apple.com/library/prerelease/content/docum...

Edit:

> encryption models for each volume in a container: no encryption, single-key encryption, or multi-key encryption with per-file keys for file data and a separate key for sensitive metadata

Nice. I hope they also include checksums for each block.

Famously missing, but not the hardest thing to add considering all the features above: Compression (which HFS+ supports!)

View on HN

> > Snapshots

> If those can be sent: Finally Time Machine done right.

If this ends up being true I can't wait—it's so frustrating watching tiny incremental backups take forever over the network. It seems like "Preparing" and "Cleaning up" take longer than moving the data.

it's so frustrating watching tiny incremental backups take forever over the network

While waiting for APFS to become stable, buy Carbon Copy Cloner. $40. I love it. It's fundamentally rsync, but tailored to OS X. For personal use a single license covers an entire household.

Every night CCC fires up on each laptop and each does an incremental clone to its own dedicated directory on my desktop machine. This clone is usually less than a few GB and takes about a minute to run. CCC doesn't have to be run daily, it can be told to run hourly instead.

Time Machine running on my desktop then copies everything off to yet another disk.

So my backup environment is:

   each laptop uses CCC to periodically clone to desktop

   the clones on the desktop are "traditional",
   CCC is told not to keep its own copies of modified files

   desktop runs time machine,
   makes hourly backups to a TM disk,
   old versions of files can be found there

This gives me 3 copies of all data on each laptop:

1) on the laptop

2) on the desktop

3) on the desktop's Time Machine Volume

If you create your users in the same order on each machine, or later go back and change the User IDs to match (under advanced options in Users and Groups) then the uids will match everywhere, and all files can be easily browsed in each location, with all file permissions intact.

CCC was what I used back in the day (IIRC it used to be free) to make the most reliable bit-for-bit backups of OS X machines. Time Machine skipping over so many files is kind of a nuisance to me.

Unless you set the privacy in Time Machine to ignore certain drives/volumes/folders, you shouldn't really miss anything with a Time Machine backup. It usually ignores those files that aren't necessary and can be created by the OS or applications (like caches or temporary files, for example). What kinds of personal files did you lose with a Time Machine backup?

Of course you can see the full list at /System/Library/CoreServices/backupd.bundle/Contents/Resources/StdExclusions.plist.

One of the biggest is iPhone backups from iTunes. If I were a normal user and my MacBook didn't boot up tomorrow, I'd be surprised to find my backup didn't include critical files like this. For example, apps that are no longer in the App Store but are in your backup can be restored. Once you don't have that backup anymore, they can't.

It's just one of the reasons Apple users are forced to have multiple backup services if they want reliable and complete backups.

I'm not sure why the iPhone backups on your system didn't get backed up. The device backups stored in ~/Library/Application Support/MobileSync/Backup/ is not in the exclusions file that I see.

P.S.: The latest app binary files may not get updated/synced on the Mac post iOS 9/iTunes upgrade.

I double checked my drive and you're correct with regard to mobile backups. That's reassuring.

It would be nice to have a utility to give a definitive diff of what's on your hard drive that's not on your: Time Machine drive / [Carbonite, Backblaze, etc] backup.

I've migrated many Macs using Carbon Copy Cloner. Works fabulously and is a great tool.

Not trying to troll, but I see so many HN comments about backups and I just don't have this need anymore. What are people using traditional backup software like time machine, carbon copy cloner, etc. for on their laptop these days?

I use google docs for all my docs and spreadsheets, occasionally I use excel or word or keynote for files but if I do I save the docs to my dropbox or google drive folder, I have my photos synced with google photos, my music is synced through itunes match (or I can sync my music library files to dropbox, or use spotify), all my code is in git repos pushed to github or bitbucket.

What else is there to back up? Is it a matter of not wanting to re-install apps manually to get your computer back to its current state? Or is it a matter of not wanting to pay for the extra storage space on dropbox?

I agree cloud based backups like Dropbox just sync in the background. Got any souce code store it in Github or Bitbucket.

Documents and source code doesn't take up much space. Unless you want music and video files which iTunes can always sync for you to your Apple mobile device.

I use a USB hard drive to back up my Thunderbird and Firefox profiles. Also my downloads and other stuff too big for Dropbox. $100 for a 4T USB 3.0 hard drive is cheap.

Sync is not backup. If you corrupt or delete a file, the destructive action will get propagated as well. Time machine and many other backup systems gives you access to previous versions of files and folders.

- Some people don't want to use the cloud to store their stuff for privacy / ethic / commercial / legal reasons.

- Some others have been burnt by the cloud failing them (corruption, data loss, copyright abuse, etc) and want an additional layer of security.

- There are many activities that are fully not covered by the cloud, such as 3D / video / music edition, art in general, programming (all those dev envs)...

- You may want / need to be able to be able to recover from a dataloss even if you are offline.

- Some file types don't match the cloud sync paradigm very well such as system and configurations, non online game saves and all stuff that needs to be at a precise place on the system.

- You got stuff you want to manipulate as files in dirs, not as an entry into app. Power users usually dislike the loss of control and freedom the apps imply.

- You like to have 3 backup because of the rule "one on site, on off site".

- You like to have all your backups at the same place.

- You have collections off files that just don't fit in the cloud, such as Tera of Videos.

- Sex tapes still need to be backed up, and you won't send that to your dropbox.

- Some people still have a shitty internet connections and can't sync reliably.

- It's less work to manage one backup than to setup all the parameters of all those apps to be sure they sync only what you want.

- You read the licence for some sync plateform and couldn't decently click "I accept".

- You got a NAS at home on which you plug your hard drive with the backups for the whole familly.

What's wrong with sex tapes in the cloud?

Ask the celebrities who were the targets of "fappening"

That's why you encrypt sensitive stuff before saving it on dropbox.

I'm glad I had cloudtobutt enabled for your comment.

The scenario you describe involves no true "backups". It's probably better to say that you only have original copies, albeit resilient to failure of local hardware. Cloud services have data-loss events, due to both technical and business reasons. They're vulnerable to a variety of SPOFs out of your control or visibility.

I've known so many people over the decades who trusted some online provider to reliably store the only copy of critical data, only to be burned. Even providers that you would have thought should be bulletproof.

My rule of thumb, based on data loss studies, is to have three copies of any critical data. Any cloud provider should only be considered to be one copy.

It's not only paranoid people who keep offline backups, it's those that understand the threat models where data can be destroyed through logical or physical access to all storage locations.

For arguments sake, I'll focus Google services. As I personally discovered recently, docs offers minimal protection for your data:

    1. Docs shared with you (others are "owner") can disappear without notice.
    2. Manual clones are the only way to keep a copy of shared documents.
    3. The GDrive agent keeps no local copies of docs, only urls.
    4. Any deletions made >25 days ago are unrecoverable.
    5. Deleted accounts have only a 5 day undelete window.

When you combine these it means a document you've contributed to may quickly disappear forever as the result of mis-actions made by someone else or their employer/educational institution. You may have assumed "all changes saved to drive" meant into your account, but you'd be wrong.

Similarly removal of items from trash in GMail/GPhotos is permanent. If someone maliciously gained access to your Google account (or one of your devices) they could quickly and easily purge your data. These deletions would quickly and efficiently propagate to all your devices and purge the canonical "cloud" copy.

The accidental permanent deletion can also occur very easily if you use the GDrive native windows sync tool.

Basically if you don't have any offline backups it is easy to become totally screwed. 2TB drives are cheap.

I have an upload speed of 50 kbyte/s at home. I'm not putting my 2TB NAS data into The Cloud™ (or even just my personal root server) any time soon.

18 years ago, while being a teenager, I misconfigured CCC only to find out that only the active user folders where synced after a full erase/reinstall upgrade of MacOS.

This was my epic "there is to kind of people those who lost data and those who will loose data" story. All my dad and mum files where gone, luckily nothing professional and no pictures (analog camera still ruled back then).

This is just a reminder that no tool is ever perfect. CCC is great, TimeMachine is sluggish but "dumbproof" to a certain extend (but you have to keep faith in a black box).

Personally I'm pretty paranoid so I perform my monthly off-site backup by rebooting my Mac on recovery partition and I perform a full disc copy to an external drive using DiskUtils (because ultimately this will be the recovery tools I'll use if things goes bad so I need to use my off-site backup...).

It really sluggish so I run it while sleeping. I get that for professionals with more than 1To you will need to get to some more serious stuff anyway.

I appreciate this was a long time ago and you've learned a lot since, but your story perfectly illustrates the importance of not just performing backups, but periodically testing them as well. That way you eliminate your blind faith in a black box by ensuring you actually have a working solution to fallback on.

I my case (and I can't emphasis more on my teenager status back then) it was a permission issue, everything looked fine from my perspective, but CCC somewhat didn't have the right permissions to sync folder of others users. So their home folder where actually empty.

From my perspective the backup was a success... This traumatizing error is when I first learned about superuser and permissions. I guess CCC has gone a long way since then and added more safeguard. But I must confess that I've sticked to a "use the goddam standard tools" since then.

Sure there must be a lot of more customizable or faster tools out there. But Time Machine stay "the backup tool my mum can't use wrong" I can't give enough to credit to Apple for that. And putting aside NSA/FBI problem, iCloud backup for iPhone is exactly as smooth. Literally every time I go to a Genius Bar, someone is about to hug a genius because of a successful restore from the iCloud of their broken/stolen iPhone.

This is how you build a "faithful" customer base and no other tech company understand that better than Apple so far.

Those points don't change the fact that you (and everyone else) should test back ups periodically.

If it's not file system permissions nor other configuration problems then it will be a failing storage medium or just dumb user error. So the only way to be sure you have a working back up is to test that back up - ideally before you actually need to use it.

To be clear I can't agree more!

I got a little lost while writing. But indeed my point was double check! Because in my case everything looked fine at first sight.

This is even why I don't use the same process for my two backup (daily/monthly). Because to be fair you can't constantly check that your daily/hourly backup isn't corrupted, you rely on its build-in safeguard and occasionally you check it.

But by using two different methods for daily vs monthly(offsite) backup, you significantly reduce the odds that the two different methods failed at the same time while you need them. (Also a monthly full clone is way easy to check than a Time Machine Disk)

Yeah, once upon a time at a startup, I wanted a file restored. We were small; the CEO was actually the one doing the backups (to QIC tape cartridges).

Of course, the recovery failed. In fact, the backups hadn't been working properly for months. I lost a few hours of work; if our file server had actually failed, not having a good backup could have literally been catastrophic. As in would the company have survived?!

I recall an ancient quip, more or less: "if you don't test your backups, you don't have backups, you have dreams".

I keep hearing this, but as a Mac user who uses Time Machine and CCC, what's a good way to test the backups? After all deleting my only system to attempt a restore seems even more dangerous.

what's a good way to test the backups?

Here's a way to do it. You need to feel comfortable around the OS X terminal command line. You only rely on OS X's built in programs, so it's an independent way to check if your backup program is doing the right thing.

As root, I used to do something like this:

   cd /
   find -x \
      . \
      -type f -print0 \
      | xargs -0 -n 100 -x md5 \
      | sort > /tmp/src.md5

Then I would run the same type of script in the destination directory and diff the results.

There are some annoyances. E.g. (from memory) the ~/Library/Caches files aren't backed up, so they will be missing at the destination. I wrote some sed commands to first remove some of these from the result checksums to keep my diff's more manageable.

Instead of the above, I currently use a Python script that I wrote, that let me fine tune things. At the heart of it is (as root, of course) using Python's os.walk to traverse a directory tree. For each file I use hashlib.sha256 to generate a checksum. I also don't descend into certain directories. Etc.

Using a few Python scripts to generate and process the source and destination checksums allows me to, at the end, use a simple

   vimdiff source.checksums destination.checksums

to quickly see the few differences that remain.

Keep in mind that the perfect is the enemy of the "good enough". Just using the basic 'find' (and not a custom Python script) is plenty good. I used that method for many years.

Many errors stick out right away. E.g. if you have 1,000,000 files checksummed in your source but only 250,000 in your destination, then you quickly know that you screwed up.

Buy a new hard drive and restore to that?

> The AFP protocol is deprecated and cannot be used to share APFS formatted volumes. Interesting.

That is really interesting. I recognised with El Capitan that it already defaults to SMB instead of AFP when you don't give a scheme.

AFP depends on having persistent, globally addressable IDs for files (CNIDs). Perhaps this is no longer available under APFS?

While I imagine this might be retrofittable (netatalk manages it, after all), I can't blame Apple for wanting to ditch AFP. It's an ancient protocol at this point, and I'm frankly just amazed it still works at all.

AFP is also a security disaster. Check out the spec for "DHX2" https://developer.apple.com/library/mac/documentation/Networ...

Can you provide some details about what's wrong with it? (Not a security person, no idea how to evaluate such a claim)

I'm very sad about the lack of AFP. It's so much easier to get going in linux than Samba.

> Famously missing, but not the hardest thing to add considering all the features above: Compression (which HFS+ supports!)

HFS+ supports compression, so compression is not a new feature, so it's not on the New Features page. No problem!

I can live with that! Be sure to make it LZ4!

What will happen to Spotlight search on network shares? At the moment, they are based on AFP.

Maybe even an end to the constant self deletion of the backups due to "inconsistencies".

I found forcing user to SMB weird given how crap their driver is...

Crap driver? It's not a 'driver', and smb has been natively supported for at least 8-10 years. It hasn't sucked in a looong time either. In fact recent OS versions have defaulted to smb v2 for file sharing.

I believe when they flipped the switch to default to SMB it was actually to SMB 3 in 10.10.

Anecdotally, I'd always found AFP in the Tiger and Leopard days to be faster than whichever version of SMB support was included at the time. Now I use the default SMB3 and it seems that 802.11ac and gigabit are bottlenecks (of course its 10 years later in the times of SSD's as well)

Apple deprecated AFP and switched to SMB2 by default in Mavericks/10.9 (https://www.apple.com/media/us/osx/2013/docs/OSX_Mavericks_C...). AFP was used only for Time Machines and connections to older Macs.

And it wasn't just anecdotally faster; I worked for a storage company specializing in Mac workflows and AFP was empirically several times faster, especially on 1GbE and 10GbE networks. This was in part due to Apple ditching Samba in Lion/10.7 (http://appleinsider.com/articles/11/03/23/inside_mac_os_x_10...) over GPL concerns and replaced it with their own shitty, incomplete implementation, which they didn't get up to Samba's standards until 10.10.

I remember this being excruciating since customers had to either buy a third-party SMB implementation like DAVE to get any value out of 10GbE connections, or hope that the applications they wanted to use over the network supported AFP.

Reply on news.ycombinator.com