Encrypted ZIP files as Sparse Bundle Backup, Mac OS X

Because: Sparse Bundles Fail!!!

MacOS has long had a wonderful feature, called a Sparse Bundle. For me, I use them mostly for data which I wish to encrypt. They work so well, and remain stable for so long, that they have lulled me into complacency, and then ...

... they fail!

In nothing other than my personal experience, when they fail, they do so spectacularly, and I can never again open them. Everything inside has disappeared, forever. I have found many web pages explaining how to recover from that, and nothing has worked.

Getting the files back from Time Machine or other backups did not work. It feels that something deeper has broken.

Inconvenient doesn't even come close to describing the consequences.

I have come up with a method for recovery, which involves using the zip command from Terminal, and if none of that makes sense to you, then let me know.

The ZIP files which this method creates contains all the files, stored recursively, i.e. in a folder structure identical to that on the source, and that us what we want. ZIP files don't provide anywhere near the convenience, for regular file access, as do Sparse Bundles, so I used them solely as a rescue/recovery tool.

I have to do this semi-manually—inconvenient, but not as inconvenient as unrecoverable data-loss—and how often I will leave to you. Some of my Sparse Bundles I back up daily.

Better methods surely exist. I have no formal training in this area. I cannot promise any level of data protection to you; I leave responsibility for your data with you. Feel free to consult a professional if your data matters to you enough. I can only document what I do.

Example Names

You will have your own names to substitute in for the example-names I have chosen. Simply for purposes of illustration, I will use:

• Outer1 – an encrypted sparse bundle
• Inner1 – the volume which appears when you open Outer1.sparsebundle
• Inner1A – a folder in Inner1
• Inner1B – another folder in Inner1
• Outer2, Inner2, Inner2A, Inner2B – a second sparse bundle and its similarly-structured contents
• Destin – another drive (volume) on which to keep the ZIP-file backups (optional, but the further you keep your backups from the original sparse bundle, the safer, I think)
• Dest1 – a folder on Destin, for Inner1
• Dest2 – the same, but for Inner2
• Inner1.zip – the ZIP file I will create and update to store the contents of the volume Inner1. (You don't have to use the same name; I find it intuitive, but you may find it confusing.)
• I find it helpful to keep all of the above names in a text file, which itself you may wish to keep inside the encrypted sparse bundle.

You do not have to replicate my folder-structure; I have concocted one simply for purposes of illustration. You can manage your folders any way you like, e.g. you can plunk the ZIP files more deeply nested, you can combine them ... anything you wish.

My Procedure

Before your create a ZIP file for the first time, you need to make any destination folders you may want. The zip command will not create them for you.

• Open your encrypted Sparse Bundle (e.g. Outer1.sparsebundle); it requires the password every time unless you check Remember password in my keychain, but that would rather defeat the purpose.
• If you need to open a network drive for the ZIP-file destination, do that also (e.g. Destin).
• Open Terminal, and enter the following commands, substituting your own names for the volumes and folders.

ZIP file initial creation

cd /Volumes/Inner1
zip -er /Volumes/Destin/Inner1.zip ./ -x "*.DS_Store"

That's as simple as I can make it. It takes all of the contents of Inner1 and ZIPs them into a file called Inner1.zip right on the root-folder of Destin.

You will be asked for a password. (You could use the same one as for the associated Sparse Bundle.)

Theoretically, you only have to do this once.

The part of the command which says ./ must be entered without quote marks, whereas other uses of the zip command require them. Do not ask me why. Also, if you omit -x "*.DS_Store", the procedure can crash and burn decisively. I don't completely understand why. But it does.

Note: if the volume Inner1, or a folder within Inner1, is small, and the destination is on the same, internal drive, this process will be fast. If Inner1 (or the folder) is large, or if the destination is on a distant network drive, it will be slow, as in many minutes or longer. That's why I sometimes do this by folder, not doing the entire source volume all at once. I personally wouldn't do this over the Internet: too slow. For storage of the ZIP files in the cloud, I make them more locally and then copy them to a secure Internet location.

ZIP file updating

Once you have created a destination ZIP file, such as the example Inner1.zip, then again open (mount) both of the sparse bundle and the destination-volume (as above), and use the following commands to maintain it.

cd /Volumes/Inner1
zip -ru /Volumes/Destin/Inner1.zip ./ -x "*.DS_Store"

It's almost exactly the same as the creation command except for the -ru part of the command instead of -er.

As far as I know, this will update changed files, and add new ones, but not remove files which you have deleted on the source volume. If you really need that, then I think that you have to start from scratch, but often it might not matter.

The above command will not ask you for the password, which you will of course need if you ever wish to extract the files from the ZIP file.

In that event, open it (e.g. double-click Inner1.zip), enter the password when asked, and (depending on the size of the file) go for a coffee, and eventually all of your files will appear.

Semi-automating

Now, let's make it more complicated, but better.

MacOS has ways to automate processes, using scripts for example, but I have learned this somewhat more cumbersome but understandable method: I keep the commands in a text file, from whence to copy and paste them into Terminal (then press <Enter>).

Let's say that i) I have two sparse bundles, and ii) I wish to ZIP some of their enclosed folders separately. Using the example folder-names listed above:

cd /Volumes/Inner1/Inner1A &&
zip -ru /Volumes/Destin/Dest1/Inner1A.zip ./ -x "*.DS_Store" &&
cd /Volumes/Inner1/Inner1B &&
zip -ru /Volumes/Destin/Dest1/Inner1B.zip ./ -x "*.DS_Store" &&
cd /Volumes/Inner2/Inner2A &&
zip -ru /Volumes/Destin/Dest2/Inner2A.zip ./ -x "*.DS_Store" &&
cd /Volumes/Inner1/Inner2B &&
zip -ru /Volumes/Destin/Dest2/Inner2B.zip ./ -x "*.DS_Store"

That shows the update command using -ru, but the first time, you will use the create command, using -er.

And again, you can store the ZIP files in whatever arrangement you like.

The && ar the end of a command makes the process simply stop if the command fails, so for example if you forgot to have all of the source and destination drives and/or folders open/mounted, the zip coomand won't keep flailing away, trying to do something which it cannot do.

I hope that the foregoing helps someone hold onto their data!

Top