Encrypted ZIP files as Sparse Bundle Backup, Mac OS X

Because: Sparse Bundles Fail!!!

MacOS has long had a wonderful feature, called a Sparse Bundle. For me, I use them mostly for data which I wish to encrypt. They work so well, and remain stable for so long, that they have lulled me into complacency, and then ...

... they fail!

In nothing other than my personal experience, when they fail, they do so spectacularly, and I can never again open them. Everything inside has disappeared, forever. I have found many web pages explaining how to recover from that, and nothing has worked.

Getting the files back from Time Machine or other backups did not work. It feels that something deeper, inside the System, has broken. Inconvenient doesn't even come close to describing the consequences.

I have come up with a method for recovery, which involves using the zip command from Terminal, and if none of the foregoing makes sense to you, then let me know.

The ZIP files which this method creates contains all the files, stored recursively, i.e. in a folder structure identical to that on the source, and that us what we want. ZIP files don't provide anywhere near the convenience, for transparent file access, as do Sparse Bundles, so I use them solely as a rescue/recovery tool.

I have to do this semi-manually—inconvenient, but not as inconvenient as unrecoverable data-loss—and how often I will leave to you. Some of my Sparse Bundles I back up daily.

Better methods surely exist. I have no formal training in this area. I have for example no idea about Apple Scripts, and am only passingly familiar with Apple Automator. My method is not auto-correcting, and when it doesn't work—always because of an error of mine, to date—then I have to go into Terminal and correct it myself.

Most importantly, I cannot promise any level of data protection to you; I leave responsibility for your data with you. Feel free to consult a professional if your data matters to you enough. I can only document what I do.

Example Names

You will have your own names to substitute in for the example-names I have chosen. Simply for purposes of illustration, I will use:

• Outer1 – an encrypted sparse bundle
• Inner1 – the volume which appears when you open Outer1.sparsebundle
• Inner1A – a folder in Inner1
• Inner1B – another folder in Inner1
• Outer2, Inner2, Inner2A, Inner2B – a second sparse bundle and its similarly-structured contents
• ZIP1 – a folder for the ZIP files ZIPped from those in Inner1
• ZIP2 – the same, but for Inner2
• SparseZipHold – a folder on your internal hard disk to hold these ZIP files or folders
• Destin – another drive (which appears as a volume, but not all volumes are external drives) on which to keep the ZIP-file backups, instead of or in addition to the interal drive (optional, but the further you keep your backups from the original sparse bundle, the safer, I think)
• Inner1.zip – the ZIP file I will create and update to store the contents of the volume Inner1. (You don't have to use the same name as that of the source-volume; I find it intuitive, but you may find it confusing.)
• I find it helpful to keep all of the above names in a text file, which itself you may wish to keep inside the encrypted sparse bundle.

You do not have to replicate my folder-structure; I have concocted one simply for purposes of illustration. You can manage your folders any way you like, e.g. you can keep any of the folders and files more deeply nested, and you don't have to use separate source or destination folders ... you can combine them any way you wish.

My Procedure

Before your create a ZIP file for the first time, you need to make any destination folders you may want. The zip command, as presented, will not create them for you.

• Open your encrypted Sparse Bundle (e.g. Outer1.sparsebundle); it requires the password every time unless you check Remember password in my keychain (but that would rather defeat the purpose).
• If you need to open an external and/or network drive for the ZIP-file destination, do that also (e.g. Destin).
• Open Terminal, and enter the following commands, substituting your own names and paths for the volumes and folders.

ZIP file initial creation

cd /Volumes/Inner1
zip -er /Volumes/Destin/Inner1.zip ./ -x "*.DS_Store"

Or, to keep the ZIP files on the internal drive, it could look something like this:

cd /Volumes/Inner1
zip -er ~/SparseZipHold/ZIP1/Inner1.zip ./ -x "*.DS_Store"

That's as simple as I can make it. It takes all of the contents of Inner1 and ZIPs them into a file called Inner1.zip, in a location of your choice.

You will be asked for a password. (You could use the same one as for the associated Sparse Bundle.)

Theoretically, you only have to do this once.

The part of the command which says ./ must be entered without quote marks, whereas other uses of the zip command require them. Do not ask me why. Also, if you omit -x "*.DS_Store", the procedure can crash and burn decisively. I don't completely understand why. But it does.

Note: if the volume Inner1, or a folder within Inner1, is small, and the destination is on the same, internal drive, this process will be fast. If Inner1 (or the folder) is large, and the destination on another external- or network-drive, then it will be slower, as in many minutes or longer, and it's for reasons of speed that I will sometimes do this by folder, not doing the entire source volume all at once.

I for example wouldn't even attempt this over the Internet, to a drive in the cloud, at the speeds available to me. It is so much faster if done on a cabled external drive or ever faster on an internal drive, after which one can optionally copy the ZIP files to a secure remote location, at your leisure.

ZIP file updating

Once you have created a destination ZIP file, such as the example Inner1.zip, then to maintain it, again open (mount) both of the sparse bundle and the destination-volume (as above), and use the following commands.

cd /Volumes/Inner1
zip -ru /Volumes/Destin/Inner1.zip ./ -x "*.DS_Store"

It's almost exactly the same as the creation command except for the -ru part of the command instead of -er.

As far as I know, this will update changed files, and add new ones, but not remove files which you have deleted on the source volume. If you really need that, then I think that you have to start from scratch, but often it might not matter.

The above command will not ask you for the password, which you will of course need if you ever wish to extract the files from the ZIP file.

In that event, open it (e.g. double-click Inner1.zip), enter the password when asked, and (depending on the size of the file) go for a coffee, and eventually all of your files will appear.

Semi-automating

Now, let's make it more complicated, but better.

I keep the commands in a text file, from whence to copy and paste them into Terminal (then press <Enter>).

Let's say that i) I have two sparse bundles, and ii) I wish to ZIP some of their enclosed folders separately. Using the example folder-names listed above:

echo "Starting Sparse-to-ZIP update process ..."
cd /Volumes/Inner1/Inner1A &&
zip -ru /Volumes/Destin/ZIP1/Inner1A.zip ./ -x "*.DS_Store" &&
cd /Volumes/Inner1/Inner1B &&
zip -ru /Volumes/Destin/ZIP1/Inner1B.zip ./ -x "*.DS_Store" &&
cd /Volumes/Inner2/Inner2A &&
zip -ru /Volumes/Destin/ZIP2/Inner2A.zip ./ -x "*.DS_Store" &&
cd /Volumes/Inner1/Inner2B &&
zip -ru /Volumes/Destin/ZIP2/Inner2B.zip ./ -x "*.DS_Store" &&
echo "Sparse-to-ZIP update process completed successfully"

That shows the update command using -ru, but the first time, you will use the create command, using -er.

And again, you can store the ZIP files in whatever arrangement you like.

The && ar the end of a command makes the process simply stop if the command fails, so for example if you forgot to have all of the source and destination drives and/or folders open/mounted, the zip command won't keep flailing away, trying to do something which it cannot do (in which case you might have to stop it with CTRL-C).

Even Better (but optional)

cd SparseZipHold/ZIP1/Inner1 &&
scp Inner1A.zip /Volumes/Destin/ZIP1/ &&

Then fill in other lines as desired to account for other files and folders, e.g.:

cd SparseZipHold/Inner2 &&
scp Inner2B.zip /Volumes/Destin/ZIP2/ &&

By the way, I have tried using wildcards, such as "*.zip", and it simply does not work. I must specify each source file individually.

I hope that the foregoing helps someone hold onto their data!

Top