Sunday, June 07, 2015

Surprising space limitations

I bought a new 1Tb portable, external (and "shock proof") USB3.0 hard drive a little while ago. But I've already got so many CT scans and 3D model objects that it's filling up. At the moment I only have one set of CT stacks and models (which are then backed up).  I don't like the idea of having a small 'working set' of objects and CT scan tiff stacks on my portable drive and then another set backup to a few places. For the CT stacks I guess that can work, since they don't normally need to be edited; new stacks can just be deposited and the stack collection can accumulate over time. But for the models that's definitely not a good idea, I'd very quickly start loosing track of what was "in" each model. I'd think that the only way to deal with that is with some sort of model version control system, which in some ways would be nice, but that's an entire area that I don't think I can even look into at this point, and while models don't take up as much space as CT-stacks, they're not small either.

 I think for the duration of the PhD I can stick with a portable drive that has all my data, CT stacks, and objects, but after that I'll almost certainly have to start splitting things up. Right now I'm using Robocopy (and also dropbox) to back up everything, and even in that process I have to split out a few things, (like separating my 3D objects from my CT-Stacks, instead of having them in the same subfolder). A problem with Robocopy (and Dropbox, if you have a shared folders) is that it's pretty easy to delete and entire set of data. Telling Robocopy to "mirror", instead of "copy"  a small, working set of CT-Stacks on one drive to a backup would delete anything on the backup that's not in the 'working' set. There are distinct commands for preventing that, but the huge amount of damage that'd result from even a rare 'mistake' in commands is too high risk. And for dropbox it's almost worse; it's easy to forget that they folder someone shares with you is editable.

The problem with CT-stacks is that for some I'm the person that scanned them and the AMNH doesn't store those files long term, so my copies are the only copies. And even for loaned items, you can't go back to the loaner and say 'hey I lost those files, can you resend them". I'd hate to loose a model that I spent a long time working on, but as long as I have the CT stack I can more or less recreate the model, but without the stack, there's nothing.

No comments: