I recently purchased Veeam Backup 3.0 to back up my three VMware ESXi Free hosts. Veeam Backup is awesome and their version 3.0 is the first version to support the free ESXi version! I love the deduplication and compression and the ease of use when making backups! On March 31st, VMware released ESXi 3.5 Update 4, which added drivers for some very nice NetXtreme quad-port Gigabit Ethernet cards, which I have in two of my three VM host servers but have been unable to use until they released an updated version with built-in drivers for that hardware.

So I upgraded yesterday when Update 4 was released (I actually just did a point release update to new Update 3 firmware the night before…doh!). The new NICs work great and now I have redundant paths to the SAN! (In one case I now have more than one NIC in the whole box that was doing SAN and LAN just on VLANs, so it’s quite nice to have multiple NICs available now!)

I was going through and upgrading VMware Tools on all of my virtual machines (the new release adds some driver support for enhanced NICs to Server 2003 and a few other minor things). One of my Linux CactiEZ VMs was being a bit picky with the yum package I was trying to install so after some troubleshooting I figured I’d restore a virtual machine from Veeam Backup (granted not 3.0.1 which I believe is out, I have the original 3.0 release installed right now) to get an earlier state and see if it helped to start fresh (my other thought was there was a repository issue but my older CactiEZ 0.4 yum was working just fine, it was my CactiEZ 0.6 box I recently set up that was having issues (it runs CentOS 4.7)).

But my restore fails, with an error relating to not being able to create the directory on the ESXi host to restore the virtual machine. The exact error is along the lines of:

Failure to restore item “VM Name Here” Cannot make directory ‘[datastore] VM Name Here’ on ‘ha-datacenter’. Soap fault. fault.RestrictedVersion.summaryDetail: ‘<RestrictedVersionFault xmlns=”urn:intervalvim25″ xsi:type=”RestrictedVersion”></RestrictedVersionFault>’, endpoint: “

The simpler error is in the status dialog box, “Restore error: Restore VM failed: Cannot make dir…”

I dig a little deeper and notice that the last couple of backup jobs scheduled to run overnight for some virtual machines have all failed completely. Nothing updated, and when I force a backup to start now it fails quickly for all VMs with an error along these lines:

Releasing VM files

CreateSnapshot failed, vmRef 224, timeout 1800000, snName “VEEAM BACKUP TEMPORARY SNAPSHOT”, snDescription “Please do not delete this snapshot. It is being used by Veeam Backup.”, memory False, quiesce True

fault.RestrictedVersion.summary

So, I’ve submitted a support ticket. Fortunately, right now I have nothing urgent that needs to be restored (CactiEZ is more of a plaything right now, at least my new 0.6 install), although obviously not keeping backups up to date is not a good thing.

I guess I’ve been running ESXi without Veeam (Veeam’s only been running for…maybe a month?) for long enough that I wasn’t considering backups when I did my ESXi upgrades, so I’ll admit first-day upgrading is jumping the gun. But Veeam is a VMware partner as far as I know, and I don’t know why they haven’t been able to work with VMware around this release to verify that their software works…it’s not like this is ESXi 4, it’s just an Update release of 3.5. At least an announcement of the incompatibility with a warning about upgrading sent to customers would have been nice, although it’s not something that was promised or anything.

I’ll keep this updated (here or in the comments) as the “story” progresses! Tomorrow I will also look into making sure I’m on the very latest point release of Veeam Backup to see if that makes a difference…just don’t have the energy left tonight to do anything else, I was up until 5:30 am last night doing a P2V of our nursery checkin system (long but successful!).

• A Dell PowerVault MD3000i iSCSI SAN (with 10-15 near-line SAS 1TB drives and dual controllers)
• At least one new server (likely a Dell PowerEdge R805 with dual quad-core AMD processors)
• Microsoft Server Datacenter licensing for the same
• An iSCSI Ethernet switch
• Battery backups for server and SAN
• Backup software and some drives for backup
• A rack for the server room (maybe. If I can find a free or dirt cheap used one, locally)

Unfortunately, my budget for all of this as assigned is going to be a bit of a tight fit to squeeze the last few bits in and I’m still trying to figure out the best and most cost-effective way to do this. I’m also struggling a bit with which backup software to use. The server we were using to mirror our data is now dead, and although we have a RAID 5 array that is nowhere near the level of protection we need to have, but I am starting from scratch as far as which data backup software to buy. I have no tapes nor the budget for a tape drive, but I do have some PATA hard drive arrays that are only half-full that are SCSI-attached and will probably work in tandem with an existing server to become my disk-based backup server. Figuring out how to do offsite backup, within the same budget, is high on my todo list :-)

My budget for all this is actually less than the retail price of just the server and SAN. Fortunately, I rarely pay retail and this is certainly no exception! The plan, if isn’t obvious, is to virtualize. I’ve already been doing virtualization for a while to some extent. I used Microsoft Virtual Server a long time ago but switched to VMware Server product when it was released for free. I’m currently running four virtual Windows machines and a virtual Linux machine for our helpdesk software. Recently, since VMware’s ESXi became available for free, I’ve started running it on our newest server, a 3-year-old Dell PowerEdge 1800 Xeon 3.0GHz system with 6GB RAM. It runs very well and I love the management interface, even just using the VMware Virtual Infrastructure Client it comes with (since we aren’t paying for Virtual Infrastructure 3, or VI3 as it’s known!).

The goal of the new system will be to move virtual machines and file shares onto the MD3000i SAN. The Dell R805 server will be the primary virtual machine host, and although my plan was to use VMware ESXi, a good friend and mentor has recommended I examine Microsoft’s new Hyper-V virtualization platform as a strong contender, especially since Virtual Machine Manager (VMM) 2008 is coming and Live Motion will be available in the future, something that VMware does (simliarly, at least) with VMotion only for a high licensing cost. I’m open to either virtualization solution at this point, but here’s my main problem: Hyper-V will only run on procesors that support the new virtualization extensions. Right now, that’s zero of our servers. The R805 will support them of course, but the PowerEdge 1800, our only server with even a 64-bit processor, is too old to have the extensions and thus cannot run Hyper-V or 64-bit guests (although it runs ESXi just fine right now). My plan was to use the PE1800 as a backup server to run critical systems as needed if the R805 was ever down. With Hyper-V, this is no longer an option (and really, with the PE1800 and ESXi I still can’t run any 64-bit guests, so Exchange 2007 is out), and my concern is being left without a secondary server should the primary fail.

So, I’m left trying to fit a second server of some sort, new enough to have virtualization extensions but cheap enough to fit into my already packed budget. Suggestions, and of course donations, are welcome :-) I have been keeping an eye on the Dell Outlet but that is mostly pointless until I have the money to spend same-day when something shows up in stock.

At the moment I’m likely going to get a couple of refurbished UPSs from RefurbUPS for battery backup. However, I will be pursuing a contact or two who may be able to help with this as well.

The final area I’m still investigating still is data backups. There are a plethora of disk-to-disk backup options, and obviously I’m limited by price. I’m not going to use Symantec’s BackupExec for various reasons. The options I am considering so far consist of the following:

• Amanda (Enterprise for Exchange, open source if it will mix with Enterprise)
• Righeous Software’s Continuous Data Protection (CDP)
• Backup Assist
• CommVault (the Small Business version available only through resellers such as Dell)
• Microsoft Data Protection Manager (DPM)

CommVault is the solution that Jason Powell and his team use at Granger Community Church. They like it and it allows for Exchange restore down to the individual message if necessary without restoring the entire data store. The other options I have done varying levels of research on; enough to know they are still a contender on both price and features but not enough to provide an in-depth comparison summary. I am also still working how exactly how many virtual machines I will be running, and how many need to run a backup agent (some things can just be backed up with scripted backups to a file server where the data can be backed up along with everything else on that server, so I don’t necessarily have to have a backup agent for every VM).

The funding will be available most likely in the next couple of weeks or so to go ahead and make these purchases. If I can hold off on some of the backup questions, I may try to wait until after the upcoming Seacoast Fall Church IT Roundtable so I have more time to bounce ideas off of the smart guys there!

This is an overview and of course is not the only research, thinking, and questioning I’ve done about this solution! So feel free to comment and suggest away; I’m just saying there’s a possibility I’ve considered your suggestion and if so, I’ll note it. More often than not though, my thinking is challenged here in the comments, so please delight me with your insights, they are much appreciated :-)

So I restarted the computer, which by that time was finished running software updates and demanding as much anyway. Logged in, and didn’t do anything else other than run DiskWarrior this time. Same process, but this time after it finished the option to Replace the repaired directory structure was available! Click, wait ten-or-so minutes. Ta-da! It works! Nothing like the good ‘ole “reboot” Windows trick to get a Mac fixed!

I grabbed some nice screenshots of DiskWarrior in action (thanks to some quick Googling to determine how to use the built-in screen shot capture feature I knew existed on the Mac — Command+Shift+4, Spacebar, click on window to capture), so I thought I’d give you a sneak peek, in case you want to get your geek on vicariously without going through the near-data-loss experience yourself!

Thanks to the commenters from my last post with suggestions, they were helpful to my sanity over the weekend since I knew I had some good alternate options if DiskWarrior failed! I could find plenty of options on my own, but it’s difficult to sort out the wheat from the chaff without spending a lot of money trying various things, so the recommendations are very much appreciated.

Who needs backup?

Yep, we do! I mentioned in my last post that this server wasn’t originally intended for critical data storage(just temporary video where the RAID 5 redundancy was plenty of backup), but over time with staff turnover and other random happenings, it has sort of become much more critical. Several years ago, backing up the 1.6TB of data on this server was cost-prohibitive. Now? 1TB hard drives are $300 each, and even nice things like the Drobo are $500, which will hold up to 4 of those drives and give 2.7TB of usable data storage for right at about $1700. Or, since the array is 1.5TB, put three 1TB drives in for $1400 and get 1.8TB usable space, which is still 0.2TB ahead of our needs for a mirrored backup.

Is this the best backup solution? I know there are less expensive NAS enclosures out there, some possibly directly-networkable and not just USB 2.0 like the Drobo is currently. Right now our internal data server (RAID 5 array) is being mirrored by another server with another RAID 5 array. We have 1TB of storage and backup between those two servers, but a third backup would be nice to have. What about maxing out a Drobo and backing everything up to it? What about other options? What about picking up an iSCSI SAN from StoreVault for $3k with 1TB, or expanding that a bit and put all our main storage there (including several virtual server hard drives) , and using the existing arrays we have for backups? Lots of questions to answer, and this almost-data-loss is a useful catalyst to demonstrate the need to spend money for backups. And by “useful catalyst” I mean, I have been asked to make sure this can’t happen again :-)

I know similar things have been a hot topic of discussion on the Church IT Podcast and in other similar forums lately, but if anyone has specific information or suggestions, you can make a blogger happy with the comments form :-D Oh yeah, and sorry about that corny title. I couldn’t resist. It’s so much fun to make my wife roll her eyes when I make weird word plays like “HurRAID!” I imagine most of you rolled yours right along with her…

This is a 1.6TB array, around since our previous Evangelism and Media pastor who did quite a bit of video editing and used the array for video storage while working on projects. Fast forward to today, when the Marketing/Communications (MarCom) department is using the system to store–everything. However, we don’t have a backup solution in place because the system wasn’t intended initially as mission-critical storage and the budget hasn’t been there to change that. 1.6TB is the largest amount of storage in one server that we have right now, so there’s no place to even copy the data for a backup, like we do with our primary file server.

I purchased DiskWarrior, which comes highly recommended, but it’s throwing a funky error about a corrupt configuration file when I try to even start it. It says to delete the old configuration file to continue, but it doesn’t exist–thus, a support request has been logged and we wait until Monday to continue the saga (I tried calling their support number but they had just closed for the weekend. Probably for the best, it was dinner time, and family time, and homework time! I didn’t get out of there until 7 pm anyway). Meanwhile we are faced with the prospect that if this doesn’t work, the only option I can think of is to ring up one of those fun services that do awesome data recovery at a price on par with their awesomeness. The array itself says it’s fine, no lost drives or anything weird like you would expect for a failed system, it appears that the partition just got corrupted somehow when the power was unceremoniously yanked. To protect the guilty, I’ll spare you the details of who and why, but let’s just say my happiness level is not quite up where I like it to be…

If you have any recommendations of good (and preferably inexpensive) RAID5, Mac filesystem recovery services, I’m open to suggestions if we end up having to go that way (and can justify the cost given the data). I will update you next week on the progress with DiskWarrior when it gets up and running.

How to change the default temp directory for MozyPro:

1. Make sure you’re running Mozy 1.8.2.3 or later. (You can the latest version at http://mozy.com/downloads/mozysetup.exe)
2. Create a new folder for Mozy to build its files in. Most people use something like D:\mozytemp\
3. Open your registry editor by click the Start menu, then Run, and typing ‘regedit’. Click OK.
4. Open HKEY_LOCAL_MACHINE\Software\mozypro\options
5. Right click the right pane and click New->String Value
6. Change the new value’s name to tempdir
7. Right click the value and select Edit (or double-click on it).
8. Change the value to the name of the folder you created. (If you created D:\mozytemp\, make the value ‘D:\mozytemp\’)
9. Save the changes and close the registry editor.
10. Open a command prompt (click the Start menu, then Run, and type ‘cmd’. Click OK.)
11. Type “net stop mozybackup” and hit Enter
12. Type “net start mozybackup” and hit Enter.

NOTE from Jason: steps 11 & 12 didn’t work for me so I just went into Admin Tools->Services and restarted the mozy service. (David says: same for me.)

The backup size went from 2.2 GB (already backed up) up to another 14 GB of stuff to back up; a lot of that is the Exchange store but SQL isn’t tiny. Because I’m throttling the uploads during the day very heavily, it will probably take maybe a week (total guesstimate) to finish the initial backup, although it may be less as I’ve gotten up to 900 kbps overnight with my first test. Hopefully I won’t run into the SQL issue Jason’s discovered, but I don’t see any indication of that yet. Kudos to MozyPro for excellent and fast support, though!

This morning, I find the backup complete, and additional incremental backups have also been made throughout the night. However, when looking at the backup history, it shows that the initial backup only transferred 2.2 GB of data — far, far less than our Exchange and Shelby SQL databases! About the size of the other files I’d chosen, in fact.

So, no Exchange or SQL backup happened at all. I called MozyPro’s technical support this morning. They answer the phones quickly, and have escalated the issue to a developer because everything looks like it should be working. I have C, D, and E hard disk partitions on this server, with D containing the SQL installation and database and E containing the Exchange installation and database. It’s all hardware-based SCSI RAID, which looks like regular disks to Windows, nothing special there. Yet for some reason the backup client is not detecting either product for backup. Beyond this issue, I’m happy with MozyPro so far and I’m actual glad I get to test their support early on. I’ve been using Mozy personally for several days at home, and I’ve been very impressed with the free version, and may upgrade to the unlimited version at some point soon!