Today there are two final tests (each venue can pick one) where the stream is run for a couple of hours to the venues to make sure things are going smoothly. Last week there were some various hiccups that they found and fixed and this past Monday the test went very smoothly. We ran the afternoon test (the other is tonight) today in our Youth Center where we’re hosting the event and just after the official test, I decided to test our bandwidth with Comcast. I kept adding streams until I was streaming the 2.5Mbps (highest available) stream seven different times! Bandwidth peaked at over 18.5 Mbps downstream with all those streams running at the same time! And I think we had some bandwidth to spare (this is on our Comcast Business internet connection). Our connection is rated for 16 Mbps down and 2 Mbps up, while I’ve seen speed tests recently as high as 30 Mbps down and 4.5 Mbps up. Certainly the almost-19Mbps speed seen here is excellent and above our rating!

I’ve posted a bandwidth graph showing our internet connection’s utilization (also on TwitPic):

Cacti Graph - Town Hall For Hope Test 7x 2.5 Mbps Stream

Comcast Fun

Of course we almost missed the test this morning because someone cut our main Comcast tap this morning just before it went under the parking lot to our building. You can see the actual cut cable (and a part of my shoe) in the picture I uploaded to TwitPic earlier. This caused a four-hour internet outage (8am to noon) that I managed to get back up once I realized (thanks to some prompting from our awesome Facilities Director Mike Moore) that the other end of our building has a completely separate cable tap from Comcast for the TVs on that end of the building! That tap was unharmed so I moved the modem to that IDF and plugged into the tap. I adjusted some VLAN configuration settings to put the firewall’s WAN port on a private VLAN with the modem’s LAN interface (it was plugged in directly before) and tada, at 11:58 am (two minutes before we were scheduled to test the Town Hall For Hope stream) the internet came back!

Comcast did come out later (during the Town Hall For Hope test in fact) and repair the cable that was cut. I’ll be moving the modem back after hours; the TVs are working so I’m going to assume the modem will be fine back on its original line as well. I’m really glad we had that second tap though, because we would have had to push the Town Hall For Hope test off until tonight when the youth group uses the room we’re using, and we wouldn’t have gotten as good of a test. And kudos to Comcast for their fast response to our issues, even though they weren’t the cause.

So, today didn’t quite go as planned, but given the issues I think we had plenty of successes. And I’m not going to worry about blocking free wifi or other bandwidth use during the Town Hall event tomorrow night; since we’re only doing one stream I think we can handle it! In fact, I just realized that if we overflow that Youth Center venue for some reason (which I doubt we will not because it’s not going to be a big event, but because there are so many other churches also hosting it), there’s no reason we can’t handle adding a feed to our main sanctuary as well if necessary. I like being prepared. Just keep the backhoe’s away from the property!

So I restarted the computer, which by that time was finished running software updates and demanding as much anyway. Logged in, and didn’t do anything else other than run DiskWarrior this time. Same process, but this time after it finished the option to Replace the repaired directory structure was available! Click, wait ten-or-so minutes. Ta-da! It works! Nothing like the good ‘ole “reboot” Windows trick to get a Mac fixed!

I grabbed some nice screenshots of DiskWarrior in action (thanks to some quick Googling to determine how to use the built-in screen shot capture feature I knew existed on the Mac — Command+Shift+4, Spacebar, click on window to capture), so I thought I’d give you a sneak peek, in case you want to get your geek on vicariously without going through the near-data-loss experience yourself!

Thanks to the commenters from my last post with suggestions, they were helpful to my sanity over the weekend since I knew I had some good alternate options if DiskWarrior failed! I could find plenty of options on my own, but it’s difficult to sort out the wheat from the chaff without spending a lot of money trying various things, so the recommendations are very much appreciated.

Who needs backup?

Yep, we do! I mentioned in my last post that this server wasn’t originally intended for critical data storage(just temporary video where the RAID 5 redundancy was plenty of backup), but over time with staff turnover and other random happenings, it has sort of become much more critical. Several years ago, backing up the 1.6TB of data on this server was cost-prohibitive. Now? 1TB hard drives are $300 each, and even nice things like the Drobo are $500, which will hold up to 4 of those drives and give 2.7TB of usable data storage for right at about $1700. Or, since the array is 1.5TB, put three 1TB drives in for $1400 and get 1.8TB usable space, which is still 0.2TB ahead of our needs for a mirrored backup.

Is this the best backup solution? I know there are less expensive NAS enclosures out there, some possibly directly-networkable and not just USB 2.0 like the Drobo is currently. Right now our internal data server (RAID 5 array) is being mirrored by another server with another RAID 5 array. We have 1TB of storage and backup between those two servers, but a third backup would be nice to have. What about maxing out a Drobo and backing everything up to it? What about other options? What about picking up an iSCSI SAN from StoreVault for $3k with 1TB, or expanding that a bit and put all our main storage there (including several virtual server hard drives) , and using the existing arrays we have for backups? Lots of questions to answer, and this almost-data-loss is a useful catalyst to demonstrate the need to spend money for backups. And by “useful catalyst” I mean, I have been asked to make sure this can’t happen again :-)

I know similar things have been a hot topic of discussion on the Church IT Podcast and in other similar forums lately, but if anyone has specific information or suggestions, you can make a blogger happy with the comments form :-D Oh yeah, and sorry about that corny title. I couldn’t resist. It’s so much fun to make my wife roll her eyes when I make weird word plays like “HurRAID!” I imagine most of you rolled yours right along with her…

This is a 1.6TB array, around since our previous Evangelism and Media pastor who did quite a bit of video editing and used the array for video storage while working on projects. Fast forward to today, when the Marketing/Communications (MarCom) department is using the system to store–everything. However, we don’t have a backup solution in place because the system wasn’t intended initially as mission-critical storage and the budget hasn’t been there to change that. 1.6TB is the largest amount of storage in one server that we have right now, so there’s no place to even copy the data for a backup, like we do with our primary file server.

I purchased DiskWarrior, which comes highly recommended, but it’s throwing a funky error about a corrupt configuration file when I try to even start it. It says to delete the old configuration file to continue, but it doesn’t exist–thus, a support request has been logged and we wait until Monday to continue the saga (I tried calling their support number but they had just closed for the weekend. Probably for the best, it was dinner time, and family time, and homework time! I didn’t get out of there until 7 pm anyway). Meanwhile we are faced with the prospect that if this doesn’t work, the only option I can think of is to ring up one of those fun services that do awesome data recovery at a price on par with their awesomeness. The array itself says it’s fine, no lost drives or anything weird like you would expect for a failed system, it appears that the partition just got corrupted somehow when the power was unceremoniously yanked. To protect the guilty, I’ll spare you the details of who and why, but let’s just say my happiness level is not quite up where I like it to be…

If you have any recommendations of good (and preferably inexpensive) RAID5, Mac filesystem recovery services, I’m open to suggestions if we end up having to go that way (and can justify the cost given the data). I will update you next week on the progress with DiskWarrior when it gets up and running.

How to change the default temp directory for MozyPro:

1. Make sure you’re running Mozy 1.8.2.3 or later. (You can the latest version at http://mozy.com/downloads/mozysetup.exe)
2. Create a new folder for Mozy to build its files in. Most people use something like D:\mozytemp\
3. Open your registry editor by click the Start menu, then Run, and typing ‘regedit’. Click OK.
4. Open HKEY_LOCAL_MACHINE\Software\mozypro\options
5. Right click the right pane and click New->String Value
6. Change the new value’s name to tempdir
7. Right click the value and select Edit (or double-click on it).
8. Change the value to the name of the folder you created. (If you created D:\mozytemp\, make the value ‘D:\mozytemp\’)
9. Save the changes and close the registry editor.
10. Open a command prompt (click the Start menu, then Run, and type ‘cmd’. Click OK.)
11. Type “net stop mozybackup” and hit Enter
12. Type “net start mozybackup” and hit Enter.

NOTE from Jason: steps 11 & 12 didn’t work for me so I just went into Admin Tools->Services and restarted the mozy service. (David says: same for me.)

The backup size went from 2.2 GB (already backed up) up to another 14 GB of stuff to back up; a lot of that is the Exchange store but SQL isn’t tiny. Because I’m throttling the uploads during the day very heavily, it will probably take maybe a week (total guesstimate) to finish the initial backup, although it may be less as I’ve gotten up to 900 kbps overnight with my first test. Hopefully I won’t run into the SQL issue Jason’s discovered, but I don’t see any indication of that yet. Kudos to MozyPro for excellent and fast support, though!

This morning, I find the backup complete, and additional incremental backups have also been made throughout the night. However, when looking at the backup history, it shows that the initial backup only transferred 2.2 GB of data — far, far less than our Exchange and Shelby SQL databases! About the size of the other files I’d chosen, in fact.

So, no Exchange or SQL backup happened at all. I called MozyPro’s technical support this morning. They answer the phones quickly, and have escalated the issue to a developer because everything looks like it should be working. I have C, D, and E hard disk partitions on this server, with D containing the SQL installation and database and E containing the Exchange installation and database. It’s all hardware-based SCSI RAID, which looks like regular disks to Windows, nothing special there. Yet for some reason the backup client is not detecting either product for backup. Beyond this issue, I’m happy with MozyPro so far and I’m actual glad I get to test their support early on. I’ve been using Mozy personally for several days at home, and I’ve been very impressed with the free version, and may upgrade to the unlimited version at some point soon!

It’s been so good, in fact, that a very long time has past since the runtime on the battery backups was tested.  Today, the test was unintentional.

Fortunately, since server room cooling has become an issue with such an enclosed space being filled with more and more machines, we are finally installing a cooling unit specifically to keep the server room cool.  A big improvement over walking into the server room and starting to sweat almost immediately, to be sure!  However, installing the cooling unit required turning off the power to the server room for a little while.  It was off for a few minutes before I headed up to our all-staff meeting this afternoon, but it was back on before I went to the meeting and the battery backups held just fine.  I knew it would need to be off for a little longer during the meeting, so I hoped the batteries would hold out.  They didn’t.  When you can’t connect to the Exchange server, or even get a new IP via DHCP over wireless, something’s up.  Or, down rather.

I still don’t know how long the battery backups lasted exactly, as everything was already back on when I made it downstairs.  Reboot everything in the right order, and half-an-hour later you’d never know anything had happened.  And with everyone in the staff meeting, I was able to warn them before the meeting broke that I would need to work on the servers for a little while and not to expect it to be operational when they went back downstairs.

It worked out all right in the end, but it’s something I need to address and haven’t had the time or resources available.  Nothing like a little priority-setting all done up nice for you :-)

Anyone have recommendations about on how you go about battery backup selection?  I took the new building opportunity when replacing the core network switches to purchase a Tripp Lite rack-mount UPS unit for each of our three network closets, which so far have worked admirably, were cheaper than comparable APC brand units, and held the network rack up even through this same power outage.

Does it make sense to buy a smaller off-the-shelf UPS for each server, or each pair of servers perhaps, or to purchase one larger unit that can handle everything, even with the sticker-shock price tag?  (Granted, several smaller units do add up themselves.)  I have a feeling I know, but I’d be interested in feedback.