Also, it’s much easier for an Exchange 2010 and Exchange 2007 box to cohabitate on a network and still allow ActiveSync and OWA access than doing the same with Exchange 2007 and Exchange 2003 (which requires a separate Exchange 2007 CAS, or Client Access Server). Granted, making it work with the ISA firewall was a little tricky, but with a little experimentation it went well and is working fully. So well in fact, that only my Mac user and my Blackberry user are on the old 2007 box now until I stuff is compatible (in the Blackberry case) and I can babysit the migration (in the Mac user’s case, with Entourage–Snow Leopard isn’t an option on our PowerPC hardware). Those will come soon enough. But frankly with Google for the help docs and processes (there’s a lot of good information directly from Microsoft out there already!), the process only required two remote nights working until 3:30am, and some time during one day to work out the ISA stuff to keep ActiveSync and OWA working.

I’m not going to elaborate on the entire installation process here. Microsoft documents it well, it requires installing Exchange 2010 on a new server (no in-place upgrades) to do the transition (that’s how I prefer it anyway, and with virtualization that’s easy!). But it was mostly smooth, similar to 2007 in many ways (different enough to require some reading but familiar enough it was much easier to pick up than 2007 was from 2003). And, as I discovered this morning, for Outlook 2003 clients to connect, you should also run this in the Exchange PowerShell console:

Set-RpcClientAccess -Server [servername] -EncryptionRequired $false

Otherwise, Outlook 2003 will stare at you (or, rather, the user) blankly and not connect (at least if you have internal encryption to Exchange disabled, which I do–I didn’t test enabling it).

Do I recommend going with 2010 now? Yes, as long as stuff you use like Blackberry and Mac supports it or you’re prepared to learn how to make it work. Also, your “now” may not be the day of General Availability depending on the size of your environment and current needs and plans :-)

Any thoughts? Do you think I should have gone with Exchange 2010 the week it was released? I think it’s a reasonably well proven product even though I didn’t participate in the testing myself like I did with Windows 7. Are you migrating soon? (Microsoft likes to call moving from one version to another of the same software a “transition.” I like the term “migration” better, but whatever. They reserve that for when you “migrate” from one of their competitors. I don’t care :-)

However, there is a flag you can set on a moderated object that will allow a moderator for a “parent” group to moderate an email once regardless if subgroups also require modification. Think a moderated all-staff list that contains a moderated group for a specific department; by default both the all-staff moderator and the department list moderator would have to approve a message to all-staff before the department recipients would receive it. If you’d rather have some groups like all-staff set so whoever moderates a message to that group auto-approves any subgroups as well (this is precisely why I wanted it, although we don’t have moderated subgroups yet), that’s why they added the flag called “BypassNestedModerationEnabled” which you can set to true with PowerShell.

The problem is, the few places that talk about that flag online call it a completely different name! Sure you can do “get-help Set-DistributionGroup -full” to see all the options (there are many) or you can find the same help online, but it’s not easy to track down if you’re looking for the wrong setting name! The correct syntax to enable this moderation bypass on a group (from within the Exchange PowerShell console) is:

Set-DistributionGroup -Identity "[group name]" -BypassNestedModerationEnabled $true

However the Exchange Team’s official blog says in it’s moderation post, in the FAQ section where it mentions nested approvals (near the end of the post), “If you set the BypassModerationEnabled flag to $true on the parent group, any messages sent to that group will bypass moderation by child groups.” Close, but it’s actually the BypassNestedModeration flag. If you do some searching, you’ll find a TechNet article called Understanding Moderated Transport which, again near the end in the Handling Multiple Moderated Recipients section, says, “To do this, you set the AutoApproveNestedDLEnabled parameter of the moderated distribution group to $true.” Which provides an even farther-off version of the same thing! At least with the correct version, you can more easily look it up in the TechNet Set-DistributionGroup topic where is is correct!

It’s likely the incorrect articles were both correct at the time they were written, during beta and release candidate cycles of Exchange 2010, with the final flag name being changed in the generally available version that came out this past Monday. I don’t know for sure as the GA version is all I’ve run, but it seems a likely explanation given that the articles are almost a month (the TechNet one) and five months (the Exchange Team blog) old. But apparently I’m the first person to write about it outside of them (that Google knows about).

So I make it through the renewal process, which required generating a new CSR (Certificate Signing Request) for a brand new certificate from the server since the original one had a bit length of 1024 and GoDaddy only accepts 2048 to 4096 bit lenghts (this is a new requirement). After completing the process and getting the certificate installed, I got a nice helpdesk call from a user this morning who has a Centro: “SSL certificate not accepted due to possible expiration.  Check device date & time and re-sync.”

Joy oh joy, exactly what I’d been looking for, another problem and wasted time!

OK, enough sarcasm (but really, can you ever have enough?). Time for Google and Daryl Hunter from the Church IT Roundtable! Although GoDaddy auto-renewed my SSL certificate, I was actually contemplating buying one of their UCC certificates to be ready for when we went to Exchange 2007. Fortunately I read Daryl Hunter’s post about Exchange 2007 without UCC certs, and stuck with the regular certificate for now, because per Palm KB article 43375, certificates with Subject Alternate Names (SANs), such as UCC certs, are not supported at all on Palm devices (“SSL v3 certificates which rely on the Subject Alternate Name field to do load balancing across virtual site names do not work with Palm OS devices.”). So a UCC cert isn’t even an option for me, but it’s cheaper to do Daryl’s method anyway! For now I don’t have to worry about it, since I just have Exchange 2003 for now, and that’s not the present issue (but we will likely be on Exchange 2007 or Exchange 2010 by the time the certificate expires). Additionally, the same article (which has a tool for installing new trusted root certificates on some Palm OS devices–but I didn’t want to mess with touching every single Palm OS device here! And, the tool works on Windows 2000 or XP only, not Vista (and I’m sure not Windows 7 either)) specifically states that, “GoDaddy Class 2 certificates do not work with Palm OS devices.” Time to drop GoDaddy!

Daryl’s favorite SSL certificate vendor (and now, mine too!) is RapidSSL Online. They sell certificates from RapidSSL.com for $17.95 per year (or cheaper, for multiple years), and they’re single root certificates (which menas you don’t have to install intermediate certificates on your server). While RapidSSL Online is cheap, RapidSSL.com directly has a 30 day trial certificate you can sign up for to test for a month, and this is the way I went. When that certificate expires I’ll be purchasing a multi-year certificate from RapidSSL Online, but I wanted to make sure it would work, and it does! I don’t know for sure, but it appears that RapidSSL.com is the company holding the root certificate, while RapidSSL Online is either a reseller or a sub-company of the parent selling the certificates at a discount (the RapidSSL.com certificates aren’t expenive but still cost a lot more than from RapidSSL Online!). Either way, RapidSSL Online claims that their RapidSSL certificates are issued by RapidSSL.com so they should be the same (I haven’t made a purchase yet), and Daryl Hunter has used RapidSSL Online successfully for years across multiple installations.

I generated a new CSR for a new certificate, again (just like I had to do for GoDaddy). I installed the free certificate on my Exchange server’s IIS (I also then exported it and imported the .pfx file onto my ISA 2004 firewall since it does the authentication up front for external clients, but that’s a pretty unique case and in most cases you want this done on the Exchange server). They were right, it’s just a single root on the certificate, signed by Equifax! I had my Palm Centro users (two had complained by this point) try syncing again. It worked! My iPhone also works fine still, and I haven’t had any negative reports from the four Palm Pre users here either. None of my users have Windows Mobile, and my one Blackberry user connects though Blackberry Professional Server rather than with ActiveSync.

So, adios GoDaddy SSL; fortunately they will refund all but $15 of my certificate (for processing since it was issued), and I’ll still come out ahead with RapidSSL Online (GoDaddy was $60 for two years, while RapidSSL Online is only $70 for five years!).

One thing I’ll have to be careful of when I go to Exchange 2007 is that once I use Windows Server 2008 to generate the CSR, it appears I will need to go to extra pains to make sure the CSR is in Printstring format instead of UTF-8, as Palm OS doesn’t support UTF-8 certificates either (Server 2003 uses Printstring by default). Daryl located this useful post while helping me troubleshoot: Ranting about Palm Centro Versamail ActiveSync and SBS 2008. Useful info, I’m sure I’ll be going back when it’s time to renew next time and Server 2008 is in place. By then, I hope we are Palm OS-free; although I loved my Treo 600 and Treo 650 both, the web is littered with forum and blog posts from people who have SSL issues with Palm OS devices (the Palm Pre and Pixi are much more flexible and up-to-date with the Palm WebOS). I was happy GoDaddy “just worked” in the past, frustrated that they “just didn’t work” this time, and happy to save money and move to a company that’s quicker/faster/easier!

Near the beginning of the year, I did a P2V (Physical to Virtual) move of the server onto our VMWare infrastructure from the old desktop. Our network, when the checkin system (Parent Pager Plus) was set up seven or eight years ago (before I was hired and was just an occasional volunteer!), wasn’t really reliable from one end of the building (where the server room is) to the other end where the checkin system was located, and thus the “desktop” server placed local to the checkin stations, which were at that time isolated from the rest of the network behind a Linksys cable/DSL router (for security). It worked, mostly, especially when we upgraded to new (but low-end) desktops for the actual checkin stations rather than the first systems we used that were only supposed to support Windows 2000 Professional and had countless hangs, errors, and just weird random stuff happen. The new systems practically ran themselves!

We built a large building addition, including a new lobby, and moved the checkin stations and server a couple of years ago. But none of the hardware changed (we added a few stations and got some (not all) of the stations set up with LCD touchscreen monitors over the years, too). A part of the new building included a new core network including managed HP ProCurve switches with fiber optic connections between the MDF and two IDFs (one of them brand new). The infrastructure could now reliably support moving the server into the server room and into more reliable hardware, so like I said, P2V was the solution! It worked great, except the server was also a Domain Controller for it’s own Active Directory subdomain, and some things didn’t go quite right with the P2V and Active Directory, and replication failed with my main domain controllers. I won’t go into details, but suffice it to say don’t P2V a DC, at least not without knowing what special precautions to take :-) After 60 days of not talking to my other Domain Controllers, the tombstone period was past by the time I looked at it, and I ended up needing to manually remove the entire subdomain from Active Directory, which is beyond the scope of this post. Suffice it to say, I managed to do so, and then I spun up a new virtual machine, running Server 2008, setting it up as a Domain Controller and recreating the subdomain I’d just cleaned up. Before I did this, I went to each checkin station and unjoined it from the old domain, and then re-joined them to the new domain.

Why set up a whole subdomain for checkin stations? Cleanliness and separation/security mainly. It’s not as important now with our current network but I still have the whole system on a separate subnet and VLAN (no Linksys router now :-) and pretty isolated. The clients and the virtual server are the only thing other than the firewall/router that’s on the subnet. And it’s what I did last time, and even though I basically ripped everything out, I was happy with the design decisions still, just not the implementation. So it’s still a subdomain, but with a Server 2008 DC that’s properly replicating to my other DCs.

What else changed? Well, we’re running SQL Express rather than MSDE 2000, for one. Also, Windows XP’s new Client Side Preferences addon was released, adding a ton of easy control via Group Policy! Using the new Preferences, I was able to reduce the user permissions while still allowing things like hidden drive maps to utilities, forcing custom registry entries to be maintained on login for many Parent Pager Plus settings that the checkin systems all shared (so if you log off and back on or reboot, those common settings return to their correct defaults regardless of whether they had been changed). I even customized the screen saver that says “TOUCH HERE TO START” in the Marquee so it is automatically pushed down to each client with the correct text and timeouts! Basically, the environment for each checkin station is very controlled with limited visibility, but there’s enough there to make troubleshooting easy if you know what to look for. I was also able to use the Preferences targeting options to very easily apply different registry settings in some cases to the checkin stations used at the manned desk area vs. the unmanned stations, so Parent Pager Plus defaults to the correct (but different) username at each login, for instance. The flexibility in the Preferences is absolutely amazing, and is the missing piece that I wished I’d had the last time I tried locking the systems down years ago with Group Policies when I failed. All checkin stations are not only joined to the domain but log in to a common domain username instead of local users. Although there are a lot of tweaks in Group Policy, there are only a couple of GPOs and thus policy processing time is short and the computers boot reasonably fast given their age.

I basically spent two (long) days dedicated entirely to this project, on a Monday and Tuesday one week in late July. In those two days, I managed to convert the old subdomain to a new one on a new server with a new database, restored the database from the old server’s backup, upgraded Parent Pager Plus to the newest version (forgot to mention this earlier but it needed to be upgraded so I went ahead and did it while I was working on it already), rejoined all computers to the new domain, set up group policies in excruciating detail and tested extensively. I think the efford was well worth it and the result is a system that feels current and up-to-date even though the hardware is still years old and I spent nothing but time! It feels good to complete a project quickly and successfully. If you have questions about any of the process including Group Policy Preferences, let me know. If I took the time to detail every change I made to do the lockdown, I’d spend a lot more time on this post and ever get it published, but my original intention was to document it all here. That may come later, but if you have specific questions let me know!

I recently purchased Veeam Backup 3.0 to back up my three VMware ESXi Free hosts. Veeam Backup is awesome and their version 3.0 is the first version to support the free ESXi version! I love the deduplication and compression and the ease of use when making backups! On March 31st, VMware released ESXi 3.5 Update 4, which added drivers for some very nice NetXtreme quad-port Gigabit Ethernet cards, which I have in two of my three VM host servers but have been unable to use until they released an updated version with built-in drivers for that hardware.

So I upgraded yesterday when Update 4 was released (I actually just did a point release update to new Update 3 firmware the night before…doh!). The new NICs work great and now I have redundant paths to the SAN! (In one case I now have more than one NIC in the whole box that was doing SAN and LAN just on VLANs, so it’s quite nice to have multiple NICs available now!)

I was going through and upgrading VMware Tools on all of my virtual machines (the new release adds some driver support for enhanced NICs to Server 2003 and a few other minor things). One of my Linux CactiEZ VMs was being a bit picky with the yum package I was trying to install so after some troubleshooting I figured I’d restore a virtual machine from Veeam Backup (granted not 3.0.1 which I believe is out, I have the original 3.0 release installed right now) to get an earlier state and see if it helped to start fresh (my other thought was there was a repository issue but my older CactiEZ 0.4 yum was working just fine, it was my CactiEZ 0.6 box I recently set up that was having issues (it runs CentOS 4.7)).

But my restore fails, with an error relating to not being able to create the directory on the ESXi host to restore the virtual machine. The exact error is along the lines of:

Failure to restore item “VM Name Here” Cannot make directory ‘[datastore] VM Name Here’ on ‘ha-datacenter’. Soap fault. fault.RestrictedVersion.summaryDetail: ‘<RestrictedVersionFault xmlns=”urn:intervalvim25″ xsi:type=”RestrictedVersion”></RestrictedVersionFault>’, endpoint: “

The simpler error is in the status dialog box, “Restore error: Restore VM failed: Cannot make dir…”

I dig a little deeper and notice that the last couple of backup jobs scheduled to run overnight for some virtual machines have all failed completely. Nothing updated, and when I force a backup to start now it fails quickly for all VMs with an error along these lines:

Releasing VM files

CreateSnapshot failed, vmRef 224, timeout 1800000, snName “VEEAM BACKUP TEMPORARY SNAPSHOT”, snDescription “Please do not delete this snapshot. It is being used by Veeam Backup.”, memory False, quiesce True

fault.RestrictedVersion.summary

So, I’ve submitted a support ticket. Fortunately, right now I have nothing urgent that needs to be restored (CactiEZ is more of a plaything right now, at least my new 0.6 install), although obviously not keeping backups up to date is not a good thing.

I guess I’ve been running ESXi without Veeam (Veeam’s only been running for…maybe a month?) for long enough that I wasn’t considering backups when I did my ESXi upgrades, so I’ll admit first-day upgrading is jumping the gun. But Veeam is a VMware partner as far as I know, and I don’t know why they haven’t been able to work with VMware around this release to verify that their software works…it’s not like this is ESXi 4, it’s just an Update release of 3.5. At least an announcement of the incompatibility with a warning about upgrading sent to customers would have been nice, although it’s not something that was promised or anything.

I’ll keep this updated (here or in the comments) as the “story” progresses! Tomorrow I will also look into making sure I’m on the very latest point release of Veeam Backup to see if that makes a difference…just don’t have the energy left tonight to do anything else, I was up until 5:30 am last night doing a P2V of our nursery checkin system (long but successful!).

One of my goals for the new system is redundancy. If something goes down, I’d really like a second system around to stay up and running, in particular when using virtualization it does create bigger single points of failure (one phyical server going down takes down multiple virtual servers), which is one of its biggest weaknesses. This can be mitigated by using shared storage (hence the SAN) and multiple servers that can take on the virtual machines the “down” physical machine can’t run temporarily, even if it runs a bit slow from the additional load (even better if your secondary server is not heavily loaded!).

Having the R805 would be great, but my next-best server is a Dell PowerEdge 1800 that’s three years old, with a single Xeon 3.0GHz processor and 6GB of RAM. It’s a very nice server, but it wouldn’t be able to shoulder a load the R805 could handle easily so it would only be able to run absolutely critical machines. Additionally, the Xeon processor is too old to support Intel’s VT (Virtualization Technology) extensions that make running a virtual server hypervisor easier, and allows 64-bit guest operating machines to run. If we make the move to Exchange 2007, there would be no backup server for it to move to, and it would be one of our most critical servers!

VMware’s ESXi hypervisor runs just fine on the PowerEdge 1800 (as long as you don’t need 64-bit guests); I have it running now with a couple of VMs and it’s barely breaking a sweat. It will work even better on the nice “new” PowerEdge 1950’s, but the other thing that having VT-capable processors will help me with hypervisor selection options. If I don’t end up running the now-free VMware ESXi, which I’ve tried and like but keeps you from using some of the really cool features unless you buy their Virtual Infrastructure/Virtual Center packages (these easily get into the same price as the hardware for both servers I bought!). Microsoft’s new Hyper-V virtualization platform (separate entirely from the old Microsoft Virtual Server product) is capable of running only on processors supporting VT, and now I’ll have two of them, the magic redundant number. It does preclude using the PowerEdge 1800 as an third backup, but down the road it will offer some of the similar moving of “live, running” virtual machines from host to host, and Microsoft is releasing their Virtual Machine Manager (VMM) soon as well for management, which will be siginificantly more cost-effective for us given charity pricing than VMware’s Virtual Center. But some of this stuff isn’t going to be here right away, and VMware’s solutions aren’t necessarily lacking anything we desperately need. What’s the answer? For now, it’s keep researching, maybe even try both when the servers arrive, and see which is the best fit for us. Microsoft certainly wants to take over the market and they’ve been successful in other areas in doing so after entering late, but it’s way too early to tell in this case, in my uneducated opinion! I do think VMware will be around for a while, and is not a poor choice from a longevity perspective yet.

Licensing is the only “kicker” with my new servers. Microsoft licenses (that link has a cool calculator, but it gives retail and not charity prices!) their Microsoft Server Datacenter software per socket (physical processor) in each server. So if you have a dual-processor system, you buy two licenses. The R805 is a dual-processor system, but the 1950s are as well, and I just doubled my processor count and therefore licensing cost! That’s OK, I’m likely going to, for now, use the Datacenter license from the PE 1800 for one processor in one of the new servers and use a Server Standard license on the PE 1800 for now that I’ll be freeing up by virtualizing to pre-licensed VMs. This will just add around $500 to the cost of getting a second server, making the hardware and licensing increases in going from one to two servers come in right at $800. That’s not bad for a second server, doubling your quad-core processing power and RAM as a consequence!

Because these servers were in the Dell Outlet, where items in your cart last only 15 minutes unless you modify them, the server have now been purchased and are estimated to ship on October 1st. I’m still waiting to pull the trigger on the MD3000i SAN and I’m still researching backups (which just had its budget cut a bit with this server swap!). I did hear from my Zones rep that October 1st begins Microsoft’s new fiscal year, and they will be announcing any pricing changes at that time. Pricing on Datacenter could go up, or stay the same (well, it could go down but how likely is that? Exactly!). Since I have the servers for sure now, I may go ahead and grab the Datacenter licensing before the end of the month for that reason if I can (my boss is out of town at the moment; he approved the server purchase earlier waiting to change airplanes at an airport in the Bahamas on a business trip. No, he called me first for some technical assistance, I didn’t bug him until I had him on the phone already. Yes, I should have asked if he needed an assistant for his business trip :-)

So, one more step completed in the process, many remaining. Also, I’m possibly going to need (or want very much to have) a cheap or free server rack that I can pick up locally. Just saying, if you happen to be throwing one away and are nearby :-)

• A Dell PowerVault MD3000i iSCSI SAN (with 10-15 near-line SAS 1TB drives and dual controllers)
• At least one new server (likely a Dell PowerEdge R805 with dual quad-core AMD processors)
• Microsoft Server Datacenter licensing for the same
• An iSCSI Ethernet switch
• Battery backups for server and SAN
• Backup software and some drives for backup
• A rack for the server room (maybe. If I can find a free or dirt cheap used one, locally)

Unfortunately, my budget for all of this as assigned is going to be a bit of a tight fit to squeeze the last few bits in and I’m still trying to figure out the best and most cost-effective way to do this. I’m also struggling a bit with which backup software to use. The server we were using to mirror our data is now dead, and although we have a RAID 5 array that is nowhere near the level of protection we need to have, but I am starting from scratch as far as which data backup software to buy. I have no tapes nor the budget for a tape drive, but I do have some PATA hard drive arrays that are only half-full that are SCSI-attached and will probably work in tandem with an existing server to become my disk-based backup server. Figuring out how to do offsite backup, within the same budget, is high on my todo list :-)

My budget for all this is actually less than the retail price of just the server and SAN. Fortunately, I rarely pay retail and this is certainly no exception! The plan, if isn’t obvious, is to virtualize. I’ve already been doing virtualization for a while to some extent. I used Microsoft Virtual Server a long time ago but switched to VMware Server product when it was released for free. I’m currently running four virtual Windows machines and a virtual Linux machine for our helpdesk software. Recently, since VMware’s ESXi became available for free, I’ve started running it on our newest server, a 3-year-old Dell PowerEdge 1800 Xeon 3.0GHz system with 6GB RAM. It runs very well and I love the management interface, even just using the VMware Virtual Infrastructure Client it comes with (since we aren’t paying for Virtual Infrastructure 3, or VI3 as it’s known!).

The goal of the new system will be to move virtual machines and file shares onto the MD3000i SAN. The Dell R805 server will be the primary virtual machine host, and although my plan was to use VMware ESXi, a good friend and mentor has recommended I examine Microsoft’s new Hyper-V virtualization platform as a strong contender, especially since Virtual Machine Manager (VMM) 2008 is coming and Live Motion will be available in the future, something that VMware does (simliarly, at least) with VMotion only for a high licensing cost. I’m open to either virtualization solution at this point, but here’s my main problem: Hyper-V will only run on procesors that support the new virtualization extensions. Right now, that’s zero of our servers. The R805 will support them of course, but the PowerEdge 1800, our only server with even a 64-bit processor, is too old to have the extensions and thus cannot run Hyper-V or 64-bit guests (although it runs ESXi just fine right now). My plan was to use the PE1800 as a backup server to run critical systems as needed if the R805 was ever down. With Hyper-V, this is no longer an option (and really, with the PE1800 and ESXi I still can’t run any 64-bit guests, so Exchange 2007 is out), and my concern is being left without a secondary server should the primary fail.

So, I’m left trying to fit a second server of some sort, new enough to have virtualization extensions but cheap enough to fit into my already packed budget. Suggestions, and of course donations, are welcome :-) I have been keeping an eye on the Dell Outlet but that is mostly pointless until I have the money to spend same-day when something shows up in stock.

At the moment I’m likely going to get a couple of refurbished UPSs from RefurbUPS for battery backup. However, I will be pursuing a contact or two who may be able to help with this as well.

The final area I’m still investigating still is data backups. There are a plethora of disk-to-disk backup options, and obviously I’m limited by price. I’m not going to use Symantec’s BackupExec for various reasons. The options I am considering so far consist of the following:

• Amanda (Enterprise for Exchange, open source if it will mix with Enterprise)
• Righeous Software’s Continuous Data Protection (CDP)
• Backup Assist
• CommVault (the Small Business version available only through resellers such as Dell)
• Microsoft Data Protection Manager (DPM)

CommVault is the solution that Jason Powell and his team use at Granger Community Church. They like it and it allows for Exchange restore down to the individual message if necessary without restoring the entire data store. The other options I have done varying levels of research on; enough to know they are still a contender on both price and features but not enough to provide an in-depth comparison summary. I am also still working how exactly how many virtual machines I will be running, and how many need to run a backup agent (some things can just be backed up with scripted backups to a file server where the data can be backed up along with everything else on that server, so I don’t necessarily have to have a backup agent for every VM).

The funding will be available most likely in the next couple of weeks or so to go ahead and make these purchases. If I can hold off on some of the backup questions, I may try to wait until after the upcoming Seacoast Fall Church IT Roundtable so I have more time to bounce ideas off of the smart guys there!

This is an overview and of course is not the only research, thinking, and questioning I’ve done about this solution! So feel free to comment and suggest away; I’m just saying there’s a possibility I’ve considered your suggestion and if so, I’ll note it. More often than not though, my thinking is challenged here in the comments, so please delight me with your insights, they are much appreciated :-)

I’m going to have to work on getting a server available to play with this at Lakeview, but it came at a great time for the Indiana District Assemblies of God office where I work one day per week: I had a server that I was about to transition into a role as a virtual server host and ESXi was released freely at just the right time to try it out! My server is a ProLiant DL580 G2 model, which isn’t on the “officially supported” HCL (Hardware Compatibility List) for ESXi, but is for the full ESX. Worth a try; I’m not planning on buying support anyway.

After registering, downloading the ISO image file, burning it to a CD, and booting the CD on the target server, the installation process completed without a hitch. It was so quick and easy, there’s no point in describing the process in detail! If you can’t make it install on supported hardware, you shouldn’t be using servers anyway :-) (You can find how-to guides online easily enough if you do need one, and I will mention that you need to change the BIOS on the DL580 G2 to indicate that the supported OS is “Linux” before installing.)

After installing, the server boots and you get a screen that you can’t actually use to do much. It tells you your IP address (if it obtained one using DHCP, which mine did) and lets you set a root password (which I recommend). Then, you need to visit the IP address of the server from a client machine on the network. This page gives you a download link to install the VMware Infrastructure Client, which you’ll need to actually setup and manage ESXi. Connect with the Client to the IP of the server using the username root (and either a blank password or the one you set earlier if you changed it–you did change it, right?).

I’m using an onboard RAID controller with SCSI storage rather than a SAN (I’ve been told ESXi does not support IDE disks but does work with SATA drives if you need it to), so I didn’t need to set up any iSCSI targets or anything like that, although that appears to be very easy if you’re fortunate enough to have a SAN.

One thing to keep in mind is that you’ll need the free VMware Converter if you want to convert Virtual Machines from a VMware Server installation to your ESXi box. It’s a rather simple process I haven’t really gone through in production yet so I won’t say more, but you can’t just copy the files over and run them.

So far I’m only running a LAMP (Linux/Apache/MySQL/Perl-or-PHP-or-Python) appliance from VirtualAppliances.net, which I absolutely love when I need a quick webserver! Installing this from the Virtual Infrastructure Client is very, very easy. File menu->Virtual Appliance->Import. I used the Import from URL function, and used this VirtualAppliances address to the .ovf file needed to install the appliance. Confirm and wait, it has to download the disks from the internet and transmit them to the ESXi server!

I’m using the LAMP server to run the HelpSpot helpdesk application. Since I had this running in a VMware Server appliance already, I simply used the “scp” command to transmit the web files from the existing appliance to the new one, and moved the MySQL database dump over as well. I logged in as root to the appliance and used “aptitude update” and then was able to use “apt-get install php5-imap” and “apt-get install php5-tidy” which are required or recommended for HelpSpot, and I used the web-based configuration to turn on the Zend engine in the Apache configuration, which HelpSpot requires. It popped up and took off like I’d never moved it! I also copied over the cron entries needed to execute the regular email checks that Helpspot does. Note that I’m leaving off a few steps involving DNS changes and firewall modifications because I gave it a new IP address, but basically the move was very easy and straightforward.

Then I ran into trouble, because I like to experiment :-) In the VMware Infrastructure Client, I right-clicked the LAMP VM and told it to “Install/Upgrade VMware Tools” on the VM (it said the Toold were out of date…wouldn’t want that now, would we?). I used the Automatic option with no Advanced Options. Everything seemed to complete successfully, and I rebooted the appliance. Oops, no network! The eth0 network interface was nowhere to be found! (Using “ifdown eth0” and then “ifup eth0” normally disables and re-enables the Ethernet interface, in case you didn’t know (I didn’t, until recently, thanks to #citrt!), but in this case only the localhost “lo” interface showed up at all.) This could be a problem, since the whole point is to be a “networK” server! I tried asking around in the #citrt Church IT Roundtable channel on IRC, where usually someone knows what to do, but I didn’t get much help on this issue from the folks currently in there when I asked. Time for Google! Without too much effort I’m pointed in the right direction, to this forum thread on the Ubuntu Forums (The VA LAMP appliance is based on Debian linux). User “modifiedmind” had the same problem as the original poster, and then found the solution and posted it later…thanks! I couldn’t quite find what to enter as the argument to the modprobe command, but I managed to track it down and this is what I had to do:

1. Edit the /etc/udev/rules.d/70-persistent-net.rules file on the appliance and delete the last two lines (the one starting “1. PCI device…” and the one beneath it, starting with “SUBSYSTEM==”net”…” (I like using the nano text editor because I’ve never spent the time to learn vi or emacs, so I did an “apt-get install nano” first; make sure to use the -w argument to nano so it doesn’t line-wrap for you, like “nano -w /filename“. Or use whatever text editor you’re comfortable with!)
2. Run this command: /etc/init.d/udev restart
3. Run this command: modprobe -r pcnet32
4. Run this command: modprobe pcnet32
5. Run this command: ifdown eth0
6. Run this command: ifup eth0

That’s it. Back to working order! So far I’m very happy with ESXi and I’m going to do everything I can to use it everywhere possible :-) It also has the capability of adding the higher-end features (HA, VMotion, VirtualCenter Manager) just like you can with ESX, if you’re willing to pay for them. I’ve never used them, this is my first ESX/ESXi experience ever, so I figure I’ll be happy without them as long as I don’t use them to see what I’m missing! The biggest thing that ESXi doesn’t have that the “full” ESX has is a “service console” that lets you control the machine locally. I’ve heard that many people have had great success running ESXi on even non-supported hardware, and it should at least run on anything that ESX will run on without a problem (just don’t try to pay for support!).

Also, the Church IT Roundtable Fall 2008 is coming up in October, but the registration price goes from $50 to $75 if you don’t register by August 8th! The actual Roundtable is October 8th and 9th, but there are pre- and post-activites planned for the day on either side if you can make it (see the schedule). My plans aren’t firm yet, but my wife and I will likely both be there!

I’m trying to get a 90-day trial of Microsoft System Center Essentials 2007 (SCE) installed. The non-profit charity pricing is under $400, so if I like it I’m hoping to buy it this Fall. Right now, I’d be happy to get it installed! After attempting to install the software (at Service Pack 1) on a virtual Server 2008 machine and failing (you have to install SQL Server Express 2005 with SP2 manually first, which I did, but it kept complaining that I needed to run the Configuration for SQL Reporting Server…which I did! The best I could, at least, but it kept complaining I hadn’t!), I finally switched to a Server 2003 virtual server. 3/4ths of the way through the install it failed saying it couldn’t contact the SQL server (that it installed) so it rolled everything back (the install and the rollback both took an hour!). I’m spending the time installing all the Windows Updates that are available for Server 2003 before trying again, which are a lot! Was trying to try it out quickly and update later, but obviously that’s not going to work! The concept of SCE is very cool but if it’s this hard to install, it better be a whole lot easier to use!

I’m probably the last person to post this, but the Fall 2008 Church IT Roundtable has an official website now! Visit (and subscribe to!) citrt2008.com for updates, details, and links to other update methods like Twitter! It’s being held at Seacoast Church in Mount Pleasant, SC on October 8-10. Be there or be…there streaming online or in the chat or something :-)

I’m getting ready to switch over to Small Business Server (SBS) 2003 Premium from a non-SBS, non-Exchange network at the Assemblies of God Indiana District Office where I work one day per week. It’s been in the works for a long time, but the official switchover is scheduled to happen June 6th and 7th (Friday and Saturday) with the 8th available if spare time is needed and some on-site support on Monday morning the 9th to work through any kinks. I’m confident in things going smoothly, but that could just be a lack of knowledge on my part (see the Dunning-Kruger effect :-) My plan is to get Postini installed in front of Exchange at the same time or shortly thereafter. I may get around to broadcasting some of the switch via webcam, but there will be some internet downtime while ISA 2004 is brought up and configured so we’ll see how that works.

In still other news, I’m going to be trialing FeedBlitz for sending out email newsletters for our Worship and Creative Ministries team in the next few weeks. I’m curious to see how that goes; I know Constant Contact is the well-known name in that space and we’re open to going with them as well. Pricing is the same for our subscriber levels, but FeedBlitz seems to have the corner on social features including publishing email from an RSS feed and now sending out messages via Twitter as well so we’re going to try it first I think.

It gets harder to blog stuff the longer I go without doing so. I seem to pressure myself to “write long, big post with a ton of juicy technical information” as the first post back from an absence. I also seem to subconciously want to post only big, important stuff to avoid wasting time with any smaller things. This contributes to not posting at all! Thus, I will attempt to be less picky about what I’m posting or the length of posts in order to keep going, while still providing some good, solid information! I may also kick up the number of shorter posts with links to other content, or republish the occasional funny comic, but will try to keep the “noise” down. Feel free to leave feedback in the comments about what you’d like to see, one way or another!

One more thing: I have Woopra set up for statistics tracking on this blog now. I don’t run the client often right now, but when I do it allows me to chat with visitors in real time! So if you’re reading this on the website and not via RSS, I can actually initate a chat with you! It’s unlikely, but keep it in mind! If you want to start a chat with me, you can do so via the Google Talk Chatback badge, currently in the sidebar menus on the site. No registration is required. If I’m not at my computer, you won’t get a response, sorry! I try to keep my availability status updated but I don’t always succeed. Try the #citrt channel on IRC or just send me an email (use the Contact Me page) if it’s that important! Blog comments are preferred if it’s a public topic or question, though!

I just found this yesterday, if you can believe I’ve found reason enough to rave already! Earlier this week I also set up the free version of the PacketTrap pt360 Tool Suite, and I’m significantly impressed. While several of the tools are part of a 30-day trial of the $1500 Pro version, the ability to easily map MAC addresses to DNS names and IP addresses is very useful, and the Dashboard, including a widget for viewing the traffic activity levels on switch ports, is nice. It didn’t blow me away like Spotlight on Windows did (and they don’t overlap too much except in some monitoring areas; pt360 is much more network oriented), but I’m keeping it handy in my arsenal for troubleshooting. It’s certainly not worth the $1500 to me for a Pro license, but someone with a much larger network might be able to justify the price tag. Although I believe I’ve run across this before, thanks to Andrew Mitry for linking to this tool in his recent blog post about free tools for IT Managers, where all the links were such high quality (the ones I did and didn’t know about) that I decided this was worth of a test run after all.

(Screenshots are from each products’ respective websites; click for larger versions.)

Something interesting to note is that while inbound traffic from the internet appeared to be blocked before the restart, I was able to use Remote Desktop from another server on the internal network to remotely instruct ISA to reboot. So it had not locked down all network access, just external. Good to know if you administer the box primarily via remote control! In fact, due to a lack of KVM switch ports, I have to manually plug the keyboard/monitor/mouse back in to ISA physically if I want to work on the console.

Although everything appeared to be functioning normally, today I got a report from a user who was getting a network error when attempting to connect to the iTunes Store from within iTunes. I tried it on my desktop, and got the same error. Fortunately, I remembered that back when I installed a prior ISA service pack (I don’t recall if it was 1 or 2), I had a similar problem and was able to track down the issue to the Compression Filter in ISA. If you go in the ISA Management Console to Configuration->Add-ins and check the Web Filters tab, by default there is a “Compression Filter” enabled (the description: “Enables HTTP/HTTPS compression”). Disabling this filter allowed iTunes Store to work just fine!

However, the reverse is true in ISA 2004 Service Pack 3. If you have disabled the Compression Filter, you must re-enable it for the iTunes Store to work in Service Pack 3! This is very useful information, so I thought I’d share! If you don’t know why iTunes Store doesn’t work, it can take a bit of Googling to determine the problem, at least it did for me originally. Perhaps the issue is more widely known by now.

So I restarted the computer, which by that time was finished running software updates and demanding as much anyway. Logged in, and didn’t do anything else other than run DiskWarrior this time. Same process, but this time after it finished the option to Replace the repaired directory structure was available! Click, wait ten-or-so minutes. Ta-da! It works! Nothing like the good ‘ole “reboot” Windows trick to get a Mac fixed!

I grabbed some nice screenshots of DiskWarrior in action (thanks to some quick Googling to determine how to use the built-in screen shot capture feature I knew existed on the Mac — Command+Shift+4, Spacebar, click on window to capture), so I thought I’d give you a sneak peek, in case you want to get your geek on vicariously without going through the near-data-loss experience yourself!

Thanks to the commenters from my last post with suggestions, they were helpful to my sanity over the weekend since I knew I had some good alternate options if DiskWarrior failed! I could find plenty of options on my own, but it’s difficult to sort out the wheat from the chaff without spending a lot of money trying various things, so the recommendations are very much appreciated.

Who needs backup?

Yep, we do! I mentioned in my last post that this server wasn’t originally intended for critical data storage(just temporary video where the RAID 5 redundancy was plenty of backup), but over time with staff turnover and other random happenings, it has sort of become much more critical. Several years ago, backing up the 1.6TB of data on this server was cost-prohibitive. Now? 1TB hard drives are $300 each, and even nice things like the Drobo are $500, which will hold up to 4 of those drives and give 2.7TB of usable data storage for right at about $1700. Or, since the array is 1.5TB, put three 1TB drives in for $1400 and get 1.8TB usable space, which is still 0.2TB ahead of our needs for a mirrored backup.

Is this the best backup solution? I know there are less expensive NAS enclosures out there, some possibly directly-networkable and not just USB 2.0 like the Drobo is currently. Right now our internal data server (RAID 5 array) is being mirrored by another server with another RAID 5 array. We have 1TB of storage and backup between those two servers, but a third backup would be nice to have. What about maxing out a Drobo and backing everything up to it? What about other options? What about picking up an iSCSI SAN from StoreVault for $3k with 1TB, or expanding that a bit and put all our main storage there (including several virtual server hard drives) , and using the existing arrays we have for backups? Lots of questions to answer, and this almost-data-loss is a useful catalyst to demonstrate the need to spend money for backups. And by “useful catalyst” I mean, I have been asked to make sure this can’t happen again :-)

I know similar things have been a hot topic of discussion on the Church IT Podcast and in other similar forums lately, but if anyone has specific information or suggestions, you can make a blogger happy with the comments form :-D Oh yeah, and sorry about that corny title. I couldn’t resist. It’s so much fun to make my wife roll her eyes when I make weird word plays like “HurRAID!” I imagine most of you rolled yours right along with her…

This is a 1.6TB array, around since our previous Evangelism and Media pastor who did quite a bit of video editing and used the array for video storage while working on projects. Fast forward to today, when the Marketing/Communications (MarCom) department is using the system to store–everything. However, we don’t have a backup solution in place because the system wasn’t intended initially as mission-critical storage and the budget hasn’t been there to change that. 1.6TB is the largest amount of storage in one server that we have right now, so there’s no place to even copy the data for a backup, like we do with our primary file server.

I purchased DiskWarrior, which comes highly recommended, but it’s throwing a funky error about a corrupt configuration file when I try to even start it. It says to delete the old configuration file to continue, but it doesn’t exist–thus, a support request has been logged and we wait until Monday to continue the saga (I tried calling their support number but they had just closed for the weekend. Probably for the best, it was dinner time, and family time, and homework time! I didn’t get out of there until 7 pm anyway). Meanwhile we are faced with the prospect that if this doesn’t work, the only option I can think of is to ring up one of those fun services that do awesome data recovery at a price on par with their awesomeness. The array itself says it’s fine, no lost drives or anything weird like you would expect for a failed system, it appears that the partition just got corrupted somehow when the power was unceremoniously yanked. To protect the guilty, I’ll spare you the details of who and why, but let’s just say my happiness level is not quite up where I like it to be…

If you have any recommendations of good (and preferably inexpensive) RAID5, Mac filesystem recovery services, I’m open to suggestions if we end up having to go that way (and can justify the cost given the data). I will update you next week on the progress with DiskWarrior when it gets up and running.

Basically, because we have ISA, I can limit what particular user groups are able to do over their VPN connection, just like any other firewall rules. Very few people get file server access at all (actually, me and one guy who connects from his church laptop) over VPN. The rest are limited to Exchange server connectivity or Remote Desktop primarily, although now that we have RPC over HTTPS in place, it’s much simpler than VPN for the user and so that’s used almost exclusively for remote Outlook access now, and is as much as most people need (if they have a laptop they have an offline copy of most of their files anyway).

For those that still require remote access to their desktop at work (especially if they don’t have a church-owned laptop), I’ve been moving from VPN with Remote Desktop access (complicated to train someone to use since the connection is separate from the RDP client) to LogMeIn.com for remote access. There’s a free version and a Pro version, with remote printing and file transfer being the main additional features of Pro. The main benefit? It’s easy and just requires a web browser, it’s fast, and not very expensive (with the special we got anyway, or the free version is of course free!). I have run into an issue with a new remote user that hasn’t gotten LogMeIn to work on their own but I haven’t had a chance to troubleshoot this yet (I’m sure it relates to the steps to get the ActiveX or Firefox plugin installed for LogMeIn initially).

We have a Terminal Services server with a handful of user licenses that we use for some volunteers that need remote access from their home computer but don’t have a dedicated desktop at work. I haven’t attempted LogMeIn through Terminal Services, but I assume it wouldn’t work properly with the multiple sessions that make Terminal Services useful, and would only allow access to the console session. For this, we still use VPN, with a CD created from the CMAK along with an auto-running tutorial created with Wink that walks users through installing the VPN connectoid (which has all of the settings preset) and starting a VPN connection. Using custom commands in the CMAK connectoid, I’ve included a Remote Desktop settings file that automatically runs upon connection, automatically opening and connecting to the Terminal Server inside the VPN once it’s connected. When Remote Desktop is closed, the connectoid logs off the VPN. The integration of VPN and Remote Desktop isn’t perfect, but it’s a lot easier this way (most of the time) than trying to get people to understand connecting to the VPN first, then connecting with Remote Desktop manually, and disconnecting in reverse. The more automated the better! These VPN connections are of course limited through ISA to be allowed to connect only to the Terminal Server, and only through the RDP protocol.

One thing’s for sure: when allowing an unmanaged computer on the network, especially as unsupervised as a remote connection is, it pays from a security standpoint to keep the leash as tight as possible! And it’s the unintentional risks (spyware, viruses, etc.) more often than malicious users that cause a problem. The best part is, protecting from one helps to protect from the other (in general).

I also updated the HelpSpot LAMP VirtualAppliance to the newest version (1.0.131 is now based on Ubuntu Server and allows you to install any Ubuntu module!) to fix the issue I had with DNS resolution, which I haven’t run into on this new version; outbound DNS works just fine now, and the PHP IMAP module installs! (As yet untested, however.) But hey, that was fun, and you’re supposed to have fun on vacation, right? At least I did it from a cottage overlooking the ocean in Maine :-) And there were minimal interruptions from the office the entire time (what there was I instigated by checking my email :-) and I returned to no emergencies or exceptionally urgent or unexpected requests. All-in-all a very good vacation! I could’ve used a little less time in the car (1300 miles in the last three days and that was just the return journey), but I’m not complaining (too loudly ;-)

Anyway, I’m back! And now you know why my posts have been sparse and not full of detailed technical info. Truthfully, I’m actually running low on detailed technical post ideas for the moment (and I do have some catch-up to take care of along with some personal stuff, which is why I’m posting this today even though I returned to work on Tuesday), but I’m sure that won’t last long. Stay tuned! I have a post coming this afternoon about a power adapter mystery/adventure that just happened, in fact…

It’s been so good, in fact, that a very long time has past since the runtime on the battery backups was tested.  Today, the test was unintentional.

Fortunately, since server room cooling has become an issue with such an enclosed space being filled with more and more machines, we are finally installing a cooling unit specifically to keep the server room cool.  A big improvement over walking into the server room and starting to sweat almost immediately, to be sure!  However, installing the cooling unit required turning off the power to the server room for a little while.  It was off for a few minutes before I headed up to our all-staff meeting this afternoon, but it was back on before I went to the meeting and the battery backups held just fine.  I knew it would need to be off for a little longer during the meeting, so I hoped the batteries would hold out.  They didn’t.  When you can’t connect to the Exchange server, or even get a new IP via DHCP over wireless, something’s up.  Or, down rather.

I still don’t know how long the battery backups lasted exactly, as everything was already back on when I made it downstairs.  Reboot everything in the right order, and half-an-hour later you’d never know anything had happened.  And with everyone in the staff meeting, I was able to warn them before the meeting broke that I would need to work on the servers for a little while and not to expect it to be operational when they went back downstairs.

It worked out all right in the end, but it’s something I need to address and haven’t had the time or resources available.  Nothing like a little priority-setting all done up nice for you :-)

Anyone have recommendations about on how you go about battery backup selection?  I took the new building opportunity when replacing the core network switches to purchase a Tripp Lite rack-mount UPS unit for each of our three network closets, which so far have worked admirably, were cheaper than comparable APC brand units, and held the network rack up even through this same power outage.

Does it make sense to buy a smaller off-the-shelf UPS for each server, or each pair of servers perhaps, or to purchase one larger unit that can handle everything, even with the sticker-shock price tag?  (Granted, several smaller units do add up themselves.)  I have a feeling I know, but I’d be interested in feedback.

We’re even protected from external SMTP exploits against our Exchange server, because we use DefenderSoft Email Threat Center (an MXLogic reseller) to accept our incoming (and outgoing, for that matter) email. Our Exchange server’s SMTP service can only accept connections from their email servers, and nowhere else, so it’s not truly open to exploit, since external servers can only get to us through them. This cuts down on spam as well (which could otherwise come through to our server, bypassing the spam filtering), which is a good side benefit.

If you don’t already keep your firewall locked down as tight as possible, keep your eye on the SANS ISC for a while. It’ll scare some sense into you :-)

This means users can connect using the standard Microsoft VPN clients, but they can only do what you allow via firewall rules. A network administrator could have full network access if needed, while users connecting to Terminal Services could be allowed to connect using only Remote Desktop (TCP port 3389), and only to the terminal server, based on their user name.

This is great! [sarcasm]It’s so great that we’ve started using LogMeIn services for some of our remote access.[/sarcasm>]Why? Well, configuring both the VPN connectoid (even with the CMAK) and Remote Desktop, not to mention actually using them, can be a bit of a chore for some users, especially if they’re setting it up at home where it’s harder to walk them through it. LogMeIn, on the other hand, has an excellent, web-based interface, very compatible remote control, and some easy options like remote printing and file transfer that are harder to set up with plain-vanilla VPN. LogMeIn has a free version, without file transfer or remote printing, which is excellent for proving basic remote support to people like family (I should know, that’s how I use it!), or connecting to your home computer remotely (yep, that I do, too). The Pro version is normally almost $13/month per remotely-controlled PC, which is a bit pricey. They have a special going on right now, however (unknown end date) where you get 5 computers for $20/month (or more at the same price, if needed) that makes it much attractive, and I’ve switched our two heaviest non-Terminal Services users over (it wouldn’t work for Terminal Services because of the multiple sessions issue), plus myself.

They also offer a service calls LogMeIn IT Reach, for the same price (and special price) as the LogMeIn Pro service, but it is targeted towards IT users managing servers remotely. The web interface to logs, shares, users, performance stats, and more is excellent! Better in some cases than the built-in Windows tools in some cases, in my opinion. And the price, at the moment, is excellent and worth every penny, if you need more than the basic features. Just a happy customer.