(Note, the service that was hosting this post, CoverItLive, has been discontinued and is offline, so this post has no content any longer. Sorry! This was a fun day and I’m sad the record is gone!)

I said back then that this blog would eventually be moved to a new location with some changes (once I had time and figured out where I wanted to put it). Well, on December 4th, 2010, I moved this blog to http://infotech.davidszpunar.com/ from http://infotech.lakeviewchurch.org./ All historical posts are intact and all of the old URLs redirect to the new ones, so for now at least it doesn’t matter how you got here. I haven’t decided for sure, but if I continue blogging, I may start a new one and leave this one as-is for posterity. Or I may continue here, you’ll just have to wait and see :-) Thank you for reading, if you have, and commenting, if you did. It’s been great!

Until then, I’m loving my new job, though it keeps me busy! And since they are coming up in a few short weeks, I’ll say it early: Merry Christmas and Happy New Year!

P.S. You can find me on Twitter or on other sites, I’m still active at some or all of them! And the IRC channel of the Church IT Roundtable where you should be too, if you work on volunteer in Church IT!

Last week, listening to my iPhone while traveling home, I heard the first verse of the song Calling All Friends by The Low Stars:

Calling all friends, and people I met on the way down.
Calling all friends, and people I don’t even know.
Calling on high, I wanna believe there’s a way now.
I’m too tired to pretend I don’t wanna be alone, I’m calling all friends.

For those working with Information Technology in churches, it’s easy to feel isolated and alone, trying to figure out what the best technology solutions are (and how to afford them!), how to best support your staff, recruit and manage volunteers, and figure out how to communicate your needs and solutions to leadership and users in ways they understand, go along with, and fund. Most churches have either a volunteer IT staff, a paid staff member who does IT as part of their job, or perhaps one full-time IT position. If you’re really large and fortunate, you may have a small team of two or more to support your environment, creating some camaraderie, but it’s still easy to feel alone, isolated and seldom understood.

Read the rest at IndyGeek.net. (NOTE on Dec. 12, 2011: IndyGeek.net is no longer operational. I am republishing the rest of the original article below, picking up from where I just left off above.)

Fortunately, Jason Powell, the IT Director at Granger Community Church (GCC) in Granger, Indiana felt that way himself several years ago, and decided to do something about it: he started blogging. The online community created by Jason’s blog led him to invite other church IT folks to GCC and have a “roundtable” discussion to see if they could benefit from sharing with each other. This was the first official Church IT Roundtable (CITRT), a term that now encompasses an unofficial group of people, discussions and community that connect from around the US and even the world so no one has to “go it alone.”

Roundtables are generally held a couple of times a year. The most recent Roundtable was held at Saddleback Church in Foothill Ranch, CA on March 10^th through 12^th. Approximately 75 people from churches around the country (and a small number of vendors) attended.

On Wednesday night, officially the optional “pre-roundtable” dinner, old friends and new ones gathered for some excellent dinner provided by the on-campus foodservices at Saddleback and some even more excellent socialization. Sharing technology and technique are excuses to have a Roundtable, and don’t get me wrong, the both were shared in abundance and the knowledge and experience is invaluable. But the real, just as tangible but less quantifiable, reason to get together is to share life with each other and forge long-lasting friendships with peers who just happen to often have resources they’re willing to share with you at and after the Roundtable. For all the technology, there’s at least a triple dose of inspiration and connection.

Why get together in person? That’s a good question, one that geeks of all stripes would probably ask in a similar situation. After all, technology and the Internet are pretty powerful now. Why not leverage blogs, social media, online chat and streaming video to accomplish everything remotely? Because that already happens, and it’s not enough! Relationships developed online can be good, and even somewhat deep, but it’s not often they are as rich, full and close as ones developed when eating, laughing, sharing and telling stories together around a table or tables. The “roundtable” events often happen around square tables, and the CITRT geeks enjoy pointing out the irony of this fact—however the national Roundtable at Saddleback actually took place around round tables! Additionally, it’s much easier to focus on sharing and developing friendships in an environment removed from daily workflow and life.

That doesn’t mean that the CITRT group foregoes the use of technology! In between Roundtable events, the group does leverage Twitter, Facebook, wikis, blogs, and IRC (Internet Relay Chat, a very old and once more widely used Internet chat protocol where the chat rooms are called “channels”) to communicate regularly, and for those who have met in person it’s that much easier to continue those friendships in between get-togethers when everyone is spread around the country. There’s a social aspect, but every day there are usually multiple technology problems and questions answered by others in the group in the IRC channel, saving those who ask countless hours of their own research, trial, error, and often even the cost of hiring a contractor or outside expert to provide advice and/or solutions.

And that is the focus of the daily Roundtable sessions in California on Thursday. In addition to a daily keynote speaker, there were two Roundtable discussion times on Thursday and one on Friday. Wednesday’s discussion started revolving around how the spiritual life of Church IT staff was affected by working in a church. Generally, a moderator stands up at the front of the room and takes topics from the group, writing them on a whiteboard. That’s how the rest of the sessions worked, but because most geeks would rather talk about technology, the spiritual discussion was a pre-picked starting topic foisted on the three rooms of Roundtable groups at the opening session (with 75 attendees, the sessions are much more manageable and more can participate if they are divided up into groups of around 25 each). After the spiritual discussion, the groups moved on to pick a set of topics ranging from email systems to storage solutions, networking to working with volunteers, and many more. The afternoon session on Thursday was divided up into four groups by type, with infrastructure in one room, management in another, helpdesk and user support in another room, and web design and support in the final, while the Thursday morning Roundtable was an open discussion of any remaining topics.

Attendees are admonished at the beginning: if the topic you have questions about isn’t covered, it’s your fault! Speak up, join the conversation, and participate so everyone can get what they need most from the group. Yes, geeks often are shy and reserved, but it’s much easier to open up with friends. Many in the group are already friends, some have met at prior Roundtable events and some were only friends online until this week, but even for those there for the first time, the pre-existing online friendships created a fast connection.

On Thursday afternoon, an unscheduled visitor stopped by, Pastor Rick Warren, founding pastor at Saddleback Church and author of the bestselling Purpose Driven Life book. For him to take an hour and a half out of his busy schedule to greet everyone individually, give a very insightful talk and stick around for individual pictures was not only unexpected and very welcome, but demonstrated a down-to-earth man with a heart for service and Christian ministry.

Friday morning was opened with a keynote from Scott Smith, CEO of Solerant, a company that was founded to provide IT services and support to churches, although they have corporate clients as well. Solerant has been a long-time supporter of the Roundtable online and in person, and Scott delivered a much-needed message from a CEO’s perspective about how communicating as a technology person to leadership needs to be carefully constructed to provide information that the leaders care about in a context of the things they care about, rather than spewing techno-speak that may very well be correct, but won’t translate into a concrete reason to provide support and resources. Scott focused on how to position projects and requests through high-level descriptions and especially by using stories and analogies that are easy to relate to outside of the geek mindset. Geeks in all fields could benefit from using his tips.

The daytime food and events were just the icing on the cake, as most attendees continued their discussions after dinner, often late into the wee hours of the morning in their hotel rooms, the hotel lobby, and for some, the pool and hot tub! This could range from group discussions to one-on-one or two-on-one teaching or assistance. The knowledge transfer happening at all levels is something most organizations probably wish they could leverage on demand.

It’s an event that’s hard to describe, as much as I’ve attempted here, and a lot of people who might benefit from the event, even if they already participate online, have wondered if it’s worth the time and expense (travel is most of the cost as the registration, including food, has always been under $100 thanks to sponsors who not only bring technology and services to display, but also in most cases participate in the discussions and truly help just like everyone else—the group encourages vendor engineers and technologists to attend and become part of the community, not just sales people!). However, without fail, first-time attendees enthusiastically said at the end that it was indescribably valuable, that they’d forged new and deep friendships, gathered excellent ideas to take home and implement, and that they couldn’t imagine not making this a part of their regular schedule whenever possible. This is my personal feeling after attending all but three Roundtable events since they started, but it was by far a widely shared opinion.

The CITRT main website is currently a wiki located at http://www.citrt.org/. The site provides links to participant blogs, Twitter lists, ways to connect to the #citrt IRC channel on the Freenode IRC network, and information and registration information for future in-person Roundtable events around the country as it becomes available (they move often or will break down into multiple regional Roundtables around the country in some cases), along with other information, and allows anyone to easily get involved. And because it’s a wiki, anyone connected to Church IT can request an account and add/update information on their own—just one more way to connect and collaborate! Every church, contrary to what it sometimes feels like, has many similar technology needs and those supporting them are not alone. And sometimes, that makes a big difference.

Also, for more technical notes, Tony Dye posted his excellent rough notes of Day 1 and Day 2, my article is a high-level overview but Tony provides a blow-by-technical-blow of the sessions he was in (and the main ones), even though it’s unedited there’s a ton of useful information there. Worth checking out, thanks for sharing Tony!

But, I’m still pretty young, and it’s time to move on to an environment that will provide some new challenges and experience in a wide variety of settings. So I’m moving to a small company with a Christian owner that provides residential and small business IT services to the Indianapolis and surrounding communities. Based in Fishers, IN, I’ll be working out of a new satellite office on the West side of Indy, not far from Lakeview in fact, and I’ll be working primarily with larger clients, including several churches in the area. What the job will look like day to day I can’t tell you precisely yet, but that will certainly be part of the excitement! And I’m still going to be involved with the Church IT Roundtable online and in person to a large extent (it’s still relevant as I’ll still be serving churches!), which I’m very excited about, as I have many close friends in the CITRT and their expertise has proved invaluable (and I have hopefully reciprocated with valuable tidbits of my own from time to time).

I’m really going to miss all of the Lakeview family on a daily basis. The staff are basically like close friends and family; it’s where I’ve spent all of my adult life in fact (and some volunteer time for years before that). God gave me peace about moving to this new position and I know He’ll provide, but I already miss everyone and I’m not gone yet!

There are still some details to be worked out about the transition, so I’m sure I’ll have some more to post later, and I certainly appreciate any prayers. It looks like I am going to the Church IT Roundtable at Saddleback Church in California on March 11-12! If you work in Church IT or you support or volunteer with Church IT in some way, you should be there! The cost should be under $100 plus travel, though final details should be coming soon.

My first day on the new job is set for February 10th, 2010 (though a few current coworkers said they hoped when I said Feb. 10th it meant 2011! Nothing like feeling wanted!).

• Deploying Windows 7 – Part 5: MDT 2010 Enhancements (from WindowsNetworking.com)
• Windows 7 Deployment Class from author of DeployVista.com (a blog with more deployment info)
• The Deployment Guys (Microsoft Technet Blog)
• MDT 2010 Webcast (a webcast from the Deployment Guys blog above)

First I downloaded MDT 2010 from Microsoft, installed it and then opened the Deployment Workbench. You’ll need the AIK (Automated Installation Kit) for Windows 7 as well for some of the steps later, which is huge (1.7 GB), so you might as well get that started downloading now, too.

What I did to learn is I found some Microsoft pages with info on MDT and some videos that showed the basics, and I watched/followed one of them, but I don’t recall exactly which video it was that I found. You don’t want to focus on the AIK, I did a while back and it’s more for OEMs like Dell making system images for presale. Similar tools; the MDT uses AIK but has the Workbench that you do most stuff from (or that I did most stuff from :-)

I did read the help in Workbench a lot, and did some Googling, plus that walkthrough video that I can’t seem to find. The documentation built-in to MDT is actually pretty good, I recommend digging in. The basic idea is you need to know what steps to go through in the Deployment Workbench. You aren’t necessarily creating an “image” for deployment as you are making the installation more automated, providing install media from a network share and also packaging some applications with silent installs together. You can optionally build a Windows 7 box, capture it with ImageX, and pull that into MDT to deploy (with or without additional applications installed during deployment) but I didn’t go that far, I’m using a stock Windows 7 Enterprise image (I imported both the 64-bit and 32-bit install discs).

Basically in MDT, you go to Deployment Shares, and create a new one. You’re basically creating a network share that will hold all the install files. You take the Windows 7 DVD for example, and Import it into the Operating Systems “subfolder” of the Deployment Share you create within MDT, and it copies the disc into a subfolder of that share for you and lets you set some properties and name the image. I haven’t done so, but there’s another folder called Out-of-Box Drivers you can import drivers into for your specific hardware.

As for application install after deployment, there were two applications I couldn’t get to install silently and thus won’t work to be installed automatically. Those two apps were iTunes and Shelby v5 (our Church Management System). Shelby doesn’t have a silent install option but it’s easy to manually install afterwards. iTunes is supposed to pass your arguments to it’s .exe installer into the .msi files inside, but it failed for me every way I tried it (always left some component uninstalled) so I gave up. You can use 7-Zip to extract the iTunes install file into it’s component .msi files and manually install them (careful of the order) if you want, which works but is “unsupported” by Apple (not that I’ve ever contacted them for support). For now, I’m not installing iTunes automatically either. I spent a few hours on iTunes so I’m pretty confident of how messed up it is :-)

In general, anything you can install silently with command line arguments will work, and anything else won’t. For Adobe Reader, I downloaded Adobe’s Customization yep Reader works fine; I actually used the Adobe Customization Wizard to make an .mst (MSI transform) and install the version with the transform so my preferences are applied and the transform automatically specifies a silent install (based on how I configured it in the wizard).

Within MDT’s Deployment Workbench, inside your Deployment Share’s Applications folder, you add applications that you want to be able to select to install during each deployment. You can create folders to organize the applications (as they display for you to select during deployment), and you can show or hide applications as you wish. You can also crate Application Bundles, which basically install a group of other applications you’ve already defined. You can use both features together to create applications but hide them (even in their own folder, like “Linked Only” or “Bundled Only” or some such), but put them all in a bundle with one name for easy selection at install time. I also created separate folders for apps that have both a 32-bit version and a 64-bit version so I can select the apppropriate one for each system as needed.

For example, I created a Mozilla Firefox application, and one each for Adobe Flash 10 ActiveX and Plugin versions (you must complete a licensing agreement just as for Adobe Reader to get the .msi versions of the ActiveX and Plugin versions of Flash for deployment like this). I hid them and put them in a subfolder, but created a “Firefox and Flash Player” app in the root that is just a Bundle that installs all three at one time, and it works great.

The Deployment Share has another “subfolder” in the tree called Task Sequences. You’ll want to create a Task Sequence for each OS (one for 32-bit and one for 64-bit in my case), giving each sequence a unique number (I just started at one, then used two for the second one, etc.). Make it a Standard Client Task Sequence (the default in the wizard), select the OS version at the next step, and optionally specifiy a product key (you can enter this during deployment or after install as well). Fill out some basic organization name info and default IE homepage, then set a Local Administrator password (optional–I left this blank here and specify it at install time in the wizard as well), and click Next one last time to create the Task Sequence.

Once your apps are defined as well as your task sequences, and your operating system install images are imported, right-click on the name of the deployment share under the Deployment Shares root in the Workbench, and choose to Update Deployment Share. This wizard will create the stuff needed to actually deploy from the share, including the LiteTouch boot images (images are also created in .wim format, and I imagine you can set it up in WDS (Windows Deployment Services) on Windows 2008 (or 2003 with updates) to use PXE booting to deploy as well if you want to get into that). I’m using the boot CD method. After the Deployment Share Update completes, use Windows Explorer to browse to the deployment share folder, and then go to the Boot subfolder. You should find a LiteTouchPE_x86.iso file and a LiteTouchPE_x64.iso file as well as the .wim versions and .xml configuration files as well. Burn the .iso files to CDs (Windows 7 support right-click-and-burn for ISOs, plenty of free options for other OSes).

Now you can boot whichever version you want on a computer, and depending on the architecture version of the CD each will only give you the OS options that are compatible on the deployment share. Basically you boot to a UI from the LiteTouch boot disc that asks for username/password/domain to access the Deployment Share. The share location is all hardcoded during the Update Deployment Share process. I don’t have it in front of me and haven’t done it since Monday, but the basic steps it goes through are: It asks for computer name, and whether you want to join the domain (if you do, it prefills the same user/pass/domain you entered earlier for share access which is handy). Then you pick which OS from the list, and on the next screen it shows you a list of apps in the folders you set up earlier (this list is pulled from the share, so if you Update the share later with app changes you don’t need to burn the disc again, in case that’s not obvious). You just check the boxes of the ones you want (like I have a 32-bit and 64-bit 7-Zip app, and I have to select which. Also, my VIPRE antivirus app has two installers depending on if I want it to be in the Laptops or Desktops group by default, so I pick the right one as well).

Then hit Finish, and come back in about an hour or so depending on the system, and it’s logged in as Local Admin with a status window showing you any errors (or not) from the app installs. I just did it for a new laptop on Monday, was very easy! I still had to install some drivers since I didn’t add them to the deployment share.

Here are the apps I got to install silently: Adobe Reader 9.2, Firefox 3.5.5, Flash Player 10 (plugin & ActiveX), CDBurnerXP 4.2.7.1801, Pidgin 2.6.4, LogMeIn Free 4.0.982, RDP Enable Script (custom batch file that enables RDP and firewall hole for it), VIPRE, 7-zip, and Office Enterprise 2007 (customized with .mst). The Deployment Workbench will actually let you create an Office 2007 customization and run the wizard and everything for you right from the app properties, which is nice, though I had my own .mst already that I used. For each app I created I selected the option to create an Application with Source Files so it would copy the whole install folder to the Deployment Folder. Also, there’s some stuff you can do that lets you automatically run the USMT on XP for example, backing up user profile to a folder on the hard drive or on the network, then have the MDT deployment run USMT again restoring state after the install, all automatically…I saw it in the video I watched but didn’t get it working (I didn’t try).

Here are the silent install commands I used for the apps I got working, for reference:

Adobe Reader 9.2
msiexec /i AcroRead.msi ALLUSERS=TRUE TRANSFORMS=AcroRead.mst /quiet
Microsoft Office Enterprise 2007
setup.exe
You can use the Office Products tab when editing the application definition to customize the app, or if you already have a .mst transform, put it in the Updates folder inside the Office installation structure and it will be automatically applied, no need to pass it in as an argument.
CDBurnerXP (the .msi, available as a separate download)
msiexec /i cdbxp_setup_4.2.7.1801.msi AI_DESKTOP_SH=0 AI_QUICKLAUNCH_SH=0 AI_STARTUP_SH=0 VIEWREADME=0 /qn
Pidgin
pidgin-2.6.4.exe /DS=0 /SMS=1 /S
LogMeIn Free (I’ll leave you to get it; the way I do it it prompts for the account to join it to after install, but it’s possible to find ways to make it auto-join to a LogMeIn.com account)
msiexec /i LogMeIn.msi /qn
Sunbelt Software VIPRE Enterprise (create MSI deployment files from the console)
MSIEXEC /I SBEAgent-ProfileNameHere.msi ALLUSERS=TRUE /quiet
7-Zip 32-bit (.msi is available if you dig on their site as a separate download, default for 32-bit is .exe)
msiexec /i 7z465.msi /qn
7-Zip 64-bit
msiexec /i 7z465-x64.msi /qn
Mozilla Firefox
Firefox Setup 3.5.5.exe -ms
Adobe Flash Player 10 for IE (ActiveX)
msiexec /i install_flash_player_10_active_x.msi /qn
Adobe Flash Player 10 for Firefox (Plugin)
msiexec /i install_flash_player_10_plugin.msi /qn
Java (get the FULL OFFLINE installer here) (thanks to Justin Moore for finding this one and commenting!)
jre-6u17-windows-i586-s.exe /s ADDLOCAL=ALL

I hope that’s helpful to someone! Or maybe me in the future :-)

Also, it’s much easier for an Exchange 2010 and Exchange 2007 box to cohabitate on a network and still allow ActiveSync and OWA access than doing the same with Exchange 2007 and Exchange 2003 (which requires a separate Exchange 2007 CAS, or Client Access Server). Granted, making it work with the ISA firewall was a little tricky, but with a little experimentation it went well and is working fully. So well in fact, that only my Mac user and my Blackberry user are on the old 2007 box now until I stuff is compatible (in the Blackberry case) and I can babysit the migration (in the Mac user’s case, with Entourage–Snow Leopard isn’t an option on our PowerPC hardware). Those will come soon enough. But frankly with Google for the help docs and processes (there’s a lot of good information directly from Microsoft out there already!), the process only required two remote nights working until 3:30am, and some time during one day to work out the ISA stuff to keep ActiveSync and OWA working.

I’m not going to elaborate on the entire installation process here. Microsoft documents it well, it requires installing Exchange 2010 on a new server (no in-place upgrades) to do the transition (that’s how I prefer it anyway, and with virtualization that’s easy!). But it was mostly smooth, similar to 2007 in many ways (different enough to require some reading but familiar enough it was much easier to pick up than 2007 was from 2003). And, as I discovered this morning, for Outlook 2003 clients to connect, you should also run this in the Exchange PowerShell console:

Set-RpcClientAccess -Server [servername] -EncryptionRequired $false

Otherwise, Outlook 2003 will stare at you (or, rather, the user) blankly and not connect (at least if you have internal encryption to Exchange disabled, which I do–I didn’t test enabling it).

Do I recommend going with 2010 now? Yes, as long as stuff you use like Blackberry and Mac supports it or you’re prepared to learn how to make it work. Also, your “now” may not be the day of General Availability depending on the size of your environment and current needs and plans :-)

Any thoughts? Do you think I should have gone with Exchange 2010 the week it was released? I think it’s a reasonably well proven product even though I didn’t participate in the testing myself like I did with Windows 7. Are you migrating soon? (Microsoft likes to call moving from one version to another of the same software a “transition.” I like the term “migration” better, but whatever. They reserve that for when you “migrate” from one of their competitors. I don’t care :-)

However, there is a flag you can set on a moderated object that will allow a moderator for a “parent” group to moderate an email once regardless if subgroups also require modification. Think a moderated all-staff list that contains a moderated group for a specific department; by default both the all-staff moderator and the department list moderator would have to approve a message to all-staff before the department recipients would receive it. If you’d rather have some groups like all-staff set so whoever moderates a message to that group auto-approves any subgroups as well (this is precisely why I wanted it, although we don’t have moderated subgroups yet), that’s why they added the flag called “BypassNestedModerationEnabled” which you can set to true with PowerShell.

The problem is, the few places that talk about that flag online call it a completely different name! Sure you can do “get-help Set-DistributionGroup -full” to see all the options (there are many) or you can find the same help online, but it’s not easy to track down if you’re looking for the wrong setting name! The correct syntax to enable this moderation bypass on a group (from within the Exchange PowerShell console) is:

Set-DistributionGroup -Identity "[group name]" -BypassNestedModerationEnabled $true

However the Exchange Team’s official blog says in it’s moderation post, in the FAQ section where it mentions nested approvals (near the end of the post), “If you set the BypassModerationEnabled flag to $true on the parent group, any messages sent to that group will bypass moderation by child groups.” Close, but it’s actually the BypassNestedModeration flag. If you do some searching, you’ll find a TechNet article called Understanding Moderated Transport which, again near the end in the Handling Multiple Moderated Recipients section, says, “To do this, you set the AutoApproveNestedDLEnabled parameter of the moderated distribution group to $true.” Which provides an even farther-off version of the same thing! At least with the correct version, you can more easily look it up in the TechNet Set-DistributionGroup topic where is is correct!

It’s likely the incorrect articles were both correct at the time they were written, during beta and release candidate cycles of Exchange 2010, with the final flag name being changed in the generally available version that came out this past Monday. I don’t know for sure as the GA version is all I’ve run, but it seems a likely explanation given that the articles are almost a month (the TechNet one) and five months (the Exchange Team blog) old. But apparently I’m the first person to write about it outside of them (that Google knows about).

I talked to my Zones sales rep, Eric Inabnit (Eric.Inabnit@zones.com, or 800-258-0882 ext. 3361), about it to see what the real deal was. He did some checking, and like James found out from his CDW rep, it appears that Adobe is consolidating their Educational and Non-Profit SKUs to simplify things, but it appears the pricing will stay relatively similar to its present levels, with a few minor adjustments. To quote Eric, he is hearing that, “they will be combining the nonprofit and academic price sheets to simplify management on their end. They are saying that if you qualified before you will still qualify, your sku’s will most likely change however pricing changes if any, will be negligible.”

Adobe will be shutting down its entire licensing system from October 7th to October 14th, however, so you cannot retrieve your license information for existing licenses nor can you order new licenses during that time. I can live with that, I wasn’t planning on any October Adobe orders.

This is good news, and while it’s by no means the final word, it does make me worry less about the potential budget impact it might have on churches! Adobe’s products are already some of the highest-priced software packages we buy that aren’t for servers (and frankly, much of our software (Microsoft, especially) costs a lot less than some single Adobe licenses), even with the reasonably significant non-profit discount.

If I discover any additional information I’ll update this post; send me any new information if you’ve got it! (Leave a comment or mention @dszp on Twitter.) Thanks, James, for bringing the Adobe changes to my attention and checking into it as well.

Or, you should be able to create a new policy with this checked, and apply it to each user’s mailbox directly (Recipient Configuration->Mailbox->right-click user, Properties->Mailbox Features->Exchange ActiveSync->Properties and then select a profile, and make sure ActiveSync is Enabled). Regardless, once Exchange is configured correctly, it appears that you need VersaMail 4.0.1 in order to connect to ActiveSync properly with Exchange 2007, and even on the Centro (where it may have already been installed), reinstalling it with this method fixed my problem. The update is supposed to be for the Centro, but I read several forum posts I found via Google that said it worked on the Treo 755p just fine (one had VersaMail 3.5.5 installed, the other had 3.5.4 installed), and it did for me. YMMV, don’t blame me for problems!

1. Open Email (VersaMail) on the phone and add a secondary account if only one exists (a dummy POP3 account is fine, just enough fake info that the account will be created, it doesn’t need to be checked but you can’t delete an ActiveSync account if it’s the only account).
2. Delete the Exchange ActiveSync account that is not working, leave the POP account in place but no need to verify/check it (it’s just a dummy account).
3. Tap the Home button to return to the phone’s Home Screen.
4. Go to http://ws.palm.com/mypalm/MyPalmGenericUser/ControllerGeneric.jsp?&action=showbonus&productName=CENTRO690P
5. Click Learn More under Palm VersaMail (Not VersaMail Personal Edition), link: here
6. On Treo device, open Web browser, type this URL into address bar: http://dl.svs.palm.com/bonus/VM40_Installer_Stan.prc (capitalization matters)
7. Hit Yes to confirm the download.
8. Hit Yes to download to Device.
9. Hit Save and Open.
10. Wait for file to download, it’s 1.21MB.
11. Hit Yes to accept the .prc file into Applications.
12. Will return to Home screen with new application icon called Install Email selected. Run it.
13. Tap the Update Now button on the screen that pops up titled “VersaMail 4.0”
14. Hit Accept to accept the license.
15. Wait for installation to complete; the phone will restart automatically.
16. Re-add the “Outlook (EAS)” Exchange account to the Email (VersaMail) application. Make sure to use “domain\user” format for the username field.

Make sure to hit Test and make sure it’s successful, then continue with the initial sync. This all assumes that you have a certificate installed on your Exchange 2007 server that functions properly with Palm OS devices; e.g. that they trust the certificate root and the certificate is not in the incorrect format and it doesn’t have SANs (Subject Alternative Names) like a UCC cert. But I covered this, and why I’m using RapidSSLOnline.com, in my last post, Palm Centro and GoDaddy SSL Certificates: Fixed! so you can read more about the server side there.

My Palm devices are all on the Sprint network, I don’t know if the same steps apply for Verizon, AT&T, or other providers, although it’s likely they would.

It worked for me! That’s why I’m writing it out here so I remember how to do it when someone else has issues, but I hope it helps others as well. I know I saw a lot of forum posts discussing Palm and ActiveSync (and I’ve run into plenty of issues myself in the past that I’ve had to deal with). Frankly, I will be very happy when PalmOS devices are dead…the Palm Pre is a good replacement, and the iPhone is an even better one. Windows Mobile I haven’t used enough to have an opinion on (it will likely stay that way), and BlackBerry I’ve only used enough to know that the pain of the last two weeks trying to solve a BlackBerry issue that mightbe solved now and might not be, isn’t worth it, but if you have to support it, the features are there if you can get them to work. But my BlackBerry and BlackBerry Professional Server woes are for another post, if I find time to write it :-)

So I make it through the renewal process, which required generating a new CSR (Certificate Signing Request) for a brand new certificate from the server since the original one had a bit length of 1024 and GoDaddy only accepts 2048 to 4096 bit lenghts (this is a new requirement). After completing the process and getting the certificate installed, I got a nice helpdesk call from a user this morning who has a Centro: “SSL certificate not accepted due to possible expiration.  Check device date & time and re-sync.”

Joy oh joy, exactly what I’d been looking for, another problem and wasted time!

OK, enough sarcasm (but really, can you ever have enough?). Time for Google and Daryl Hunter from the Church IT Roundtable! Although GoDaddy auto-renewed my SSL certificate, I was actually contemplating buying one of their UCC certificates to be ready for when we went to Exchange 2007. Fortunately I read Daryl Hunter’s post about Exchange 2007 without UCC certs, and stuck with the regular certificate for now, because per Palm KB article 43375, certificates with Subject Alternate Names (SANs), such as UCC certs, are not supported at all on Palm devices (“SSL v3 certificates which rely on the Subject Alternate Name field to do load balancing across virtual site names do not work with Palm OS devices.”). So a UCC cert isn’t even an option for me, but it’s cheaper to do Daryl’s method anyway! For now I don’t have to worry about it, since I just have Exchange 2003 for now, and that’s not the present issue (but we will likely be on Exchange 2007 or Exchange 2010 by the time the certificate expires). Additionally, the same article (which has a tool for installing new trusted root certificates on some Palm OS devices–but I didn’t want to mess with touching every single Palm OS device here! And, the tool works on Windows 2000 or XP only, not Vista (and I’m sure not Windows 7 either)) specifically states that, “GoDaddy Class 2 certificates do not work with Palm OS devices.” Time to drop GoDaddy!

Daryl’s favorite SSL certificate vendor (and now, mine too!) is RapidSSL Online. They sell certificates from RapidSSL.com for $17.95 per year (or cheaper, for multiple years), and they’re single root certificates (which menas you don’t have to install intermediate certificates on your server). While RapidSSL Online is cheap, RapidSSL.com directly has a 30 day trial certificate you can sign up for to test for a month, and this is the way I went. When that certificate expires I’ll be purchasing a multi-year certificate from RapidSSL Online, but I wanted to make sure it would work, and it does! I don’t know for sure, but it appears that RapidSSL.com is the company holding the root certificate, while RapidSSL Online is either a reseller or a sub-company of the parent selling the certificates at a discount (the RapidSSL.com certificates aren’t expenive but still cost a lot more than from RapidSSL Online!). Either way, RapidSSL Online claims that their RapidSSL certificates are issued by RapidSSL.com so they should be the same (I haven’t made a purchase yet), and Daryl Hunter has used RapidSSL Online successfully for years across multiple installations.

I generated a new CSR for a new certificate, again (just like I had to do for GoDaddy). I installed the free certificate on my Exchange server’s IIS (I also then exported it and imported the .pfx file onto my ISA 2004 firewall since it does the authentication up front for external clients, but that’s a pretty unique case and in most cases you want this done on the Exchange server). They were right, it’s just a single root on the certificate, signed by Equifax! I had my Palm Centro users (two had complained by this point) try syncing again. It worked! My iPhone also works fine still, and I haven’t had any negative reports from the four Palm Pre users here either. None of my users have Windows Mobile, and my one Blackberry user connects though Blackberry Professional Server rather than with ActiveSync.

So, adios GoDaddy SSL; fortunately they will refund all but $15 of my certificate (for processing since it was issued), and I’ll still come out ahead with RapidSSL Online (GoDaddy was $60 for two years, while RapidSSL Online is only $70 for five years!).

One thing I’ll have to be careful of when I go to Exchange 2007 is that once I use Windows Server 2008 to generate the CSR, it appears I will need to go to extra pains to make sure the CSR is in Printstring format instead of UTF-8, as Palm OS doesn’t support UTF-8 certificates either (Server 2003 uses Printstring by default). Daryl located this useful post while helping me troubleshoot: Ranting about Palm Centro Versamail ActiveSync and SBS 2008. Useful info, I’m sure I’ll be going back when it’s time to renew next time and Server 2008 is in place. By then, I hope we are Palm OS-free; although I loved my Treo 600 and Treo 650 both, the web is littered with forum and blog posts from people who have SSL issues with Palm OS devices (the Palm Pre and Pixi are much more flexible and up-to-date with the Palm WebOS). I was happy GoDaddy “just worked” in the past, frustrated that they “just didn’t work” this time, and happy to save money and move to a company that’s quicker/faster/easier!

Near the beginning of the year, I did a P2V (Physical to Virtual) move of the server onto our VMWare infrastructure from the old desktop. Our network, when the checkin system (Parent Pager Plus) was set up seven or eight years ago (before I was hired and was just an occasional volunteer!), wasn’t really reliable from one end of the building (where the server room is) to the other end where the checkin system was located, and thus the “desktop” server placed local to the checkin stations, which were at that time isolated from the rest of the network behind a Linksys cable/DSL router (for security). It worked, mostly, especially when we upgraded to new (but low-end) desktops for the actual checkin stations rather than the first systems we used that were only supposed to support Windows 2000 Professional and had countless hangs, errors, and just weird random stuff happen. The new systems practically ran themselves!

We built a large building addition, including a new lobby, and moved the checkin stations and server a couple of years ago. But none of the hardware changed (we added a few stations and got some (not all) of the stations set up with LCD touchscreen monitors over the years, too). A part of the new building included a new core network including managed HP ProCurve switches with fiber optic connections between the MDF and two IDFs (one of them brand new). The infrastructure could now reliably support moving the server into the server room and into more reliable hardware, so like I said, P2V was the solution! It worked great, except the server was also a Domain Controller for it’s own Active Directory subdomain, and some things didn’t go quite right with the P2V and Active Directory, and replication failed with my main domain controllers. I won’t go into details, but suffice it to say don’t P2V a DC, at least not without knowing what special precautions to take :-) After 60 days of not talking to my other Domain Controllers, the tombstone period was past by the time I looked at it, and I ended up needing to manually remove the entire subdomain from Active Directory, which is beyond the scope of this post. Suffice it to say, I managed to do so, and then I spun up a new virtual machine, running Server 2008, setting it up as a Domain Controller and recreating the subdomain I’d just cleaned up. Before I did this, I went to each checkin station and unjoined it from the old domain, and then re-joined them to the new domain.

Why set up a whole subdomain for checkin stations? Cleanliness and separation/security mainly. It’s not as important now with our current network but I still have the whole system on a separate subnet and VLAN (no Linksys router now :-) and pretty isolated. The clients and the virtual server are the only thing other than the firewall/router that’s on the subnet. And it’s what I did last time, and even though I basically ripped everything out, I was happy with the design decisions still, just not the implementation. So it’s still a subdomain, but with a Server 2008 DC that’s properly replicating to my other DCs.

What else changed? Well, we’re running SQL Express rather than MSDE 2000, for one. Also, Windows XP’s new Client Side Preferences addon was released, adding a ton of easy control via Group Policy! Using the new Preferences, I was able to reduce the user permissions while still allowing things like hidden drive maps to utilities, forcing custom registry entries to be maintained on login for many Parent Pager Plus settings that the checkin systems all shared (so if you log off and back on or reboot, those common settings return to their correct defaults regardless of whether they had been changed). I even customized the screen saver that says “TOUCH HERE TO START” in the Marquee so it is automatically pushed down to each client with the correct text and timeouts! Basically, the environment for each checkin station is very controlled with limited visibility, but there’s enough there to make troubleshooting easy if you know what to look for. I was also able to use the Preferences targeting options to very easily apply different registry settings in some cases to the checkin stations used at the manned desk area vs. the unmanned stations, so Parent Pager Plus defaults to the correct (but different) username at each login, for instance. The flexibility in the Preferences is absolutely amazing, and is the missing piece that I wished I’d had the last time I tried locking the systems down years ago with Group Policies when I failed. All checkin stations are not only joined to the domain but log in to a common domain username instead of local users. Although there are a lot of tweaks in Group Policy, there are only a couple of GPOs and thus policy processing time is short and the computers boot reasonably fast given their age.

I basically spent two (long) days dedicated entirely to this project, on a Monday and Tuesday one week in late July. In those two days, I managed to convert the old subdomain to a new one on a new server with a new database, restored the database from the old server’s backup, upgraded Parent Pager Plus to the newest version (forgot to mention this earlier but it needed to be upgraded so I went ahead and did it while I was working on it already), rejoined all computers to the new domain, set up group policies in excruciating detail and tested extensively. I think the efford was well worth it and the result is a system that feels current and up-to-date even though the hardware is still years old and I spent nothing but time! It feels good to complete a project quickly and successfully. If you have questions about any of the process including Group Policy Preferences, let me know. If I took the time to detail every change I made to do the lockdown, I’d spend a lot more time on this post and ever get it published, but my original intention was to document it all here. That may come later, but if you have specific questions let me know!

But how do you stop these “rogue” DHCP servers from accidentally or intentionally wreaking havoc on your network if plugged in? There are a couple of options, all of which involve managed switches and I’m going to talk in particular about HP ProCurve switches since that’s what I have (and love). I know at least some Cisco and Dell switches have similar functionality, and likely others.

You could do something as extreme as locking down every port with MAC address security so the entire port will shut down if anyone plugs an unauthorized computer in. This isn’t a bad method, if you have your network fully documented, don’t make changes often, and want the extra management overhead. I have a Church IT friend here in Indianapolis who I know does just that…awesome! I lock down some ports like this–nursery checkin stations or public internet terminals primarily. But for the rest of the network, I finally got around to implementing something I knew existed but never had time to research until now: DHCP Snooping. The cool name is just a side benefit!

If your switch(es) support DHCP Snooping, it’s pretty easy to turn on, but you need to know a little about your network first. Specifically, you need to know:

• Which switch port(s) your valid DHCP server is connected to.
• Which switch port(s) are uplinks to other managed switches.
• What VLANs do you want DHCP Snooping protection enabled on?
• Optionally, what the IP address is of your DHCP server (or at least which IP is assigned to the server in each VLAN where you want DHCP Snooping enabled).

If you choose to configure the authorized DHCP server IP address(es) list, the switches will require the DHCP reply to come from one of the authorized IPs; if you don’t configure the list then only the switch port source matters.

Let’s review briefly how DHCP works at a high level. Computer or device is connected to the network and turned on. It sends a DHCP broadcast request to the local segment and a DHCP server that receives the request allocates an available IP address and replies with that IP to the requesting client device, which they has a “lease” on the IP until the expiration time defined by the server. At varying intervals before the lease expires, the client sends a renewal request to the originating DHCP server asking if it can keep the IP longer, and the server replies that (usually) yes it can and extends the expiration. The client has no idea what DHCP servers are available initially, hence the broadcast request. If there are multiple DHCP servers on the network that see the request, they will all respond, and the client just picks the one that responds fastest and discards the rest.

Because the client accepts the first DHCP reply it receives, a cable/DSL router will often “beat” the correct DHCP server to the reply in some percentage of cases, creating a difficult to troubleshoot problem (which can be subtle if the IP, subnet, DNS and gateway addresses issued by the rogue DHCP server are similar in many ways to the legitimate settings, and more pronounced if they differ entirely). Tracking down the source of an unknown rogue DHCP server usually involves digging into the switch address tables and mapping IPs to MAC addresses to switch ports–a fun exercise if you want to learn about how switches work and have plenty of free time, but otherwise quite annoying! Even if the rogue server is sending the correct IP/subnet/DNS settings, its list of “available IPs” to hand out is maintained separately from the valid DHCP server, and thus you will end up with two devices being issued the same IP at some point, causing an IP conflict which may lead you to discover the existance of the rogue DHCP device after you’ve finished pulling your hair out :-)

So what is DHCP Snooping? It’s just the switch forcing valid DHCP replies (not requests from clients–a DHCP reply is the server issuing an IP assignment to a client who requested one) to only come from valid DHCP servers that you specify and tell the switch about. Your DHCP server is plugged into a particular port on your switch. You configure DHCP Snooping to know that that port is “trusted” for DHCP replies. If a device is plugged into an untrusted port (all ports by default), if it tries to send a DHCP reply, the switch drops it, and it never goes anywhere! If all you have is one switch and one server, this is really simple. With multiple switches, it’s still simple but you do need to make the change on any switch where you enable DHCP Snooping. You’ll need to trust the port on a secondary switch that is the uplink to the switch where the DHCP replies will be coming from.

How about a brief example. Let’s say Switch 1 is your “core” network switch, and has your DHCP server plugged into port 1. Switch 2 is a secondary switch, and port 24 of Switch 1 is uplinked to port 24 of Switch 2. All the other ports on both switches have clients or other servers plugged into them, and let’s say you’re only using one flat VLAN (call it VLAN 1).

On switch 1, you need to tell it that port 1 is trusted so the DHCP server can send its replies on that port. You can optionally tell it port 24 is trusted since your other switch is connected to that port (and it’s a rogue DHCP server we hope!), but since it will only be sending DHCP replies out too the other switch and not have them coming back “in” the port, it’s not required. Switch 2 requires that you make port 24 trusted, since it will be receiving DHCP replies for its clients incoming on that port from the server that’s connected to Switch 1. In more complex networks, there may be reasons to have DHCP traverse both directions of an uplink port, and since switches are generally trusted to not randomly sprout internal DHCP servers, it’s probably easier to just make all uplink ports, regardless of direction, trusted for DHCP Snooping purposes. However, this only applies to uplinks to switches that support DHCP Snooping and have it turned on–don’t trust a port that has an unmanaged switch connected or one without DHCP snooping enabled, or any client on that switch can then send DHCP replies to any other client on the managed switch, defeating your entire protection! So only mark ports trusted if they are connected to a DHCP server or if they are connected to another switch which also has DHCP Snooping enabled.

Optionally, tell your switches the valid IP address(es) of the DHCP server so they can drop replies from invalid IP addresses, even on trusted ports. I did this only on my “core” switch rather than every one of my managed switches that supports snooping, just for ease of management.

Sounds great! How do you do it? Well I could post the mechanics but it’s been describe elsewhere very simply. You have to use the command line on ProCurve switches, not the command line menu or the web interface. Get to the command line, type “config” to enter configuration mode, and then follow the directions here (the article is good if you ignore the misspelled word “rogue” in the title):

Preventing Rogue DHCP Servers with HP Procurve Switches

Don’t forget to turn off option 82 per that article…I didn’t try leaving it on but it works for me with it turned off. You may need to check this out in more detail if you’re doing any sort of multi-VLAN setup with routing where you use DHCP Relay to get other subnets to the DHCP server, I haven’t tested that. And I’d run the first command last (just plain “dhcp-snooping”), it turns the filtering on but if you set the options first (and you did it correctly) you won’t prevent any good traffic by configuring first, then enabling!

Another excellent resource is from HP themselves, a four-page PDF titled, “How to configure DHCP Snooping on ProCurve switches.” Definitely read and understand both the above blog post and this PDF document before setting this up! And make sure to test during scheduled maintenance/downtime…if you get it wrong, your network will probably stop working :-) Also, there are additional options on some ProCurve switches called arp-protect that use the dhcp-snooping database to verify arp packets and prevent arp spoofing attacks. However, this is even easier to screw up and block even good stuff–I don’t recommend you play with it unless you really know what you’re doing :-)

Some of my ProCurve switches, namely the 2810-24G units and all of the 1800 series, don’t support DHCP Snooping. You can check the manual, or at the command line (except on the 1800 series which has no command line) type dhcp-snooping followed by a space and question mark (“dhcp-snooping ?”) to see if it provides you with help about the command. If the command doesn’t exist, your switch model doesn’t support it. It’s working for me on the 5304xl and the 2650s, but not the 2524 or the 2810-24G (the last one surprises me, the 2524 doesn’t). Switches that don’t support it are still going to be vulerable to rogue DHCP servers, but the damage will be limited to that segment at least and not your whole network!

Any comments? Are you using DHCP Snooping? Have you run into a situation where you wish you’d had it turned on but didn’t? (I have, fortunately few and far between. But at least once it was one of my servers that I had accidentally configured impoperly! “Rogue DHCP server” doesn’t mean malicious or even end-user created, it can just as easily be “the server admin messed up” :-)

Oh yeah, one more tip! On ProCurve switches at least, once you have DHCP Snooping set up, you can get a few details and stats about the assignments. Try these three commands to get a configuration report, view statistics, and view the current bindings databaes:

show dhcp-snooping

snow dhcp-snooping stats

show dhcp-snooping binding

Tada! The End (you thought I’d never get here, didn’t you? :-)

UPDATE: Forgot to provide a link to this article for further reading about ARP Spoofing protection that I briefly mentioned. Good description and flowchart, but there are side-effects you may not realize at first so like I said, be careful with arp-protect :-) Exploiting arp is usually intentional, not accidental like rogue DHCP often is, so hopefully it will be less of a problem especially in churches!

UPDATE 2: Derek Schwab reminded me that the ProCurve switches that support DHCP Snooping are layer 3 switches, while the ones that don’t are layer 2 (and thus don’t function at the IP level where DHCP does). Thanks Derek, you’re right and I didn’t make the connection myself–that’s what friends are for!

Here’s what’s changed in two years: Meraki has since redefined their entire business and offers much more expensive solutions, and no Meraki Minis. Also, the campground can now get DSL and not just satellite internet, which is awesome. And although Meraki is for my purposes defunct, Open-Mesh has taken over where Meraki left off and has a similar device at the same price, with better accessories and more power!

This time, we’re covering more ground as well. So I just ordered nine Open-Mesh OM1P Professional Mini Routers. And six 7 dbi antennas, plus three Indoor Wallplug Enclosures. This time I’m going to be covering more area, and I’m hoping that using some larger antennas as well as the reports I’ve heard that the Open-Mesh devices have better range than the Meraki units out of the box mean that we’ll have a very successful network this time! We’ll also have two or three DSL lines to serve as injection gateways, which should be a major improvement over the horrendous satellite connection we had before (if you could call it a connection half of the time when it wasn’t, you know, connected :-)

I plan on taking some pictures and documenting the setup more than last time, and if I find the time I might even blog some of it!

Did I mention my whole order including shipping was under $550? That’s cool.

The best tools I’ve found for dealing with the financial and invoicing sides of side work (pun fully intended :-) are these:

FreshBooks for invoicing. Free for up to 3 clients, a bit high but reasonable for more, and integrates with way more third-party services than their competitors (although some competitors are a bit cheaper and do things a little differently). They’re on  Twitter at @FreshBooks if you want to follow them or keep in touch.

IAC-EZ for accounting. If you’re making more than a few bucks, FreshBooks will help you invoice clients and track time, but expenses, taxes, and other actual accounting stuff you should track can be taken care of in IAC-EZ. I’ve gotten to know the owner of IAC-EZ over email and Twitter and was involved in beta-testing the product, and it’s not only in active development with new features and fixes, but the owner and others know their stuff in the acccounting and business worlds and aren’t shy about helping if you ask. There’s a trial and then it’s $20/mo. If you do a little side work it probably won’t be affordable but if you have enough you need to track your finances better, you can (and probably should) spare the cash. Oh yeah, and it integrates with FreshBooks, so whatever you do in FreshBooks is pulled right into IAC-EZ, avoiding entering things twice! This alone is reason enough to pick this over something else if you use FreshBooks. They are on Twitter at @IACEZ and so is the owner as @IAC_Heather.

Minibooks from Groovysquared for FreshBooks from your iPhone. I helped beta-test this app as well, and if you have an iPhone and use FreshBooks, it’s worth checking out. It’s very polished and although it’s missing some features like invoicing by time and sending estimates, those are being worked on and the feature set that is there is much more complete compared to the official FreshBooks iPhone app, which just tracks time. Minibooks does time tracking in an extremely elegant way (better than anything I’ve seen!), lets you browse and edit your clients, create, view, and edit invoices, and mark invoices as paid. It’s kind of like FreshBooks-in-a-phone. I’m missing the invoice-from-time-tracking and estimates features personally, but that’s not hard to do from my computer for now.

I use all of these tools personally and really like them. I know friends using FreshBooks and IAC-EZ who like them as well. Disclosure: the FreshBooks and IAC-EZ links are referral links and I’ll get credit and a small referral fee if you use them (you can find them easily online, guessing the domain names even, if you don’t want to use the referral links. But remember, I’m a Church IT guy :-) I received a free copy of Minibooks for my beta testing efforts but otherwise am uncompensated by Groovysquared. None of the companies saw or directly influenced this post before I published it. Lakeview Church does not endorse these products or benefit from their use in any way (other than their IT guy being more financially able to stick around at the job he loves :-)

The problem that some people didn’t realize at first when they posted this update method, was that the .ipcc file also changes an option that disables Visual Voicemail and Missed Calls, at least in some cases. I don’t recall which of the many sites it was where I read this, but the solution is quite simple for anyone out there who might need it. From the Home screen, navigate to Settings->General->Network->Cellular Data Network. The section called Visual Voicemail has a field called “APN” and if it’s set to “wap.cingular” and you use the US iPhone carrier, that appears to be what’s causing problems. The “APN” field should read: “acds.voicemail” (without the quotes).

Note that this is for informational and troubleshooting purposes only and changing any of the settings or files here is not something I endorse. Tethering is a service that you may or may not be allowed to do and is between you and your service provider; please refer to their agreement terms for details and specifics. I didn’t come up with the solution above, I just happened to read it somewhere and I’ve been contacted by a couple of people who mentioned they had problems that were fixed when I pointed out the above change that I read about somewhere.

These are some text messaging services that may be useful:

• http://texthub.com/ (another church where I know the Church IT guy, I forget which one, is using this)
• http://www.jarbyco.com/ (Granger Community Church has used this, as have a few others)
• http://www.textmarks.com/ (Granger is using this some too and we are planning to use it at Lakeview at least among staff if not generally)
• http://www.polleverywhere.com/ (this lets you show poll results on the screen that people text in, but I donâ€t think it does mass texting—Jarbyco can do this too I think, at least where people sendin questions via text message and you can have someone put them up on the screen)

Iâ€d start there and see where it gets you! Should be cheaper than phone tree most likely, def. cheaper than mailing postcards and youâ€ll probably get more teen response than from either of those anyway :-) [remember this was an email to the Youth dept.] I would start testing with a small group if youâ€re going to test multiple services, so you arenâ€t switching everyone from one system to another just to test…like set up 4-5 students with one service and see how you like the service and price before putting everyone in.

I later ran into another service worthy of comparison: http://www.churchtextingmanager.com/. I’m sure there are others out there. Other than light testing of Poll Everywhere and TextMarks, I’ve not used any of these services personally or professionally and can’t vouch for any of them, but it’s probably a good list to start your own research!

Today there are two final tests (each venue can pick one) where the stream is run for a couple of hours to the venues to make sure things are going smoothly. Last week there were some various hiccups that they found and fixed and this past Monday the test went very smoothly. We ran the afternoon test (the other is tonight) today in our Youth Center where we’re hosting the event and just after the official test, I decided to test our bandwidth with Comcast. I kept adding streams until I was streaming the 2.5Mbps (highest available) stream seven different times! Bandwidth peaked at over 18.5 Mbps downstream with all those streams running at the same time! And I think we had some bandwidth to spare (this is on our Comcast Business internet connection). Our connection is rated for 16 Mbps down and 2 Mbps up, while I’ve seen speed tests recently as high as 30 Mbps down and 4.5 Mbps up. Certainly the almost-19Mbps speed seen here is excellent and above our rating!

I’ve posted a bandwidth graph showing our internet connection’s utilization (also on TwitPic):

Cacti Graph - Town Hall For Hope Test 7x 2.5 Mbps Stream

Comcast Fun

Of course we almost missed the test this morning because someone cut our main Comcast tap this morning just before it went under the parking lot to our building. You can see the actual cut cable (and a part of my shoe) in the picture I uploaded to TwitPic earlier. This caused a four-hour internet outage (8am to noon) that I managed to get back up once I realized (thanks to some prompting from our awesome Facilities Director Mike Moore) that the other end of our building has a completely separate cable tap from Comcast for the TVs on that end of the building! That tap was unharmed so I moved the modem to that IDF and plugged into the tap. I adjusted some VLAN configuration settings to put the firewall’s WAN port on a private VLAN with the modem’s LAN interface (it was plugged in directly before) and tada, at 11:58 am (two minutes before we were scheduled to test the Town Hall For Hope stream) the internet came back!

Comcast did come out later (during the Town Hall For Hope test in fact) and repair the cable that was cut. I’ll be moving the modem back after hours; the TVs are working so I’m going to assume the modem will be fine back on its original line as well. I’m really glad we had that second tap though, because we would have had to push the Town Hall For Hope test off until tonight when the youth group uses the room we’re using, and we wouldn’t have gotten as good of a test. And kudos to Comcast for their fast response to our issues, even though they weren’t the cause.

So, today didn’t quite go as planned, but given the issues I think we had plenty of successes. And I’m not going to worry about blocking free wifi or other bandwidth use during the Town Hall event tomorrow night; since we’re only doing one stream I think we can handle it! In fact, I just realized that if we overflow that Youth Center venue for some reason (which I doubt we will not because it’s not going to be a big event, but because there are so many other churches also hosting it), there’s no reason we can’t handle adding a feed to our main sanctuary as well if necessary. I like being prepared. Just keep the backhoe’s away from the property!

I recently purchased Veeam Backup 3.0 to back up my three VMware ESXi Free hosts. Veeam Backup is awesome and their version 3.0 is the first version to support the free ESXi version! I love the deduplication and compression and the ease of use when making backups! On March 31st, VMware released ESXi 3.5 Update 4, which added drivers for some very nice NetXtreme quad-port Gigabit Ethernet cards, which I have in two of my three VM host servers but have been unable to use until they released an updated version with built-in drivers for that hardware.

So I upgraded yesterday when Update 4 was released (I actually just did a point release update to new Update 3 firmware the night before…doh!). The new NICs work great and now I have redundant paths to the SAN! (In one case I now have more than one NIC in the whole box that was doing SAN and LAN just on VLANs, so it’s quite nice to have multiple NICs available now!)

I was going through and upgrading VMware Tools on all of my virtual machines (the new release adds some driver support for enhanced NICs to Server 2003 and a few other minor things). One of my Linux CactiEZ VMs was being a bit picky with the yum package I was trying to install so after some troubleshooting I figured I’d restore a virtual machine from Veeam Backup (granted not 3.0.1 which I believe is out, I have the original 3.0 release installed right now) to get an earlier state and see if it helped to start fresh (my other thought was there was a repository issue but my older CactiEZ 0.4 yum was working just fine, it was my CactiEZ 0.6 box I recently set up that was having issues (it runs CentOS 4.7)).

But my restore fails, with an error relating to not being able to create the directory on the ESXi host to restore the virtual machine. The exact error is along the lines of:

Failure to restore item “VM Name Here” Cannot make directory ‘[datastore] VM Name Here’ on ‘ha-datacenter’. Soap fault. fault.RestrictedVersion.summaryDetail: ‘<RestrictedVersionFault xmlns=”urn:intervalvim25″ xsi:type=”RestrictedVersion”></RestrictedVersionFault>’, endpoint: “

The simpler error is in the status dialog box, “Restore error: Restore VM failed: Cannot make dir…”

I dig a little deeper and notice that the last couple of backup jobs scheduled to run overnight for some virtual machines have all failed completely. Nothing updated, and when I force a backup to start now it fails quickly for all VMs with an error along these lines:

Releasing VM files

CreateSnapshot failed, vmRef 224, timeout 1800000, snName “VEEAM BACKUP TEMPORARY SNAPSHOT”, snDescription “Please do not delete this snapshot. It is being used by Veeam Backup.”, memory False, quiesce True

fault.RestrictedVersion.summary

So, I’ve submitted a support ticket. Fortunately, right now I have nothing urgent that needs to be restored (CactiEZ is more of a plaything right now, at least my new 0.6 install), although obviously not keeping backups up to date is not a good thing.

I guess I’ve been running ESXi without Veeam (Veeam’s only been running for…maybe a month?) for long enough that I wasn’t considering backups when I did my ESXi upgrades, so I’ll admit first-day upgrading is jumping the gun. But Veeam is a VMware partner as far as I know, and I don’t know why they haven’t been able to work with VMware around this release to verify that their software works…it’s not like this is ESXi 4, it’s just an Update release of 3.5. At least an announcement of the incompatibility with a warning about upgrading sent to customers would have been nice, although it’s not something that was promised or anything.

I’ll keep this updated (here or in the comments) as the “story” progresses! Tomorrow I will also look into making sure I’m on the very latest point release of Veeam Backup to see if that makes a difference…just don’t have the energy left tonight to do anything else, I was up until 5:30 am last night doing a P2V of our nursery checkin system (long but successful!).