Did you upgrade to Exchange 2007 but are having issues with Palm OS devices like the Palm Treo 755p and the Palm Centro? I did this past weekend, and I have one Centro that worked fine, but three other PalmOS devices (two Treo 755p units and a Centro) had issues. They would connect, say Receiving, and eventually error out saying they couldn’t establish a connection with the server. Microsoft provides a very useful site at https://www.testexchangeconnectivity.com/ that will let you test ActiveSync (I tested without Autoconfiguration since PalmOS is not capable of autoconfiguration), and after testing, my users passed and could connect. But their phones wouldn’t! I made sure to set the Default Exchange ActiveSync Mailbox Policy in Exchange 2007 (under Organization Configuration->Client Access) so the checkbox to “Allow non-provisionable devices” is checked. (I should note that an iPhone and four Palm Pre devices are using ActiveSync successfully on the same server, so I know it’s configured correctly on teh server-side.)

Or, you should be able to create a new policy with this checked, and apply it to each user’s mailbox directly (Recipient Configuration->Mailbox->right-click user, Properties->Mailbox Features->Exchange ActiveSync->Properties and then select a profile, and make sure ActiveSync is Enabled). Regardless, once Exchange is configured correctly, it appears that you need VersaMail 4.0.1 in order to connect to ActiveSync properly with Exchange 2007, and even on the Centro (where it may have already been installed), reinstalling it with this method fixed my problem. The update is supposed to be for the Centro, but I read several forum posts I found via Google that said it worked on the Treo 755p just fine (one had VersaMail 3.5.5 installed, the other had 3.5.4 installed), and it did for me. YMMV, don’t blame me for problems!

1. Open Email (VersaMail) on the phone and add a secondary account if only one exists (a dummy POP3 account is fine, just enough fake info that the account will be created, it doesn’t need to be checked but you can’t delete an ActiveSync account if it’s the only account).
2. Delete the Exchange ActiveSync account that is not working, leave the POP account in place but no need to verify/check it (it’s just a dummy account).
3. Tap the Home button to return to the phone’s Home Screen.
4. Go to http://ws.palm.com/mypalm/MyPalmGenericUser/ControllerGeneric.jsp?&action=showbonus&productName=CENTRO690P
5. Click Learn More under Palm VersaMail (Not VersaMail Personal Edition), link: here
6. On Treo device, open Web browser, type this URL into address bar: http://dl.svs.palm.com/bonus/VM40_Installer_Stan.prc (capitalization matters)
7. Hit Yes to confirm the download.
8. Hit Yes to download to Device.
9. Hit Save and Open.
10. Wait for file to download, it’s 1.21MB.
11. Hit Yes to accept the .prc file into Applications.
12. Will return to Home screen with new application icon called Install Email selected. Run it.
13. Tap the Update Now button on the screen that pops up titled “VersaMail 4.0”
14. Hit Accept to accept the license.
15. Wait for installation to complete; the phone will restart automatically.
16. Re-add the “Outlook (EAS)” Exchange account to the Email (VersaMail) application. Make sure to use “domain\user” format for the username field.

Make sure to hit Test and make sure it’s successful, then continue with the initial sync. This all assumes that you have a certificate installed on your Exchange 2007 server that functions properly with Palm OS devices; e.g. that they trust the certificate root and the certificate is not in the incorrect format and it doesn’t have SANs (Subject Alternative Names) like a UCC cert. But I covered this, and why I’m using RapidSSLOnline.com, in my last post, Palm Centro and GoDaddy SSL Certificates: Fixed! so you can read more about the server side there.

My Palm devices are all on the Sprint network, I don’t know if the same steps apply for Verizon, AT&T, or other providers, although it’s likely they would.

It worked for me! That’s why I’m writing it out here so I remember how to do it when someone else has issues, but I hope it helps others as well. I know I saw a lot of forum posts discussing Palm and ActiveSync (and I’ve run into plenty of issues myself in the past that I’ve had to deal with). Frankly, I will be very happy when PalmOS devices are dead…the Palm Pre is a good replacement, and the iPhone is an even better one. Windows Mobile I haven’t used enough to have an opinion on (it will likely stay that way), and BlackBerry I’ve only used enough to know that the pain of the last two weeks trying to solve a BlackBerry issue that mightbe solved now and might not be, isn’t worth it, but if you have to support it, the features are there if you can get them to work. But my BlackBerry and BlackBerry Professional Server woes are for another post, if I find time to write it :-)

We have many Palm phones running Palm OS, in particular we have a lot of Palm Centros although we have some other models as well (but they all run Palm OS, not Windows Mobile). We’ve had GoDaddy SSL certificates for a while for our Exchange 2003 server. Until now, I’ve never had an issue with GoDaddy certificates where the phone would reject them, but yesterday I renewed the two-year SSL certificate we had (since it expires October 3rd and I don’t want to let it run out–again :-)

So I make it through the renewal process, which required generating a new CSR (Certificate Signing Request) for a brand new certificate from the server since the original one had a bit length of 1024 and GoDaddy only accepts 2048 to 4096 bit lenghts (this is a new requirement). After completing the process and getting the certificate installed, I got a nice helpdesk call from a user this morning who has a Centro: “SSL certificate not accepted due to possible expiration.  Check device date & time and re-sync.”

Joy oh joy, exactly what I’d been looking for, another problem and wasted time!

OK, enough sarcasm (but really, can you ever have enough?). Time for Google and Daryl Hunter from the Church IT Roundtable! Although GoDaddy auto-renewed my SSL certificate, I was actually contemplating buying one of their UCC certificates to be ready for when we went to Exchange 2007. Fortunately I read Daryl Hunter’s post about Exchange 2007 without UCC certs, and stuck with the regular certificate for now, because per Palm KB article 43375, certificates with Subject Alternate Names (SANs), such as UCC certs, are not supported at all on Palm devices (“SSL v3 certificates which rely on the Subject Alternate Name field to do load balancing across virtual site names do not work with Palm OS devices.”). So a UCC cert isn’t even an option for me, but it’s cheaper to do Daryl’s method anyway! For now I don’t have to worry about it, since I just have Exchange 2003 for now, and that’s not the present issue (but we will likely be on Exchange 2007 or Exchange 2010 by the time the certificate expires). Additionally, the same article (which has a tool for installing new trusted root certificates on some Palm OS devices–but I didn’t want to mess with touching every single Palm OS device here! And, the tool works on Windows 2000 or XP only, not Vista (and I’m sure not Windows 7 either)) specifically states that, “GoDaddy Class 2 certificates do not work with Palm OS devices.” Time to drop GoDaddy!

Daryl’s favorite SSL certificate vendor (and now, mine too!) is RapidSSL Online. They sell certificates from RapidSSL.com for $17.95 per year (or cheaper, for multiple years), and they’re single root certificates (which menas you don’t have to install intermediate certificates on your server). While RapidSSL Online is cheap, RapidSSL.com directly has a 30 day trial certificate you can sign up for to test for a month, and this is the way I went. When that certificate expires I’ll be purchasing a multi-year certificate from RapidSSL Online, but I wanted to make sure it would work, and it does! I don’t know for sure, but it appears that RapidSSL.com is the company holding the root certificate, while RapidSSL Online is either a reseller or a sub-company of the parent selling the certificates at a discount (the RapidSSL.com certificates aren’t expenive but still cost a lot more than from RapidSSL Online!). Either way, RapidSSL Online claims that their RapidSSL certificates are issued by RapidSSL.com so they should be the same (I haven’t made a purchase yet), and Daryl Hunter has used RapidSSL Online successfully for years across multiple installations.

I generated a new CSR for a new certificate, again (just like I had to do for GoDaddy). I installed the free certificate on my Exchange server’s IIS (I also then exported it and imported the .pfx file onto my ISA 2004 firewall since it does the authentication up front for external clients, but that’s a pretty unique case and in most cases you want this done on the Exchange server). They were right, it’s just a single root on the certificate, signed by Equifax! I had my Palm Centro users (two had complained by this point) try syncing again. It worked! My iPhone also works fine still, and I haven’t had any negative reports from the four Palm Pre users here either. None of my users have Windows Mobile, and my one Blackberry user connects though Blackberry Professional Server rather than with ActiveSync.

So, adios GoDaddy SSL; fortunately they will refund all but $15 of my certificate (for processing since it was issued), and I’ll still come out ahead with RapidSSL Online (GoDaddy was $60 for two years, while RapidSSL Online is only $70 for five years!).

One thing I’ll have to be careful of when I go to Exchange 2007 is that once I use Windows Server 2008 to generate the CSR, it appears I will need to go to extra pains to make sure the CSR is in Printstring format instead of UTF-8, as Palm OS doesn’t support UTF-8 certificates either (Server 2003 uses Printstring by default). Daryl located this useful post while helping me troubleshoot: Ranting about Palm Centro Versamail ActiveSync and SBS 2008. Useful info, I’m sure I’ll be going back when it’s time to renew next time and Server 2008 is in place. By then, I hope we are Palm OS-free; although I loved my Treo 600 and Treo 650 both, the web is littered with forum and blog posts from people who have SSL issues with Palm OS devices (the Palm Pre and Pixi are much more flexible and up-to-date with the Palm WebOS). I was happy GoDaddy “just worked” in the past, frustrated that they “just didn’t work” this time, and happy to save money and move to a company that’s quicker/faster/easier!

Back in July (I’ve been meaning to write about it since!) I did some maintenance and upgrades on our nursery checkin system. Originally installed on a domain using Windows Server 2000 years ago (although it’s been running on Server 2003 for years), the “server “was and old Dell desktop workstation without even room in the chassis for a second hard drive to run a software RAID mirror. I never got an acceptable configuration through using a domain user and group policy to lock down the system while allowing enough rights to troubleshoot the seven checkin stations (all running Windows XP Pro), and in fact they were all using Local Administer local users, not domain users at all! The system worked, but there were other reasons for some changes.

Near the beginning of the year, I did a P2V (Physical to Virtual) move of the server onto our VMWare infrastructure from the old desktop. Our network, when the checkin system (Parent Pager Plus) was set up seven or eight years ago (before I was hired and was just an occasional volunteer!), wasn’t really reliable from one end of the building (where the server room is) to the other end where the checkin system was located, and thus the “desktop” server placed local to the checkin stations, which were at that time isolated from the rest of the network behind a Linksys cable/DSL router (for security). It worked, mostly, especially when we upgraded to new (but low-end) desktops for the actual checkin stations rather than the first systems we used that were only supposed to support Windows 2000 Professional and had countless hangs, errors, and just weird random stuff happen. The new systems practically ran themselves!

We built a large building addition, including a new lobby, and moved the checkin stations and server a couple of years ago. But none of the hardware changed (we added a few stations and got some (not all) of the stations set up with LCD touchscreen monitors over the years, too). A part of the new building included a new core network including managed HP ProCurve switches with fiber optic connections between the MDF and two IDFs (one of them brand new). The infrastructure could now reliably support moving the server into the server room and into more reliable hardware, so like I said, P2V was the solution! It worked great, except the server was also a Domain Controller for it’s own Active Directory subdomain, and some things didn’t go quite right with the P2V and Active Directory, and replication failed with my main domain controllers. I won’t go into details, but suffice it to say don’t P2V a DC, at least not without knowing what special precautions to take :-) After 60 days of not talking to my other Domain Controllers, the tombstone period was past by the time I looked at it, and I ended up needing to manually remove the entire subdomain from Active Directory, which is beyond the scope of this post. Suffice it to say, I managed to do so, and then I spun up a new virtual machine, running Server 2008, setting it up as a Domain Controller and recreating the subdomain I’d just cleaned up. Before I did this, I went to each checkin station and unjoined it from the old domain, and then re-joined them to the new domain.

Why set up a whole subdomain for checkin stations? Cleanliness and separation/security mainly. It’s not as important now with our current network but I still have the whole system on a separate subnet and VLAN (no Linksys router now :-) and pretty isolated. The clients and the virtual server are the only thing other than the firewall/router that’s on the subnet. And it’s what I did last time, and even though I basically ripped everything out, I was happy with the design decisions still, just not the implementation. So it’s still a subdomain, but with a Server 2008 DC that’s properly replicating to my other DCs.

What else changed? Well, we’re running SQL Express rather than MSDE 2000, for one. Also, Windows XP’s new Client Side Preferences addon was released, adding a ton of easy control via Group Policy! Using the new Preferences, I was able to reduce the user permissions while still allowing things like hidden drive maps to utilities, forcing custom registry entries to be maintained on login for many Parent Pager Plus settings that the checkin systems all shared (so if you log off and back on or reboot, those common settings return to their correct defaults regardless of whether they had been changed). I even customized the screen saver that says “TOUCH HERE TO START” in the Marquee so it is automatically pushed down to each client with the correct text and timeouts! Basically, the environment for each checkin station is very controlled with limited visibility, but there’s enough there to make troubleshooting easy if you know what to look for. I was also able to use the Preferences targeting options to very easily apply different registry settings in some cases to the checkin stations used at the manned desk area vs. the unmanned stations, so Parent Pager Plus defaults to the correct (but different) username at each login, for instance. The flexibility in the Preferences is absolutely amazing, and is the missing piece that I wished I’d had the last time I tried locking the systems down years ago with Group Policies when I failed. All checkin stations are not only joined to the domain but log in to a common domain username instead of local users. Although there are a lot of tweaks in Group Policy, there are only a couple of GPOs and thus policy processing time is short and the computers boot reasonably fast given their age.

I basically spent two (long) days dedicated entirely to this project, on a Monday and Tuesday one week in late July. In those two days, I managed to convert the old subdomain to a new one on a new server with a new database, restored the database from the old server’s backup, upgraded Parent Pager Plus to the newest version (forgot to mention this earlier but it needed to be upgraded so I went ahead and did it while I was working on it already), rejoined all computers to the new domain, set up group policies in excruciating detail and tested extensively. I think the efford was well worth it and the result is a system that feels current and up-to-date even though the hardware is still years old and I spent nothing but time! It feels good to complete a project quickly and successfully. If you have questions about any of the process including Group Policy Preferences, let me know. If I took the time to detail every change I made to do the lockdown, I’d spend a lot more time on this post and ever get it published, but my original intention was to document it all here. That may come later, but if you have specific questions let me know!

Has anyone ever plugged a Cable/DSL router into your network without authorization? Those things have DHCP servers on by default, you probably know that. And you want DHCP at home, and for that matter at work, too. But only one per network, or things get nasty really fast! (There are some legitimate redundant DHCP configuration options but never involving a consumer appliance!)

But how do you stop these “rogue” DHCP servers from accidentally or intentionally wreaking havoc on your network if plugged in? There are a couple of options, all of which involve managed switches and I’m going to talk in particular about HP ProCurve switches since that’s what I have (and love). I know at least some Cisco and Dell switches have similar functionality, and likely others.

You could do something as extreme as locking down every port with MAC address security so the entire port will shut down if anyone plugs an unauthorized computer in. This isn’t a bad method, if you have your network fully documented, don’t make changes often, and want the extra management overhead. I have a Church IT friend here in Indianapolis who I know does just that…awesome! I lock down some ports like this–nursery checkin stations or public internet terminals primarily. But for the rest of the network, I finally got around to implementing something I knew existed but never had time to research until now: DHCP Snooping. The cool name is just a side benefit!

If your switch(es) support DHCP Snooping, it’s pretty easy to turn on, but you need to know a little about your network first. Specifically, you need to know:

• Which switch port(s) your valid DHCP server is connected to.
• Which switch port(s) are uplinks to other managed switches.
• What VLANs do you want DHCP Snooping protection enabled on?
• Optionally, what the IP address is of your DHCP server (or at least which IP is assigned to the server in each VLAN where you want DHCP Snooping enabled).

If you choose to configure the authorized DHCP server IP address(es) list, the switches will require the DHCP reply to come from one of the authorized IPs; if you don’t configure the list then only the switch port source matters.

Let’s review briefly how DHCP works at a high level. Computer or device is connected to the network and turned on. It sends a DHCP broadcast request to the local segment and a DHCP server that receives the request allocates an available IP address and replies with that IP to the requesting client device, which they has a “lease” on the IP until the expiration time defined by the server. At varying intervals before the lease expires, the client sends a renewal request to the originating DHCP server asking if it can keep the IP longer, and the server replies that (usually) yes it can and extends the expiration. The client has no idea what DHCP servers are available initially, hence the broadcast request. If there are multiple DHCP servers on the network that see the request, they will all respond, and the client just picks the one that responds fastest and discards the rest.

Because the client accepts the first DHCP reply it receives, a cable/DSL router will often “beat” the correct DHCP server to the reply in some percentage of cases, creating a difficult to troubleshoot problem (which can be subtle if the IP, subnet, DNS and gateway addresses issued by the rogue DHCP server are similar in many ways to the legitimate settings, and more pronounced if they differ entirely). Tracking down the source of an unknown rogue DHCP server usually involves digging into the switch address tables and mapping IPs to MAC addresses to switch ports–a fun exercise if you want to learn about how switches work and have plenty of free time, but otherwise quite annoying! Even if the rogue server is sending the correct IP/subnet/DNS settings, its list of “available IPs” to hand out is maintained separately from the valid DHCP server, and thus you will end up with two devices being issued the same IP at some point, causing an IP conflict which may lead you to discover the existance of the rogue DHCP device after you’ve finished pulling your hair out :-)

So what is DHCP Snooping? It’s just the switch forcing valid DHCP replies (not requests from clients–a DHCP reply is the server issuing an IP assignment to a client who requested one) to only come from valid DHCP servers that you specify and tell the switch about. Your DHCP server is plugged into a particular port on your switch. You configure DHCP Snooping to know that that port is “trusted” for DHCP replies. If a device is plugged into an untrusted port (all ports by default), if it tries to send a DHCP reply, the switch drops it, and it never goes anywhere! If all you have is one switch and one server, this is really simple. With multiple switches, it’s still simple but you do need to make the change on any switch where you enable DHCP Snooping. You’ll need to trust the port on a secondary switch that is the uplink to the switch where the DHCP replies will be coming from.

How about a brief example. Let’s say Switch 1 is your “core” network switch, and has your DHCP server plugged into port 1. Switch 2 is a secondary switch, and port 24 of Switch 1 is uplinked to port 24 of Switch 2. All the other ports on both switches have clients or other servers plugged into them, and let’s say you’re only using one flat VLAN (call it VLAN 1).

On switch 1, you need to tell it that port 1 is trusted so the DHCP server can send its replies on that port. You can optionally tell it port 24 is trusted since your other switch is connected to that port (and it’s a rogue DHCP server we hope!), but since it will only be sending DHCP replies out too the other switch and not have them coming back “in” the port, it’s not required. Switch 2 requires that you make port 24 trusted, since it will be receiving DHCP replies for its clients incoming on that port from the server that’s connected to Switch 1. In more complex networks, there may be reasons to have DHCP traverse both directions of an uplink port, and since switches are generally trusted to not randomly sprout internal DHCP servers, it’s probably easier to just make all uplink ports, regardless of direction, trusted for DHCP Snooping purposes. However, this only applies to uplinks to switches that support DHCP Snooping and have it turned on–don’t trust a port that has an unmanaged switch connected or one without DHCP snooping enabled, or any client on that switch can then send DHCP replies to any other client on the managed switch, defeating your entire protection! So only mark ports trusted if they are connected to a DHCP server or if they are connected to another switch which also has DHCP Snooping enabled.

Optionally, tell your switches the valid IP address(es) of the DHCP server so they can drop replies from invalid IP addresses, even on trusted ports. I did this only on my “core” switch rather than every one of my managed switches that supports snooping, just for ease of management.

Sounds great! How do you do it? Well I could post the mechanics but it’s been describe elsewhere very simply. You have to use the command line on ProCurve switches, not the command line menu or the web interface. Get to the command line, type “config” to enter configuration mode, and then follow the directions here (the article is good if you ignore the misspelled word “rogue” in the title):

Preventing Rogue DHCP Servers with HP Procurve Switches

Don’t forget to turn off option 82 per that article…I didn’t try leaving it on but it works for me with it turned off. You may need to check this out in more detail if you’re doing any sort of multi-VLAN setup with routing where you use DHCP Relay to get other subnets to the DHCP server, I haven’t tested that. And I’d run the first command last (just plain “dhcp-snooping”), it turns the filtering on but if you set the options first (and you did it correctly) you won’t prevent any good traffic by configuring first, then enabling!

Another excellent resource is from HP themselves, a four-page PDF titled, “How to configure DHCP Snooping on ProCurve switches.” Definitely read and understand both the above blog post and this PDF document before setting this up! And make sure to test during scheduled maintenance/downtime…if you get it wrong, your network will probably stop working :-) Also, there are additional options on some ProCurve switches called arp-protect that use the dhcp-snooping database to verify arp packets and prevent arp spoofing attacks. However, this is even easier to screw up and block even good stuff–I don’t recommend you play with it unless you really know what you’re doing :-)

Some of my ProCurve switches, namely the 2810-24G units and all of the 1800 series, don’t support DHCP Snooping. You can check the manual, or at the command line (except on the 1800 series which has no command line) type dhcp-snooping followed by a space and question mark (“dhcp-snooping ?”) to see if it provides you with help about the command. If the command doesn’t exist, your switch model doesn’t support it. It’s working for me on the 5304xl and the 2650s, but not the 2524 or the 2810-24G (the last one surprises me, the 2524 doesn’t). Switches that don’t support it are still going to be vulerable to rogue DHCP servers, but the damage will be limited to that segment at least and not your whole network!

Any comments? Are you using DHCP Snooping? Have you run into a situation where you wish you’d had it turned on but didn’t? (I have, fortunately few and far between. But at least once it was one of my servers that I had accidentally configured impoperly! “Rogue DHCP server” doesn’t mean malicious or even end-user created, it can just as easily be “the server admin messed up” :-)

Oh yeah, one more tip! On ProCurve switches at least, once you have DHCP Snooping set up, you can get a few details and stats about the assignments. Try these three commands to get a configuration report, view statistics, and view the current bindings databaes:

show dhcp-snooping

snow dhcp-snooping stats

show dhcp-snooping binding

Tada! The End (you thought I’d never get here, didn’t you? :-)

UPDATE: Forgot to provide a link to this article for further reading about ARP Spoofing protection that I briefly mentioned. Good description and flowchart, but there are side-effects you may not realize at first so like I said, be careful with arp-protect :-) Exploiting arp is usually intentional, not accidental like rogue DHCP often is, so hopefully it will be less of a problem especially in churches!

UPDATE 2: Derek Schwab reminded me that the ProCurve switches that support DHCP Snooping are layer 3 switches, while the ones that don’t are layer 2 (and thus don’t function at the IP level where DHCP does). Thanks Derek, you’re right and I didn’t make the connection myself–that’s what friends are for!

Two years ago, almost to the day (which I just noticed!), I went up to the Indiana District Assemblies of God campground in Hartford City,  IN. Purpose? Set up and test four Meraki Mini mesh routers with their satellite internet connection. You can read about my initial escapade here and here (corny play on words and all :-)

Here’s what’s changed in two years: Meraki has since redefined their entire business and offers much more expensive solutions, and no Meraki Minis. Also, the campground can now get DSL and not just satellite internet, which is awesome. And although Meraki is for my purposes defunct, Open-Mesh has taken over where Meraki left off and has a similar device at the same price, with better accessories and more power!

This time, we’re covering more ground as well. So I just ordered nine Open-Mesh OM1P Professional Mini Routers. And six 7 dbi antennas, plus three Indoor Wallplug Enclosures. This time I’m going to be covering more area, and I’m hoping that using some larger antennas as well as the reports I’ve heard that the Open-Mesh devices have better range than the Meraki units out of the box mean that we’ll have a very successful network this time! We’ll also have two or three DSL lines to serve as injection gateways, which should be a major improvement over the horrendous satellite connection we had before (if you could call it a connection half of the time when it wasn’t, you know, connected :-)

I plan on taking some pictures and documenting the setup more than last time, and if I find the time I might even blog some of it!

Did I mention my whole order including shipping was under $550? That’s cool.

If you’re like many Church IT guys, you work at a church. For church pay. And likely do some side work to help put food on the table, pay off debts, take pilot lessons, buy motorcycles, or upgrade your homebrew DVR to record HDTV.

The best tools I’ve found for dealing with the financial and invoicing sides of side work (pun fully intended :-) are these:

FreshBooks for invoicing. Free for up to 3 clients, a bit high but reasonable for more, and integrates with way more third-party services than their competitors (although some competitors are a bit cheaper and do things a little differently). They’re on  Twitter at @FreshBooks if you want to follow them or keep in touch.

IAC-EZ for accounting. If you’re making more than a few bucks, FreshBooks will help you invoice clients and track time, but expenses, taxes, and other actual accounting stuff you should track can be taken care of in IAC-EZ. I’ve gotten to know the owner of IAC-EZ over email and Twitter and was involved in beta-testing the product, and it’s not only in active development with new features and fixes, but the owner and others know their stuff in the acccounting and business worlds and aren’t shy about helping if you ask. There’s a trial and then it’s $20/mo. If you do a little side work it probably won’t be affordable but if you have enough you need to track your finances better, you can (and probably should) spare the cash. Oh yeah, and it integrates with FreshBooks, so whatever you do in FreshBooks is pulled right into IAC-EZ, avoiding entering things twice! This alone is reason enough to pick this over something else if you use FreshBooks. They are on Twitter at @IACEZ and so is the owner as @IAC_Heather.

Minibooks from Groovysquared for FreshBooks from your iPhone. I helped beta-test this app as well, and if you have an iPhone and use FreshBooks, it’s worth checking out. It’s very polished and although it’s missing some features like invoicing by time and sending estimates, those are being worked on and the feature set that is there is much more complete compared to the official FreshBooks iPhone app, which just tracks time. Minibooks does time tracking in an extremely elegant way (better than anything I’ve seen!), lets you browse and edit your clients, create, view, and edit invoices, and mark invoices as paid. It’s kind of like FreshBooks-in-a-phone. I’m missing the invoice-from-time-tracking and estimates features personally, but that’s not hard to do from my computer for now.

I use all of these tools personally and really like them. I know friends using FreshBooks and IAC-EZ who like them as well. Disclosure: the FreshBooks and IAC-EZ links are referral links and I’ll get credit and a small referral fee if you use them (you can find them easily online, guessing the domain names even, if you don’t want to use the referral links. But remember, I’m a Church IT guy :-) I received a free copy of Minibooks for my beta testing efforts but otherwise am uncompensated by Groovysquared. None of the companies saw or directly influenced this post before I published it. Lakeview Church does not endorse these products or benefit from their use in any way (other than their IT guy being more financially able to stick around at the job he loves :-)

The new iPhone 3.0 software comes with the ability to tether your phone to your laptop via Bluetooth or USB so you can use it to get on the internet, but for subscribers in some countries the feature comes disabled when you upgrade to the 3.0 software version. However, some people discovered a way to supposedly use Beta 4 of iTunes for Windows (specifically version 8.2.0.10) (I hear it works on any iTunes version on the Mac) and a special carrier update file ending in the .ipcc file extension to enable the tethering option on their iPhone 3G units now.

The problem that some people didn’t realize at first when they posted this update method, was that the .ipcc file also changes an option that disables Visual Voicemail and Missed Calls, at least in some cases. I don’t recall which of the many sites it was where I read this, but the solution is quite simple for anyone out there who might need it. From the Home screen, navigate to Settings->General->Network->Cellular Data Network. The section called Visual Voicemail has a field called “APN” and if it’s set to “wap.cingular” and you use the US iPhone carrier, that appears to be what’s causing problems. The “APN” field should read: “acds.voicemail” (without the quotes).

Note that this is for informational and troubleshooting purposes only and changing any of the settings or files here is not something I endorse. Tethering is a service that you may or may not be allowed to do and is between you and your service provider; please refer to their agreement terms for details and specifics. I didn’t come up with the solution above, I just happened to read it somewhere and I’ve been contacted by a couple of people who mentioned they had problems that were fixed when I pointed out the above change that I read about somewhere.

Most churches are using email now to connect with their members. Sure many were slow on the uptake and there are still some that don’t use it…there are enough churches around to cover all of the technological (or lack thereof) spectrum. We’ve certainly been through a few technolgies and right now we use (or are starting to use at least) Vertical Response for email, with their 10,000 free emails per month for non-profits. But technology continues to move forward, and the “latest and greatest” used to be cell phone text messaging (SMS, or Short Message Service–did anyone even know what the acronym stood for until reading it here? :-) but of course that’s already being edged out in many places by Twitter and other social media. We haven’t actually done any text messaging to cell phones that I know of here (and we may bypass this altogether for Twitter, which we’ve only recently begun to semi-officially use Twitter for the Church, although I’ve been using it personally for a long time), but I’ve done some research on options that I passed along to our Youth department back in January after doing some Googling and talking to some Church IT guys to see what they were doing. Some of the options are pretty cool, so for reference, here’s the email (slightly edited) I sent to our Youth dept. back in January in case you find it helpful:

These are some text messaging services that may be useful:

• http://texthub.com/ (another church where I know the Church IT guy, I forget which one, is using this)
• http://www.jarbyco.com (Granger Community Church has used this, as have a few others)
• http://www.textmarks.com (Granger is using this some too and we are planning to use it at Lakeview at least among staff if not generally)
• http://www.polleverywhere.com (this lets you show poll results on the screen that people text in, but I don’t think it does mass texting—Jarbyco can do this too I think, at least where people sendin questions via text message and you can have someone put them up on the screen)

I’d start there and see where it gets you! Should be cheaper than phone tree most likely, def. cheaper than mailing postcards and you’ll probably get more teen response than from either of those anyway :-) [remember this was an email to the Youth dept.] I would start testing with a small group if you’re going to test multiple services, so you aren’t switching everyone from one system to another just to test…like set up 4-5 students with one service and see how you like the service and price before putting everyone in.

I later ran into another service worthy of comparison: http://www.churchtextingmanager.com/. I’m sure there are others out there. Other than light testing of Poll Everywhere and TextMarks, I’ve not used any of these services personally or professionally and can’t vouch for any of them, but it’s probably a good list to start your own research!

We, along with over 6,000 other venues, are hosting the Dave Ramsey Town Hall For Hope event tomorrow night (Thursday the 23rd of April). There have been several tests planned because the event is being distributed via the internet (primarily) to most event host locations. As you can imagine, that’s quite a bit of bandwidth and with a large event, a lot of redundancy and backups that LifeChurch.tv (the live event host) gets to have in place to make things run smoothly!

Today there are two final tests (each venue can pick one) where the stream is run for a couple of hours to the venues to make sure things are going smoothly. Last week there were some various hiccups that they found and fixed and this past Monday the test went very smoothly. We ran the afternoon test (the other is tonight) today in our Youth Center where we’re hosting the event and just after the official test, I decided to test our bandwidth with Comcast. I kept adding streams until I was streaming the 2.5Mbps (highest available) stream seven different times! Bandwidth peaked at over 18.5 Mbps downstream with all those streams running at the same time! And I think we had some bandwidth to spare (this is on our Comcast Business internet connection). Our connection is rated for 16 Mbps down and 2 Mbps up, while I’ve seen speed tests recently as high as 30 Mbps down and 4.5 Mbps up. Certainly the almost-19Mbps speed seen here is excellent and above our rating!

I’ve posted a bandwidth graph showing our internet connection’s utilization (also on TwitPic):

Cacti Graph - Town Hall For Hope Test 7x 2.5 Mbps Stream

Comcast Fun

Of course we almost missed the test this morning because someone cut our main Comcast tap this morning just before it went under the parking lot to our building. You can see the actual cut cable (and a part of my shoe) in the picture I uploaded to TwitPic earlier. This caused a four-hour internet outage (8am to noon) that I managed to get back up once I realized (thanks to some prompting from our awesome Facilities Director Mike Moore) that the other end of our building has a completely separate cable tap from Comcast for the TVs on that end of the building! That tap was unharmed so I moved the modem to that IDF and plugged into the tap. I adjusted some VLAN configuration settings to put the firewall’s WAN port on a private VLAN with the modem’s LAN interface (it was plugged in directly before) and tada, at 11:58 am (two minutes before we were scheduled to test the Town Hall For Hope stream) the internet came back!

Comcast did come out later (during the Town Hall For Hope test in fact) and repair the cable that was cut. I’ll be moving the modem back after hours; the TVs are working so I’m going to assume the modem will be fine back on its original line as well. I’m really glad we had that second tap though, because we would have had to push the Town Hall For Hope test off until tonight when the youth group uses the room we’re using, and we wouldn’t have gotten as good of a test. And kudos to Comcast for their fast response to our issues, even though they weren’t the cause.

So, today didn’t quite go as planned, but given the issues I think we had plenty of successes. And I’m not going to worry about blocking free wifi or other bandwidth use during the Town Hall event tomorrow night; since we’re only doing one stream I think we can handle it! In fact, I just realized that if we overflow that Youth Center venue for some reason (which I doubt we will not because it’s not going to be a big event, but because there are so many other churches also hosting it), there’s no reason we can’t handle adding a feed to our main sanctuary as well if necessary. I like being prepared. Just keep the backhoe’s away from the property!

UPDATE 10:45 am 4/1/2009: Silly me, I forgot until right before I was about to go to bed last night that there was a Veeam Backup 3.0.1 release I needed to upgrade to from 3.0. (I even mentioned remembering the upgrade in my post at the last minute before I published it.) Upgrading fixed the problem and it appears to be working normally again now! Apologies for the fuss; if you have Veeam Backup and ESXi Free make sure to upgrade both if you haven’t! I’m leaving the post with details of the errors below; if someone finds the information they’ll at least know they need to upgrade :-) Maybe I should just call it an April Fools joke? Ha.

I recently purchased Veeam Backup 3.0 to back up my three VMware ESXi Free hosts. Veeam Backup is awesome and their version 3.0 is the first version to support the free ESXi version! I love the deduplication and compression and the ease of use when making backups! On March 31st, VMware released ESXi 3.5 Update 4, which added drivers for some very nice NetXtreme quad-port Gigabit Ethernet cards, which I have in two of my three VM host servers but have been unable to use until they released an updated version with built-in drivers for that hardware.

So I upgraded yesterday when Update 4 was released (I actually just did a point release update to new Update 3 firmware the night before…doh!). The new NICs work great and now I have redundant paths to the SAN! (In one case I now have more than one NIC in the whole box that was doing SAN and LAN just on VLANs, so it’s quite nice to have multiple NICs available now!)

I was going through and upgrading VMware Tools on all of my virtual machines (the new release adds some driver support for enhanced NICs to Server 2003 and a few other minor things). One of my Linux CactiEZ VMs was being a bit picky with the yum package I was trying to install so after some troubleshooting I figured I’d restore a virtual machine from Veeam Backup (granted not 3.0.1 which I believe is out, I have the original 3.0 release installed right now) to get an earlier state and see if it helped to start fresh (my other thought was there was a repository issue but my older CactiEZ 0.4 yum was working just fine, it was my CactiEZ 0.6 box I recently set up that was having issues (it runs CentOS 4.7)).

But my restore fails, with an error relating to not being able to create the directory on the ESXi host to restore the virtual machine. The exact error is along the lines of:

Failure to restore item “VM Name Here” Cannot make directory ‘[datastore] VM Name Here’ on ‘ha-datacenter’. Soap fault. fault.RestrictedVersion.summaryDetail: ‘<RestrictedVersionFault xmlns=”urn:intervalvim25″ xsi:type=”RestrictedVersion”></RestrictedVersionFault>’, endpoint: “

The simpler error is in the status dialog box, “Restore error: Restore VM failed: Cannot make dir…”

I dig a little deeper and notice that the last couple of backup jobs scheduled to run overnight for some virtual machines have all failed completely. Nothing updated, and when I force a backup to start now it fails quickly for all VMs with an error along these lines:

Releasing VM files

CreateSnapshot failed, vmRef 224, timeout 1800000, snName “VEEAM BACKUP TEMPORARY SNAPSHOT”, snDescription “Please do not delete this snapshot. It is being used by Veeam Backup.”, memory False, quiesce True

fault.RestrictedVersion.summary

So, I’ve submitted a support ticket. Fortunately, right now I have nothing urgent that needs to be restored (CactiEZ is more of a plaything right now, at least my new 0.6 install), although obviously not keeping backups up to date is not a good thing.

I guess I’ve been running ESXi without Veeam (Veeam’s only been running for…maybe a month?) for long enough that I wasn’t considering backups when I did my ESXi upgrades, so I’ll admit first-day upgrading is jumping the gun. But Veeam is a VMware partner as far as I know, and I don’t know why they haven’t been able to work with VMware around this release to verify that their software works…it’s not like this is ESXi 4, it’s just an Update release of 3.5. At least an announcement of the incompatibility with a warning about upgrading sent to customers would have been nice, although it’s not something that was promised or anything.

I’ll keep this updated (here or in the comments) as the “story” progresses! Tomorrow I will also look into making sure I’m on the very latest point release of Veeam Backup to see if that makes a difference…just don’t have the energy left tonight to do anything else, I was up until 5:30 am last night doing a P2V of our nursery checkin system (long but successful!).