But how do you stop these “rogue” DHCP servers from accidentally or intentionally wreaking havoc on your network if plugged in? There are a couple of options, all of which involve managed switches and I’m going to talk in particular about HP ProCurve switches since that’s what I have (and love). I know at least some Cisco and Dell switches have similar functionality, and likely others.

You could do something as extreme as locking down every port with MAC address security so the entire port will shut down if anyone plugs an unauthorized computer in. This isn’t a bad method, if you have your network fully documented, don’t make changes often, and want the extra management overhead. I have a Church IT friend here in Indianapolis who I know does just that…awesome! I lock down some ports like this–nursery checkin stations or public internet terminals primarily. But for the rest of the network, I finally got around to implementing something I knew existed but never had time to research until now: DHCP Snooping. The cool name is just a side benefit!

If your switch(es) support DHCP Snooping, it’s pretty easy to turn on, but you need to know a little about your network first. Specifically, you need to know:

• Which switch port(s) your valid DHCP server is connected to.
• Which switch port(s) are uplinks to other managed switches.
• What VLANs do you want DHCP Snooping protection enabled on?
• Optionally, what the IP address is of your DHCP server (or at least which IP is assigned to the server in each VLAN where you want DHCP Snooping enabled).

If you choose to configure the authorized DHCP server IP address(es) list, the switches will require the DHCP reply to come from one of the authorized IPs; if you don’t configure the list then only the switch port source matters.

Let’s review briefly how DHCP works at a high level. Computer or device is connected to the network and turned on. It sends a DHCP broadcast request to the local segment and a DHCP server that receives the request allocates an available IP address and replies with that IP to the requesting client device, which they has a “lease” on the IP until the expiration time defined by the server. At varying intervals before the lease expires, the client sends a renewal request to the originating DHCP server asking if it can keep the IP longer, and the server replies that (usually) yes it can and extends the expiration. The client has no idea what DHCP servers are available initially, hence the broadcast request. If there are multiple DHCP servers on the network that see the request, they will all respond, and the client just picks the one that responds fastest and discards the rest.

Because the client accepts the first DHCP reply it receives, a cable/DSL router will often “beat” the correct DHCP server to the reply in some percentage of cases, creating a difficult to troubleshoot problem (which can be subtle if the IP, subnet, DNS and gateway addresses issued by the rogue DHCP server are similar in many ways to the legitimate settings, and more pronounced if they differ entirely). Tracking down the source of an unknown rogue DHCP server usually involves digging into the switch address tables and mapping IPs to MAC addresses to switch ports–a fun exercise if you want to learn about how switches work and have plenty of free time, but otherwise quite annoying! Even if the rogue server is sending the correct IP/subnet/DNS settings, its list of “available IPs” to hand out is maintained separately from the valid DHCP server, and thus you will end up with two devices being issued the same IP at some point, causing an IP conflict which may lead you to discover the existance of the rogue DHCP device after you’ve finished pulling your hair out :-)

So what is DHCP Snooping? It’s just the switch forcing valid DHCP replies (not requests from clients–a DHCP reply is the server issuing an IP assignment to a client who requested one) to only come from valid DHCP servers that you specify and tell the switch about. Your DHCP server is plugged into a particular port on your switch. You configure DHCP Snooping to know that that port is “trusted” for DHCP replies. If a device is plugged into an untrusted port (all ports by default), if it tries to send a DHCP reply, the switch drops it, and it never goes anywhere! If all you have is one switch and one server, this is really simple. With multiple switches, it’s still simple but you do need to make the change on any switch where you enable DHCP Snooping. You’ll need to trust the port on a secondary switch that is the uplink to the switch where the DHCP replies will be coming from.

How about a brief example. Let’s say Switch 1 is your “core” network switch, and has your DHCP server plugged into port 1. Switch 2 is a secondary switch, and port 24 of Switch 1 is uplinked to port 24 of Switch 2. All the other ports on both switches have clients or other servers plugged into them, and let’s say you’re only using one flat VLAN (call it VLAN 1).

On switch 1, you need to tell it that port 1 is trusted so the DHCP server can send its replies on that port. You can optionally tell it port 24 is trusted since your other switch is connected to that port (and it’s a rogue DHCP server we hope!), but since it will only be sending DHCP replies out too the other switch and not have them coming back “in” the port, it’s not required. Switch 2 requires that you make port 24 trusted, since it will be receiving DHCP replies for its clients incoming on that port from the server that’s connected to Switch 1. In more complex networks, there may be reasons to have DHCP traverse both directions of an uplink port, and since switches are generally trusted to not randomly sprout internal DHCP servers, it’s probably easier to just make all uplink ports, regardless of direction, trusted for DHCP Snooping purposes. However, this only applies to uplinks to switches that support DHCP Snooping and have it turned on–don’t trust a port that has an unmanaged switch connected or one without DHCP snooping enabled, or any client on that switch can then send DHCP replies to any other client on the managed switch, defeating your entire protection! So only mark ports trusted if they are connected to a DHCP server or if they are connected to another switch which also has DHCP Snooping enabled.

Optionally, tell your switches the valid IP address(es) of the DHCP server so they can drop replies from invalid IP addresses, even on trusted ports. I did this only on my “core” switch rather than every one of my managed switches that supports snooping, just for ease of management.

Sounds great! How do you do it? Well I could post the mechanics but it’s been describe elsewhere very simply. You have to use the command line on ProCurve switches, not the command line menu or the web interface. Get to the command line, type “config” to enter configuration mode, and then follow the directions here (the article is good if you ignore the misspelled word “rogue” in the title):

Preventing Rogue DHCP Servers with HP Procurve Switches

Don’t forget to turn off option 82 per that article…I didn’t try leaving it on but it works for me with it turned off. You may need to check this out in more detail if you’re doing any sort of multi-VLAN setup with routing where you use DHCP Relay to get other subnets to the DHCP server, I haven’t tested that. And I’d run the first command last (just plain “dhcp-snooping”), it turns the filtering on but if you set the options first (and you did it correctly) you won’t prevent any good traffic by configuring first, then enabling!

Another excellent resource is from HP themselves, a four-page PDF titled, “How to configure DHCP Snooping on ProCurve switches.” Definitely read and understand both the above blog post and this PDF document before setting this up! And make sure to test during scheduled maintenance/downtime…if you get it wrong, your network will probably stop working :-) Also, there are additional options on some ProCurve switches called arp-protect that use the dhcp-snooping database to verify arp packets and prevent arp spoofing attacks. However, this is even easier to screw up and block even good stuff–I don’t recommend you play with it unless you really know what you’re doing :-)

Some of my ProCurve switches, namely the 2810-24G units and all of the 1800 series, don’t support DHCP Snooping. You can check the manual, or at the command line (except on the 1800 series which has no command line) type dhcp-snooping followed by a space and question mark (“dhcp-snooping ?”) to see if it provides you with help about the command. If the command doesn’t exist, your switch model doesn’t support it. It’s working for me on the 5304xl and the 2650s, but not the 2524 or the 2810-24G (the last one surprises me, the 2524 doesn’t). Switches that don’t support it are still going to be vulerable to rogue DHCP servers, but the damage will be limited to that segment at least and not your whole network!

Any comments? Are you using DHCP Snooping? Have you run into a situation where you wish you’d had it turned on but didn’t? (I have, fortunately few and far between. But at least once it was one of my servers that I had accidentally configured impoperly! “Rogue DHCP server” doesn’t mean malicious or even end-user created, it can just as easily be “the server admin messed up” :-)

Oh yeah, one more tip! On ProCurve switches at least, once you have DHCP Snooping set up, you can get a few details and stats about the assignments. Try these three commands to get a configuration report, view statistics, and view the current bindings databaes:

show dhcp-snooping

snow dhcp-snooping stats

show dhcp-snooping binding

Tada! The End (you thought I’d never get here, didn’t you? :-)

UPDATE: Forgot to provide a link to this article for further reading about ARP Spoofing protection that I briefly mentioned. Good description and flowchart, but there are side-effects you may not realize at first so like I said, be careful with arp-protect :-) Exploiting arp is usually intentional, not accidental like rogue DHCP often is, so hopefully it will be less of a problem especially in churches!

UPDATE 2: Derek Schwab reminded me that the ProCurve switches that support DHCP Snooping are layer 3 switches, while the ones that don’t are layer 2 (and thus don’t function at the IP level where DHCP does). Thanks Derek, you’re right and I didn’t make the connection myself–that’s what friends are for!

Here’s what’s changed in two years: Meraki has since redefined their entire business and offers much more expensive solutions, and no Meraki Minis. Also, the campground can now get DSL and not just satellite internet, which is awesome. And although Meraki is for my purposes defunct, Open-Mesh has taken over where Meraki left off and has a similar device at the same price, with better accessories and more power!

This time, we’re covering more ground as well. So I just ordered nine Open-Mesh OM1P Professional Mini Routers. And six 7 dbi antennas, plus three Indoor Wallplug Enclosures. This time I’m going to be covering more area, and I’m hoping that using some larger antennas as well as the reports I’ve heard that the Open-Mesh devices have better range than the Meraki units out of the box mean that we’ll have a very successful network this time! We’ll also have two or three DSL lines to serve as injection gateways, which should be a major improvement over the horrendous satellite connection we had before (if you could call it a connection half of the time when it wasn’t, you know, connected :-)

I plan on taking some pictures and documenting the setup more than last time, and if I find the time I might even blog some of it!

Did I mention my whole order including shipping was under $550? That’s cool.

Today there are two final tests (each venue can pick one) where the stream is run for a couple of hours to the venues to make sure things are going smoothly. Last week there were some various hiccups that they found and fixed and this past Monday the test went very smoothly. We ran the afternoon test (the other is tonight) today in our Youth Center where we’re hosting the event and just after the official test, I decided to test our bandwidth with Comcast. I kept adding streams until I was streaming the 2.5Mbps (highest available) stream seven different times! Bandwidth peaked at over 18.5 Mbps downstream with all those streams running at the same time! And I think we had some bandwidth to spare (this is on our Comcast Business internet connection). Our connection is rated for 16 Mbps down and 2 Mbps up, while I’ve seen speed tests recently as high as 30 Mbps down and 4.5 Mbps up. Certainly the almost-19Mbps speed seen here is excellent and above our rating!

I’ve posted a bandwidth graph showing our internet connection’s utilization (also on TwitPic):

Cacti Graph - Town Hall For Hope Test 7x 2.5 Mbps Stream

Comcast Fun

Of course we almost missed the test this morning because someone cut our main Comcast tap this morning just before it went under the parking lot to our building. You can see the actual cut cable (and a part of my shoe) in the picture I uploaded to TwitPic earlier. This caused a four-hour internet outage (8am to noon) that I managed to get back up once I realized (thanks to some prompting from our awesome Facilities Director Mike Moore) that the other end of our building has a completely separate cable tap from Comcast for the TVs on that end of the building! That tap was unharmed so I moved the modem to that IDF and plugged into the tap. I adjusted some VLAN configuration settings to put the firewall’s WAN port on a private VLAN with the modem’s LAN interface (it was plugged in directly before) and tada, at 11:58 am (two minutes before we were scheduled to test the Town Hall For Hope stream) the internet came back!

Comcast did come out later (during the Town Hall For Hope test in fact) and repair the cable that was cut. I’ll be moving the modem back after hours; the TVs are working so I’m going to assume the modem will be fine back on its original line as well. I’m really glad we had that second tap though, because we would have had to push the Town Hall For Hope test off until tonight when the youth group uses the room we’re using, and we wouldn’t have gotten as good of a test. And kudos to Comcast for their fast response to our issues, even though they weren’t the cause.

So, today didn’t quite go as planned, but given the issues I think we had plenty of successes. And I’m not going to worry about blocking free wifi or other bandwidth use during the Town Hall event tomorrow night; since we’re only doing one stream I think we can handle it! In fact, I just realized that if we overflow that Youth Center venue for some reason (which I doubt we will not because it’s not going to be a big event, but because there are so many other churches also hosting it), there’s no reason we can’t handle adding a feed to our main sanctuary as well if necessary. I like being prepared. Just keep the backhoe’s away from the property!

What was my solution this time? VMware Player and Windows XP! I grabbed a Windows XP virtual machine and ran it on my laptop in VMware Player, which worked just fine on Windows 7 (another reason not to run the Comcast stuff…who knows if it’s Windows 7 compatible). The biggest “issue” I had was that I had to disable all protocols bound to the LAN adapter on my laptop except for the VMware Bridge Adapter (to allow the virtual machine network access). That way the virtual machine got the DHCP and proxy settings from the cable modem when I turned it on, rather than my laptop (leaving the VM unable to connect). Once the VM had the “only” network connection, the wizard proceed normally and I got everything connected just fine (the wizard is much more streamlined than it used to be at least).

After it’s working, I just turned off the cable modem and plugged the WAN port of my wireless router into the modem, and turned it back on. Everything was smooth sailing from there. It does apparently lock to your MAC address but that is reset when the modem reboots.

Interestingly, the modem is an RCA brand modem, which is funny since I asked specifically when I talked to Comcast yesterday what brand the modem would be and they specifically said a Surfboard, which is by Motorola. Oh well, as long as it works I don’t care too much. Better than my old DLink from last time!

Now, if only Comcast would expand their trial of 16-20Mbit speeds (instead of 6Mbit) from South Bend to Indy, life would be awesome!

I’ve gotten a few draft posts written but nothing finished yet (I know it’s been a while!). However, the big news today is that Veeam Backup 3.0 was released this morning, which I already have a license for and I’m working to clean off the server that I’ll be running it on! It will definitely get a review when it’s up and running. I’m excited!

If a laptop user establishes a VPN connection to your corporate VPN server, and doesn’t use split tunneling (in other words, from the time they’re connected, all traffic goes through the VPN as its default gateway no matter what), assuming that you’re using a VPN client that verifies the identity of the server (rather than blindly trusting DNS, which is easily spoofable on a wireless network), the user moves from the realm of insecurity into a much more secure environment, similar to being plugged into your wired network at the office. Of course, then your office WAN connection has to support everything they do, including web browsing!

However, using a free or paid “VPN” service from a company that just turns your wireless connection into a VPN-enabled “wired” connection is only going to help thwart unencrypted wifi sniffing and other such attacks. Unless you also use SSL and other encryption technologies, those services are just giving you a wired internet connection just like your home connection rather than the easier-to-sniff unencrypted wireless. It’s better than nothing, but it’s not like an encrypted pipe into your own network.

Don’t discount unencrypted wireless attacks. It’s never happened to me, but if you hop over and read some of Security Monkey’s case files at you’ll discover that there’s a lot of bad stuff going on in the world on computers :-) Those case files are slightly modified true stories from this guy’s career! His old 2005-2007 podcast episodes are worth listening to for some cool security tips and tools as well, to digress for a moment!

I don’t have a good answer; VPN connections to the office make internet run very slowly unless you have the WAN bandwidth to support fast throughput to and from all your remote users including web browsing! But that’s a much more secure way to operate. The number of ways wireless can be hijacked, sniffed, spoofed, and hacked, especially if it’s unencrypted to begin with, is downright scary! At the very least use SSL with verified certificates for anything you do of any importance (or if passwords are transmitted) on an encrypted wireless connection. As an IT guy, I can tell you (or myself) whether a particular session (POP3, IMAP, RPC over HTTP, HTTPS, etc.) is happening over an encrypted connection or not and can be careful. However, the average user is, obviously, not going to know or even care necessarily if Outlook is using POP3 unencrypted or via SSL, or using RPC over HTTPS securely. And if they log into Gmail, they’re not likely to know that although their password is always encrypted on login, their email is transmitted in the clear unless they initiate the session using SSL from the start (using https://mail.google.com/ rather than http://mail.google.com)./ Even if their email contains passwords and confirmations for other accounts!

Stuart mentioned WiTopia on his comment to Tony’s original post. I’d never heard of them before, but I’ve seen similar services to their personalVPN product. That service appears to be, like I mentioned above, just a way to get a “wired quality” connection to the internet over unsecured wireless. An admirable service and a worthy goal even with its limitations, but what caught my eye even more was their SecureMyWifi service. It’s still a wireless service but it has to do with your own on-campus wireless access. It lets you move away from using WPA with a Pre-Shared Key (PSK), also known as WPA-Personal, and use their RADIUS services to authenticate users individually to your encrypted wireless access points. It seems a bit pricey (to me–it’s currently a $99 setup fee, $99/year for one access point, and $14.95/year for each additional access point), but we have the same thing set up using Microsoft’s free (built-in on Windows Server 2003) IAS RADIUS server in-house. If you aren’t familiar with how to set it all up, the WiTopia service could be quite beneficial! They charge per access point, but at Lakeview we have a centrally-managed access points system with one controller that takes care of authentication. I assume that the WiTopia service is based on unique RADIUS keys for each access point client; since the central controller (currently running 12 access points) acts as a single client, it should look like “one” access point to the service. Whether or not this is allowed with their terms of service I have no idea; we are not likely going to use the service since I already do this in-house for free, but I would recommend reading the terms and/or contacting them if you plan on doing something similar to remain in the spirit of their offering.

Before making a decision there are certainly things to evaluate, and I definitely want to get my hands on an NSA box for a while to test first. I like the ISA 2004 firewall interface that we’re currently running and I want to make sure I’m comfortable managing SonicWALL if we go that route.

I just found this yesterday, if you can believe I’ve found reason enough to rave already! Earlier this week I also set up the free version of the PacketTrap pt360 Tool Suite, and I’m significantly impressed. While several of the tools are part of a 30-day trial of the $1500 Pro version, the ability to easily map MAC addresses to DNS names and IP addresses is very useful, and the Dashboard, including a widget for viewing the traffic activity levels on switch ports, is nice. It didn’t blow me away like Spotlight on Windows did (and they don’t overlap too much except in some monitoring areas; pt360 is much more network oriented), but I’m keeping it handy in my arsenal for troubleshooting. It’s certainly not worth the $1500 to me for a Pro license, but someone with a much larger network might be able to justify the price tag. Although I believe I’ve run across this before, thanks to Andrew Mitry for linking to this tool in his recent blog post about free tools for IT Managers, where all the links were such high quality (the ones I did and didn’t know about) that I decided this was worth of a test run after all.

(Screenshots are from each products’ respective websites; click for larger versions.)

Something interesting to note is that while inbound traffic from the internet appeared to be blocked before the restart, I was able to use Remote Desktop from another server on the internal network to remotely instruct ISA to reboot. So it had not locked down all network access, just external. Good to know if you administer the box primarily via remote control! In fact, due to a lack of KVM switch ports, I have to manually plug the keyboard/monitor/mouse back in to ISA physically if I want to work on the console.

Although everything appeared to be functioning normally, today I got a report from a user who was getting a network error when attempting to connect to the iTunes Store from within iTunes. I tried it on my desktop, and got the same error. Fortunately, I remembered that back when I installed a prior ISA service pack (I don’t recall if it was 1 or 2), I had a similar problem and was able to track down the issue to the Compression Filter in ISA. If you go in the ISA Management Console to Configuration->Add-ins and check the Web Filters tab, by default there is a “Compression Filter” enabled (the description: “Enables HTTP/HTTPS compression”). Disabling this filter allowed iTunes Store to work just fine!

However, the reverse is true in ISA 2004 Service Pack 3. If you have disabled the Compression Filter, you must re-enable it for the iTunes Store to work in Service Pack 3! This is very useful information, so I thought I’d share! If you don’t know why iTunes Store doesn’t work, it can take a bit of Googling to determine the problem, at least it did for me originally. Perhaps the issue is more widely known by now.

I recently upgraded the firmware on all of the HP ProCurve switches we installed as part of our building project last year. It is so nice to have managed switches! We have ProCurve Manager Plus (we have version 2.1), HPs switch management application, and it makes upgrading the software on all of the beefy HP switches a breeze. What it doesn’t do is update the “smaller” switches, a.k.a. “less expensive,” such as the 1700-24 or the 1800 Series, which are web-managed only. Those switches must be upgraded using the web-based interface. We only have about four switches from those series, in various “edge” locations outside of our three primary network closets. The 1700-24 switch we have is currently running firmware version 1.05, while version 1.09 is the current release. This is supposed to be easy, just log into the web interface, click go to the Software Upgrade area, pick the firmware file and click Apply.

Unfortunately, I got an error message when I did this; the same error in Firefox 2 and Internet Explorer 6. The top frame where I had put the update file location would reload to a “page not found” page, while the lower frame where a status bar appeared remained frozen at 0%. I re-downloaded the update, tried several more times from both browsers, no dice. This is what the error looked like:

I decide to take advantage of ProCurve’s free technical support, and give them a call. I did wait on hold for about 20 minutes, but I was first directed to a live operator who asked me which product I was calling about so I could be placed in the correct support queue! I ended up speaking with Brandy, who tried an upgrade on her switch along with me and then collected a screenshot (see above) of my problem and the configuration file from my switch, via email. She had me try upgrading to version 1.07, the one in between my current version and the newest, with the same results. All of this seemed perfectly reasonable, and I didn’t feel like I was getting a script at any time.

Since it still didn’t work, Brandy made sure she had my number and was going to verify the information I sent her with an internal support person and get back to me. She called back less than two hours later, and although I missed her call, she left a voicemail and sent a follow-up email, and said they had decided to replace the switch if I would reply with my preferred shipping address and the switch model and serial number information! How easy is that?

Overall, although I have not received the replacement switch yet, I am very satisfied with the level of support I received on this issue, especially since this is possibly the least-expensive managed switch we have purchased from HP. The included technical support (and of course the lifetime warranty) were two huge reasons for choosing ProCurve, and the investment is bearing some fruit in those areas! The only other time I have needed to call them was when our Wireless xl switch controller module died within a month or two of installing it. They replaced it (we paid over $2000 for it) next business day (died on a Friday so we were down for the weekend), with perhaps a shorter phone call than this one! The fact that they offer a lifetime warranty on this module is amazing to me, because it is essentially a computer–you can clearly see the RAM modules, the processor, and other components on the module when it is not inside a host switch. Even HP desktops, with similar components (OK, the quality of the components may not be exactly similar), don’t get a lifetime warranty!

Go ProCurve! I am a fan…I hope I remain one!

It does go a lot faster with two people; one person would likely take several days to make one of these runs! We set up a makeshift spindle to allow the cable to somewhat easily unwind as we pulled it through the ceiling; I’ve provided a picture I snapped to show our (somewhat ingenious, if I do say so myself) cable pulling setup! The second run, with the setup shown in the picture, actually splits halfway through and three of the lines go one way (to our children’s auditorium) and the other three continue on to the office of our Facilities Manager (technically, Facilities Director, but then I can’t say FMer :-) (If you don’t get that, you had to be at the Roundtable at CoR in October!) Being this active for two days took a bit of a toll on my legs, but my eyes relished the break from the oft-near computer screens!

While I was at it, I threw in the VersaMail 3.5 EAS Meeting Invitation Update for good measure, just in case I needed it (and it was free!). I got all the updates available, basically; I must be used to Windows Updates where you should install it all, just in case it closes a huge gaping security hole or fixes something you don’t care enough about to research every time :-)

Everything went much more smoothly than expected! I installed VersaMail 3.5, installed and applied the updates, and did a sync. After the first sync, I was able to modify my account preferences to specify that I wanted to be notified “As items arrive,” which enables Direct Push.

So far, so good. Except that several minutes later, I hear my “New Mail Alert” sound and the message I get says, “EAS Account: Please press the Sync button.” Well, that’s fun. I go from email every 30 minutes to an annoying notice to manually “automatically” sync every five minutes! What is this? A Microsoft-like “improvement” from Palm?! Oh no! Things like “I wonder what would happen if I threw this phone through the window” started to go through my mind, but instead of following through with that I reverted to my backup plan:

Must…use…Google. Which I did, and I found this thread on the Palm forums that sounded like a broken record of my problem (not helpful) until I got to the last reply in the thread (which was helpful!). It had links to a blog entry from You Had Me At EHLO about Direct Push and Heartbeats, where, right there under list item 3 under the heading “Deployment Considerations for Direct Push” (I know, so easy to find in such a “short” entry :-) it talks about firewall connection timeouts with a link to an MSKB article (the same one the forum post linked to): 905013, Enterprise firewall configuration for Exchange ActiveSync Direct Push Technology. It’s reasonably short and sweet, and they’re kind enough to include step-by-step instructions for making the needed configuration change to our ISA 2004 firewall. Seriously, the instructions are so good and easy, I won’t even repeat them here. Click, click, click, type numbers, click, click, Apply, done. Or something like that.

I initiated a manual sync again to establish the connection with the new timeout values, and waited. Fifteen minutes later, no sign of the EAS Account error message! And now I get new emails popping up on my Treo usually before they show up in Outlook, whether connected via Cached Exchange Mode or not! Time will tell how good of a thing this actually is, but the concept is excellent!

One tool mentioned in the podcast this time is called Remote Task Manager, which is a remote control (at a granular level, not just a remote desktop viewer tool) for networked PCs. It sounds very useful and worth checking out the demo when I have the time!

I ran into a small issue with the SteadyState/Firefox setup that was a relatively easy fix: Firefox tried to update itself and the theme when new versions came out. Why it does this as a limited user when it can’t run the upgrade (for the program itself; the theme should work if it weren’t locked down) is beyond me, maybe I’ll file a bug report or something. Anyway, to get rid of the upgrade reminder, I logged in as Administrator and installed the Firefox program upgrade. Then I unlocked the profile and disabled Disk Protection, logged in as the locked down user, not not locked down, and upgraded the theme. Then I changed the Options (Tools->Options->Advanced->Update) and unchecked all of the automatic update options. Now updates won’t automatically (try to) apply, and I don’t even have to worry about security holes much because of the Disk Protection feature. I also took the opportunity to install the Auto Reset Browser extension and disable the old auto-restart mechanism (see below for the reasons).

Accessing Firefox Settings

To get to the Firefox settings, because of the R-Kiosk extension disabling menu access, I had to use the Firefox (safe mode) option from the Start menu, tell the statup box to disable add-ons and restart, and then it came up with no theme and no extensions active. I made my settings changes, installed the Auto Reset Browser extension, re-enabled the theme and the R-Kiosk extension, and restarted. Back to normal, with all changes made!

Firefox Auto-Restart Method

Paul Marc left a comment on my original post asking about how I made Firefox auto-restart if closed and on idle. I was using a batch file called start.bat that I found online, but I can’t seem to locate it again with Google (I recall it took some searching to find originally as well). I’ll have to grab the bookmark off of one of the computers I set it up on when I am able.

It seemed like it was a great solution when I set it up. However, I had several issues crop up in actual use. Sometimes it would get “stuck” in a loop of starting unending new Firefox windows as fast as the computer would open them. The only solution was to log off or restart (or kill the script, but the Task Manager won’t open under lockdown!). This only happens sometimes, and I’m not exactly sure why, but it makes the system unusable when it does happen.

I have made the above changes on three of the four computers (the last one isn’t switched yet because I ran out of time), setting them to not use the start.bat file, and instead installing the Auto Reset Browser extension in Firefox. It restarts the browser after every five minutes idle. The downside is, if a user closes the browser manually, it doesn’t reopen automatically. There is one icon on the desktop though, to open Firefox, so I don’t think this will be an issue, although it’s not as nice as the original solution when it worked correctly. And either way, closing manually or on idle, Firefox still runs the Clear Private Data option I had set up (per my original post) to get rid of the prior user’s cookies or other saved information.

Network Connection Details

In my original post, I neglected to include details of the network connections for the locked down systems. It’s pretty simple: stick the computers on the same VLAN (wired) as the free Wi-Fi internet access. I added each system’s MAC address into the Nomadix gateway so it doesn’t ask for a username or password, and I can control bandwidth on a per-computer basis (they don’t have much). The free Wi-Fi is firewalled so only OpenDNS can be contacted over the DNS ports, so they are subject to the OpenDNS adult site blocking we have in place, just like everyone else.

Overview

Our new youth facility now has a four-computer internet café. I’ve already written twice about my plans and research leading up to implementation, specifically about computer lockdown software. A couple of weeks ago, I mentioned briefly that we had changed course and decided to use Microsoft Windows SteadyState as our lockdown software of choice, mainly due to the price (free!).

We (Dutch volunteer Jeroen and I) were physically installing the computers/monitors/etc. in the youth lobby area when I thought, “hey, Microsoft just released some updated lockdown software, let’s try it out.” We hadn’t yet purchased the Fortres Grand software, although I had it approved. So I downloaded SteadyState, installed it, and messed around for a few minutes. It was so easy, even my mom could do it! Well, okay, I’ve been teaching her computers for a while and I might still have to walk her through this one over the phone, but I have no doubt she’d make it work :-) The installation went smoothly, the lockdown options (we wanted pretty much the tightest lockdown possible) were easy to select, and the hard disk protection (which discards changes on reboot) was easy to enable and control from within the main SteadyState console. I haven’t had experience with Microsoft’s old Shared Computer Toolkit, but from what I understand it was more difficult to combine all the options together into one functional system, and they appear to have fixed all of this in SteadyState.

Lockdown Features

In the SteadyState console, there are three items under Global Computer Settings: Set Computer Restrictions, Schedule Software Updates, and Protect the Hard Disk. The Set Computer Restrictions option lets you change things such as whether to display the last username in the logon screen, prevent users from writing to USB drives, turn the Welcome Screen on and off, and other miscellaneous things that affect the whole computer, not just particular user(s). I turned most of these on. I’m not writing this with access to the computers I set up, so I’m going from memory on this (and everything else) but if you have any questions about specifics please leave a comment!

You can create or import users/profiles that SteadyState can then manage with a selection of lockdown options going from low to high security, but at each level it just selects a more restricted subset of the detailed options and lets you customize away. This is similar to the functionality of the Fortres 101 software. All we tested was the highest security possible, locking down almost everything and only allowing the Mozilla Firefox executable to run. However, we did have to allow command prompt access to get the Firefox auto-restart trick below to work, although with the GUI and keyboard shortcuts this locked down, no one should be able to access the command line except through the batch file the Firefox shortcut links to for this trick to work.

Testing the lockdown settings to find the right mix can be a bit tricky because you must save the settings, log out, log in as the limited user, test, log out, and log back on to the administrative account again. It’s tedious, but once you have what you want, you can duplicate the settings more easily on other systems. The Export/Import Profile function works, but it imports a default user profile with the lockdown settings. Be careful with this, because it means you must wait until after you import a user into SteadyState from an exported profile before logging in and doing any customization to their desktop (display options, Start Menu positioning, etc.) as any customization you’ve done will be deleted if you import a user over top of your existing user! Found this out the hard way — once :-)

Firefox Does Its Own Privacy Work

Firefox has some great options for “Clearing Private Data” such as cache, cookies, history, saved passwords, authenticated sessions, etc., which for most Firefox users is either a manual option or something it prompts you to do when you close Firefox. Because of the multi-user environment, we instead set the options, available through the Firefox Tools->Options panel, to automatically clear private data when the browser was closed, with no prompting. That way someone logged into Gmail, Hotmail, Facebook, or lets face it, MySpace (one site I still refuse to sign up for :-) will be logged out when Firefox closes, safe for the next person to use. Let’s face it, these are teenagers we’re talking about here — do you think they’re going to remember to log off? Not likely in the vast majority of cases.

I found a batch file with some Google searching (I’ll have to re-locate it and post an update if anyone is interested) that, when run via a command line or a shortcut and passed the path to a .exe file, runs the file but monitors it and if the process ends, it restarts it automatically. So Firefox is in the Startup folder in the Start Menu, but run with this batch script. When someone closes Firefox, it clears their data, is automatically restarted, and goes back to the youth homepage automatically, ready for the next user!

But what if people don’t close the browser? We set up a Scheduled Task to kill the firefox.exe process after 5 minutes of the computer being idle. Same effect as the user closing the browser, and it automatically reopens still. This is a touch buggy, as occasionally Firefox will instead of reopening once, reopen window after window after window after window…and of course the computer is so locked down you can’t kill the process manually. It requires a logout or restart to fix. This is still on my “to track down” list, but it’s the last little piece of the puzzle, and generally it works fine. I’m sure it’s an issue with either the batch file, the scheduled task, or both interacting somehow.

Thematic Full Screen

The theme we chose for Firefox is called NASA Night Launch. It’s a beautiful theme, which shows an awesome shuttle launch shot as the blank background before a tab finishes rendering, and has equally nice toolbar backgrounds and a custom throbber (the top-right icon that moves while a page is loading, if you didn’t know). The grays and blacks in this theme look wonderful with our current homepage, www.infusionstudents.com, as well as the black LCD monitors mounted to the wall (pictures to follow later). A new version of this theme was released on July 22nd, after we set up the computers, so I will consider upgrading the theme at some point soon.

To make the slickest looking interface possible, we applied the R-Kiosk extension to Firefox to force it into fullscreen mode when it starts, getting rid of the title bar and any non-themed borders. We did apply the change to user.js that provides the navigation menu so the address bar and back/forward function. It looks really good with this extension combined with the theme!

While looking to see what the theme and extension we used are called, I just ran into an extension called Auto Reset Browser that for some reason I’ve never seen before. It looks like it might be a more elegant solution to my earlier problem, but I don’t know if it will help keep Firefox open if someone manually closes it. I will have to investigate further as time allows.

Disk Protection

SteadyState’s disk protection option, which you must enable separately from the policy lockdown settings, basically makes the hard drive immutable for most purposes. Do anything, reboot, and you’re back where you started last time. Fortres Grand’s Clean Slate product has similar functionality. Microsoft has made what I hear are improvements (compared to the Shared Computer Toolkit) in this functionality in that you enable and disable this option from the SteadyState control console just like all the policy options. Give it some time to make a cache file for the temporary disk changes, reboot, and the disk is protected.

The nicest thing is, if you’re an Adminstrator running SteadyState, and you log in, install a new program, and reboot — oops, if the disk protection was on you’d lose all your changes! You can unlock the disk for a time in the console, however. But the best option Microsoft added was a modification to the Log Off screen, prompting you that disk protection is on and giving you the option to discard all changes — or, keep the changes, restarting to merge the cache onto the hard drive automatically. That’s a no-brainer option that will continue to save my behind as I update these systems in the future I’m sure, long past initial setup! I’ve already used it for a few tweaks here and there.

No Manual Needed

SteadyState scores high marks for ease of use; I’ve still not read the manual and only referenced the help file (which opens automatically with the console) a few times. (Well, I did use the manual to refresh my memory while writing this post, but only because I don’t have access to the real systems at the moment. And this is the first time I’ve even opened it.)

Physical Installation

I don’t currently have any pictures of the computers handy, so I will leave photos and a description of the mounting process (which comprised more than 50% of the entire operation) to a future post.

Yet Another Alternate Option

In very related news, I did received a reply, although a bit late for me and not really a fit anyway at this point (due to the cost), from when I emailed and asked the guys at the Casting From The Server Room podcast for a reminder of what software they had run across as a Deep Freeze competitor. They mentioned it (CompuGuard CornerStone) in an old episode which I couldn’t remember, and their “show notes wiki” had been lost without a backup. Thanks for the response, guys! Always good to check out alternative options and at least keep abreast of what’s available in the future. They replied to my question back in March on the air, but I missed three episodes in an otherwise unbroken string of probably 30-40 of their episodes I’ve listened to without skipping (wouldn’t you know it was in one of those!), and when I grabbed the back-episode to check out I heard my name again (they’ve mentioned my comments twice in more recent shows since — and inspired the new last name pronunciation guide in my About David page)!

There’s not even that much to tell. The setup was the easiest part: unpack, plug in to power. Place near window for best signal. Plug internet line into the one next to the satellite modem. And that part had been done for us! We primarily tested the existing network using VisiWave to document signal strength, and moved the fourth access point around to various locations to make sure when we order four more, they will cover what we want them to (they will). The VisiWave mapping was the most time-consuming part of the trip (besides waiting for the slow/disconnected internet), but I haven’t had time to pull useful reports out of that data yet.

The Meraki Dashboard is the truly novel and useful tool. You can place your nodes on a map, view how they are interconnected, monitor bandwidth usage and speeds by node and by user, block or whitelist users, set up a splash page, security, and quite a few other nice tweaks that I wouldn’t have thought of but make perfect sense when you see them!

I took a couple of screenshots of the node map overview, using standard and satellite maps:

If you hold your mouse over a node (in the real Dashboard, not these pictures of course! But you knew that…), the route to the internet turns green (one of the gray lines between nodes in the standard map), and some external text shows some additional status information. The number on a node is the number of users in the last 24 hours. These pictures just scratch the surface of the control interface, which is well thought out and feature rich. But that’s all I have time for, so you’ll have to grab some of your own Minis and mess around!

Oh yeah…sorry for the joke in the title. I do love my bad puns…

UPDATE: On Feb. 21st, 2012, after a new comment and response below, I wrote a post that’s a bit of a followup to this one, over at my current (though still infrequently-updated) blog: Ubiquity UniFi vs. Open Mesh.

If they had a second internet connection, “injecting” another point of internet access would be an option, and the network would automatically send traffic to the best internet access point. Thus, the mesh part of mesh networking. I’ve been wanting to try the Meraki products for a while, so I’m excited! More details to come when we’re done!

The trip to the campgrounds is about two hours each way, so we’ll only have three or four hours of actual set up and testing time.

Basically, because we have ISA, I can limit what particular user groups are able to do over their VPN connection, just like any other firewall rules. Very few people get file server access at all (actually, me and one guy who connects from his church laptop) over VPN. The rest are limited to Exchange server connectivity or Remote Desktop primarily, although now that we have RPC over HTTPS in place, it’s much simpler than VPN for the user and so that’s used almost exclusively for remote Outlook access now, and is as much as most people need (if they have a laptop they have an offline copy of most of their files anyway).

For those that still require remote access to their desktop at work (especially if they don’t have a church-owned laptop), I’ve been moving from VPN with Remote Desktop access (complicated to train someone to use since the connection is separate from the RDP client) to LogMeIn.com for remote access. There’s a free version and a Pro version, with remote printing and file transfer being the main additional features of Pro. The main benefit? It’s easy and just requires a web browser, it’s fast, and not very expensive (with the special we got anyway, or the free version is of course free!). I have run into an issue with a new remote user that hasn’t gotten LogMeIn to work on their own but I haven’t had a chance to troubleshoot this yet (I’m sure it relates to the steps to get the ActiveX or Firefox plugin installed for LogMeIn initially).

We have a Terminal Services server with a handful of user licenses that we use for some volunteers that need remote access from their home computer but don’t have a dedicated desktop at work. I haven’t attempted LogMeIn through Terminal Services, but I assume it wouldn’t work properly with the multiple sessions that make Terminal Services useful, and would only allow access to the console session. For this, we still use VPN, with a CD created from the CMAK along with an auto-running tutorial created with Wink that walks users through installing the VPN connectoid (which has all of the settings preset) and starting a VPN connection. Using custom commands in the CMAK connectoid, I’ve included a Remote Desktop settings file that automatically runs upon connection, automatically opening and connecting to the Terminal Server inside the VPN once it’s connected. When Remote Desktop is closed, the connectoid logs off the VPN. The integration of VPN and Remote Desktop isn’t perfect, but it’s a lot easier this way (most of the time) than trying to get people to understand connecting to the VPN first, then connecting with Remote Desktop manually, and disconnecting in reverse. The more automated the better! These VPN connections are of course limited through ISA to be allowed to connect only to the Terminal Server, and only through the RDP protocol.

One thing’s for sure: when allowing an unmanaged computer on the network, especially as unsupervised as a remote connection is, it pays from a security standpoint to keep the leash as tight as possible! And it’s the unintentional risks (spyware, viruses, etc.) more often than malicious users that cause a problem. The best part is, protecting from one helps to protect from the other (in general).

Right now, content filtering on the public wireless is being provided by ScrubIT, a free DNS-based filtering service. Not bad but not as much control or information as I want; it’s a temporary solution (and I haven’t been given an account at ScrubIT yet, so I have no control at all). Matthew Irvine has a couple of excellent posts on his new blog, techlesia, talking about the open source SmoothWall Express firewall and DansGuardian content filter. I have a bit of Linux experience, dabbling at best, but not anything extensive enough for me to set up DansGuardian on a production machine, although I might play with it virtually (SmoothWall Express, if we needed a firewall, might be an option since it is plug-and-play, but we already have ISA 2004). The company SmoothWall has a commercial version of both products, with the content filter called Corporate Guardian, and from the preliminary pricing I’ve found it appears to be much, much less expensive than most of the commercial filtering boxes I’ve researched so far, which translates into “actually affordable.”

I think the Corporate Guardian looks the most promising, since they turn DansGuardian into a commercially-supported product, with the main benefit being that it’s plug-and-play, in addition to blacklist and updates subscriptions. Everyone wins. However, their evaluation terms concern me a bit. The terms state, in part, “You may not communicate the results of your evaluation with other companies, organizations or persons not employed by your company or organization, unless this has been agreed in writing beforehand with SmoothWall.” They also state that after the evaluation, you will “Not make public any notes, analyses, computations, studies or other documents prepared as part of this evaluation unless this has been agreed in writing beforehand with SmoothWall.”

Why does this concern me? Well, I want to share my findings with you on this blog, and these terms say I have to get their permission first. This seems to run counter to the company’s open source products philosophy, and makes me think they are scared of how their product compares to other similar products if someone were to write a review on their blog, for instance. Sure, I could ask for permission to write a review, but if it’s not positive, why would they let me post it? They can do what they want, but I’m not very happy with these particular terms and I’m seriously debating whether or not it’s worth giving up my ability to comment on my findings in order to evaluate the software beyond the claims they make on their website. Is anyone else using SmoothWall’s commercial products, and if so, are you limited in your ability to comment on your company’s use of the products similar to the terms of the evaluation terms, or does that clause go away after you’ve made the purchase?

Thanks Matthew for getting me started on this particular content filter! If I can get past the terms above I’m willing to give it a shot and maybe save some serious money in the process. Or I may find that the open source versions are functional enough and easy enough to set up for my needs; now I just have to find the time to test it.

But, with 802.1x, there’s the possibility of making the ports automatically members of the public VLAN for free access. But when a computer connects that can authenticate via 802.1x, it can bump them onto the employee VLAN. Sweet. But I need to do some manual-reading and testing on our ProCurve switches. Is it worth the effort? Is the Windows XP SP2 802.1x supplicant good enough, or would we need to pay for a third party supplicant? I’ve noticed that for wireless, the Windows 802.1x supplicant seems to be much better that it was originally, and most laptops are coming with even better software built-in from the manufacturer. A year or two ago, I wouldn’t implement an 802.1x-based network with the Windows XP client if you paid me. Well, depends on how much, but it would hurt anyway…

I’d love to put the Gigabit or even 10/100 regular managed switches with all the SNMP and other “big network” goodness including 802.1X security and all the bells and whistles everywhere I need an extra port, but I’m probably going to have to pick up at least one 1700-series switch in the very near future to pick up the slack until the additional home runs I want become a reality. At least I’m still getting a lifetime warranty, and I’ve never had a single problem with any of the ProCurve switches I’ve purchased (not so with Linksys!). Even the 408 switches that I’ve purchased for the same purpose (but unmanaged) in the past have been rock-solid, and they replaced some pretty flaky Linksys workgroup switches!