I’m in the middle of migrating from our VPN3k to using our ASA’s to terminate the VPN connections.  Everything was going along smoothly until I noticed that I couldn’t authenticate using an account in a specific group on our ACS server.  I did a RADIUS debug and some Googling and discovered that the VPN3k uses wildcard masking in its downloadable ACLs, while the ASA expects ACLs to use VLSM masks.

After a few minutes of looking through the multiple groups on the ACS server I discovered that lots of them had downloadable ACLs and I figured I was going to have to re-write them all as I migrated those users over to the ASA.  I took a few aspirin to quell the building headache and then started doing some keyword searches and found a way around all that extra work.  A lovely little command called acl-netmask-convert.

The default value for this command is “standard”, but after changing it to “wildcard” I was able to authenticate without issue.  I then tested the efficacy of the downloadable ACL and found it worked the same as it did with the VPN3k.

Thank you Cisco!  :0)

A few days ago I was writing a regular expression for a string match on one of the ASAs at work.  It should have been a simple exercise as all I wanted to do was block files with the “.lnk” extension via http inspection.  After perusing my notes, I came up with this: “\.[Ll][Nn][Kk]$”.  I thought to my self, “Great! That should take care of the file no matter what case letters are used.”  How wrong I was.

I attempted to test the regex using the fantastic test feature in the ASA.  I checked it against “file.lnk” and was quite surprised when the message came up “match failed”.  After much searching through the ASA documentation pertaining to regular expressions I noticed that while there was mention of the beginning of line anchor “^”, the end of line anchor “$” was missing.  Figuring it must be an oversight in the documentation, I checked the documentation of the IPS, which also supports regex string matching, and sure enough, there was the “$” fully documented.  I tried several variations of the regex and finally achieved a partial success by leaving off the end line anchor.  While this did match “file.lnk”, it also matched “www.lnk.mydomain.com”.  I worked with some of the folks in our Systems department and they assured me that the way I had originally written the regex string should have worked.

With no success finding an answer via web search, I posted on the Cisco Support Community forums.  Within a few hours I got a reply from a TAC engineer who relayed the following to me: “The ASA/PIX Regex engine does not support the end of string anchor ($). Bug CSCsm89915 has been filed as an enhancement to get this functionality into the regex engine. Unfortunately there has been no movement on that enhancement sine 2008.”

I am at a loss as to why something that one would think it would a relatively minor fix like this has yet to be corrected.  Regular expressions are very powerful and highly useful tools.  I realize that I can work around this by adding additional “match not” statements and additional regex strings, but this really shouldn’t be necessary.

I guess I’ll email our Cisco account rep and ask him to drive the bug as was suggested by the TAC engineer.

Quite often changes made to network devices need to be done after hours so as to minimize the impact of potential network downtime.  Time frames for maintenance windows vary from company to company with most falling between 1:00 AM and 5:00 AM.

Here are some general rules to maintenance window work:

1. Plan and document the changes that need to be made.  This may be as simple as writing out access list changes or making sure you have all the tools needed to replace a hardware module.  Planning for major changes can be stressful so once you have a written plan (often called a scope of work or SOW) have someone else look it over to make sure you haven’t missed anything obvious.  You will also want a back out plan to roll back changes if the task is unsuccessful or is running long and will need to be rescheduled due to unforeseen difficulties.
2. Submit your plan for approval.  Usually the person doing the work is not a supervisor and will need permission before the work can be done.
3. Schedule you work.  Many larger companies have a group dedicated to 24X7 monitoring that keeps a calendar of work scheduled for maintenance windows.  They will add your work to the calendar after checking to see if it will interfere with any other scheduled work.  Typically you would contact them before you start and after the change is complete.  If this is a major project the monitoring group may require a conference bridge open so that interested parties can check on the status of the work in real time.  This is more typical in troubleshooting situations.
4. Back up the configs!  Even if you work is routine and has a miniscule chance of disaster, it pays to copy the device configuration files to a dedicated backup server or even to your laptop.  You don’t want to be trying to remember all the static routes on your router at 2:00 AM.
5. Perform the work.  If you run into a stumbling block it often pays to take a short break and get up and walk around to clear your head.  A few months back I was doing reorganizing and consolodating several DMZ’s on our ASA firewall and I couldn’t pass any traffic on one of them.  After about 30 minutes of troubleshooting I went to the break room and sat down for a soda.  I started writing out what I had done and comparing it to my plan and noticed there was a minor difference in one of the access lists.  After correcting it traffic flow resumed.
6. Roll back changes (if necessary).  Put everything back to the way it was.
7. Test, test, test.  Make sure your changes or roll back works as expected.  Fix or roll back as needed.
8. Complete the change.  This may entail reporting to the monitoring group or closing an electronic ticket.

The above rules should give a good idea of how the work should flow in a maintenance window.  What many rookies overlook is the needed to be physically as well as mentally prepared.

1. Unless the overnight is your normal shift, chances are you’re running on less than normal amounts of sleep.  Try to take a short nap before you have to begin.  Everyone is different, but I usually try to get a 1-2 hour nap after work if I have work early the next morning.
2. Caffeine is your best friend in a maintenance window.  Having something caffeinated an hour before you begin work helps to get your sleep-deprived brain into gear.  If I have to travel to our main office (1.5 hours away) I take along a couple of energy drinks just in case.
3. Keep the distractions to a minimum and concentrate on your task.  Never mind updating your Facebook status… do what you came to do and go home to get some sleep.
4. If things run long and you are too tired to drive find a place to take power nap.  Set a timer for 20 minutes and lay down on the lobby couch or, as long as you are in a safe location, in your locked car.  It doesn’t seem like a long time, but it’s well documented that 20 minutes is generally enough to get you focused.  Whatever you do, don’t drive tired!

Have you got a maintenance window ritual that you swear by?  Share it with us!