Quite often changes made to network devices need to be done after hours so as to minimize the impact of potential network downtime.  Time frames for maintenance windows vary from company to company with most falling between 1:00 AM and 5:00 AM.

Here are some general rules to maintenance window work:

1. Plan and document the changes that need to be made.  This may be as simple as writing out access list changes or making sure you have all the tools needed to replace a hardware module.  Planning for major changes can be stressful so once you have a written plan (often called a scope of work or SOW) have someone else look it over to make sure you haven’t missed anything obvious.  You will also want a back out plan to roll back changes if the task is unsuccessful or is running long and will need to be rescheduled due to unforeseen difficulties.
2. Submit your plan for approval.  Usually the person doing the work is not a supervisor and will need permission before the work can be done.
3. Schedule you work.  Many larger companies have a group dedicated to 24X7 monitoring that keeps a calendar of work scheduled for maintenance windows.  They will add your work to the calendar after checking to see if it will interfere with any other scheduled work.  Typically you would contact them before you start and after the change is complete.  If this is a major project the monitoring group may require a conference bridge open so that interested parties can check on the status of the work in real time.  This is more typical in troubleshooting situations.
4. Back up the configs!  Even if you work is routine and has a miniscule chance of disaster, it pays to copy the device configuration files to a dedicated backup server or even to your laptop.  You don’t want to be trying to remember all the static routes on your router at 2:00 AM.
5. Perform the work.  If you run into a stumbling block it often pays to take a short break and get up and walk around to clear your head.  A few months back I was doing reorganizing and consolodating several DMZ’s on our ASA firewall and I couldn’t pass any traffic on one of them.  After about 30 minutes of troubleshooting I went to the break room and sat down for a soda.  I started writing out what I had done and comparing it to my plan and noticed there was a minor difference in one of the access lists.  After correcting it traffic flow resumed.
6. Roll back changes (if necessary).  Put everything back to the way it was.
7. Test, test, test.  Make sure your changes or roll back works as expected.  Fix or roll back as needed.
8. Complete the change.  This may entail reporting to the monitoring group or closing an electronic ticket.

The above rules should give a good idea of how the work should flow in a maintenance window.  What many rookies overlook is the needed to be physically as well as mentally prepared.

1. Unless the overnight is your normal shift, chances are you’re running on less than normal amounts of sleep.  Try to take a short nap before you have to begin.  Everyone is different, but I usually try to get a 1-2 hour nap after work if I have work early the next morning.
2. Caffeine is your best friend in a maintenance window.  Having something caffeinated an hour before you begin work helps to get your sleep-deprived brain into gear.  If I have to travel to our main office (1.5 hours away) I take along a couple of energy drinks just in case.
3. Keep the distractions to a minimum and concentrate on your task.  Never mind updating your Facebook status… do what you came to do and go home to get some sleep.
4. If things run long and you are too tired to drive find a place to take power nap.  Set a timer for 20 minutes and lay down on the lobby couch or, as long as you are in a safe location, in your locked car.  It doesn’t seem like a long time, but it’s well documented that 20 minutes is generally enough to get you focused.  Whatever you do, don’t drive tired!

Have you got a maintenance window ritual that you swear by?  Share it with us!

I perform a wide variety of duties at my job and I’m constantly taking notes in MS  OneNote of the things I learn to make my job easier.  As I’m the only LAN engineer, I don’t get to share these with anyone at work, so I thought I’d pass them along to the community at large.  If you have a different solution to the same problem, please let me know so I can add it to my repertoire!  🙂