Surviving after-hours support


I like to joke that DBA stands for ‘Doing Business After-hours’ not Database Administrator. If you have been a DBA long-enough, you have found yourself in the unenviable position of having to deploy production code after business hours or even having to respond to frantic zero-dark-thirty calls for tier 1 environment issues ranging from performance problems to full-blown disaster recovery.

Surviving and even thriving in these type of situations is all a matter of planning and common sense.  I will explain using two examples.

Performing planned after-hours work:

In these situations, practicing the tasks is the best way to prevent surprises.  Since this work is planned, you should have the opportunity to practice in a non-production environment.  Even if a non-production environment is not available, creating a test environment using Oracle VirtualBox is an easy way create a similar environment to test.  Document the process you are going to follow, especially if you have to wake up to perform these changes.  Having the practiced steps laid out helps when you’re half asleep.

Middle of the night panic calls:

The first thing to remember in a panic situation is don’t panic.  Good decisions are seldom made in a panicked state of mind.  I know it it easy to get swept up in a panic situation, so you should start by asking questions, which will hopefully start others thinking about the situation more clearly.  Getting as much information up front helps the decision process.  Another useful tool is to have your environments documented ahead of time so that you can refer to this information easily.  In the event of a disaster recovery,  you will be thankful if you have a tested and documented recovery process.

Additional Tips

In the event of a middle of the night panic or assistance call, don’t immediately reach for the caffeine.  I reach for Gatorade first to get me hydrated while my thought processes ramp up.  Having coffee or a power drink immediately may keep you from getting back to sleep in the event that the assistance requested turns out to be brief.  If it looks like it is going to be a long night, then reach for the power drink or coffee.

Make sure you can connect to your environments remotely and that your connection is reliable and quick.  Ensure you can access connection account and password information readily.

Keep informed about any after-hours work in your environments, even if it does not involve you.  Attend deployment meetings and stay familiar with the deployment calendar.  Problems can arise when bad code is deployed or unexpected results are seen.