Keep your Oracle binaries!

I am a pack rat.  I admit it.  I rarely throw away old software or IT books.  I still have Oracle version 6 books.  The last time I got rid of software was when I threw out 3 boxes of 3.5″ floppy drives.  I still have old versions of Windows and other software on CD, though.  Not DVD, but CD.  Anyone need a copy of Windows NT 3.51?

My point is that we typically don’t think about preserving Oracle installation media files after the installation is concluded.  More often they are removed to reclaim space in a staging area with the thought that we can always download them again.

After the installation, flash forward to the point where upgrades have not occurred and the hardware is starting to show its age.  Then, just before the work day begins, a failure on a legacy storage appliance causes all the production databases to go down.

If you are lucky, there is a disaster recovery plan in place.  You are even luckier if it has been tested.  At the end of the failure, there are going to be lessons learned and plan revisions.

I recently experienced just such an occurrence at a client site.  A legacy storage appliance did fail, taking the production databases with it.  You are probably thinking that something like this very rarely, if ever, happens but I have seen network storage appliance failure of this kind twice within the last five years at different places and with different storage appliance makers.

After the initial smoke cleared, three options for recovery were available.  In order of preference, they were recover the storage appliance, rebuild servers and databases and recover from backups, and fail over to a disaster recovery site.

While the first option was in progress, it was discovered that the disaster recovery site option would only bring designated critical systems back online, but not every production database.  It was further discovered that this option would only allow fail over and not fail back and not allow running systems from sites other than the DR site.  As such, this option was abandoned.

This left the option of rebuilding servers and databases and recovering from backups while the storage appliance was being worked on.  Unfortunately, a recent acquisition had brought many databases running on unsupported versions of Oracle into the environment.  Now there was a scramble to secure the installation media for these versions.

Installation media is available through Oracle Technology Network ( or Oracle Software Delivery Cloud (  However, as of the writing of this blog, versions 11g release 1 and 10g release 2 are no longer available for direct download.  You can request previous versions of the installation media using the instructions in My Oracle Support (MOS) note 1071023.1.  Oracle provides this statement (links included) on the Oracle Technology Network site when viewing Oracle Database software downloads:

Oracle Database 10.2 and 11.1 are no longer available for download. The software is available as a media or FTP request for those customers who own a valid Oracle Database product license for any edition. To request access to these releases, follow the instructions in Oracle Support Document 1071023.1 (Requesting Physical Shipment or Download URL for Software Media) from My Oracle Support. NOTE: for Oracle Database 10.2, you should request even if you want to install a later patch set. Once you install you can then apply any 10.2 patch set. Similarly, for 11.1 request which must be applied before installing Patch sets can be downloaded from the Patches and Updates tab on My Oracle Support.

In most of the recent client sites I have been to, virtual machines are the standard, replacing individual servers.  These VMs are hosted on large appliances that allow near instantaneous recovery in the event of a failure.  In non-virtual environments, I have seen backups performed at the OS level, where the Oracle binaries are typically installed.  In either of these cases, it may not be necessary to keep copies of the Oracle installation media.

My recommendation is to keep not only the installation media of every Oracle software product in your environment, but also keep copies of all patches applied.  With storage as cheap and accessible as it is now, there is no  real reason not to.  You never know when you will need it again.

SQL Server Health Check v3

My health check script continues to evolve.

–This is a non-intrusive script written to give an overall informational view of a
–SQL Server 2005-2016(CTP3) server instance for the purpose of first-time review.

–Version 3 additions
–User statistics identification
–License information
–Database data and log size with total

Information gathered
-Node name
-License information
-Product Version, level, and Edition
-Instance parameters
-Start time and up time
-Database listing
-Database uptime
-Database data and log size with total
-Database file size and growth settings
-Is this a Cluster Node?
-Nodes in Cluster
-Is AlwaysOn enabled (2012 and above)?
-AlwaysOn status
-Memory usage per database
-Memory usage of in-memory OLTP tables
-Last backup time per database
-No log backups for FULL or BULK_LOGGED recovery model databases in last 30 days
-Databases with no backups at all in the last 30 says
-Backups for the previous week per database
-Jobs that failed in the last 24 hours
-Missing indexes
-Duplicate indexes
-High index fragmentation check
-Wait stats
-Users and roles
-Job information
-Existing linked server listing
-User statistics identification

use master
———————————————————-Node name

SELECT SERVERPROPERTY(‘ComputerNamePhysicalNetBIOS’) AS [CurrentNodeName];
———————————————————-Product Version, level, and Edition

SELECT SERVERPROPERTY(‘productversion’) AS ProductVersion,
SERVERPROPERTY (‘productlevel’) AS ProductLevel
SERVERPROPERTY (‘edition’) AS Edition;

———————————————————-License information

print N’License information’;


———————————————————-Instance parameters

print N’Instance Parameter’;
FROM sys.configurations
ORDER BY name ;
———————————————————-Database listing

print N’Database list with Status and Recovery Model’;

SELECT substring(name,1,40) AS name, substring(state_desc,1,20) AS STATE,
substring(recovery_model_desc,1,20) AS RECOVERY_MODEL
FROM sys.databases
order by name;

———————————————————-Database startup time

print N’Start time’;

SELECT DATEADD(ms,-sample_ms,GETDATE() )AS StartTime
FROM sys.dm_io_virtual_file_stats(1,1);
———————————————————-Database start time uptime

print N’Up time’;
DECLARE @server_start_time DATETIME,
@seconds_diff INT,
@years_online INT,
@days_online INT,
@hours_online INT,
@minutes_online INT,
@seconds_online INT ;

SELECT @server_start_time = login_time
FROM master.sys.sysprocesses
WHERE spid = 1 ;

SELECT @seconds_diff = DATEDIFF(SECOND, @server_start_time, GETDATE()),
@years_online = @seconds_diff / 31536000,
@seconds_diff = @seconds_diff % 31536000,
@days_online = @seconds_diff / 86400,
@seconds_diff = @seconds_diff % 86400,
@hours_online = @seconds_diff / 3600,
@seconds_diff = @seconds_diff % 3600,
@minutes_online = @seconds_diff / 60,
@seconds_online = @seconds_diff % 60 ;

SELECT @server_start_time AS server_start_time,
@years_online AS years_online,
@days_online AS days_online,
@hours_online AS hours_online,
@minutes_online AS minutes_online,
@seconds_online AS seconds_online ;
SELECT substring(name,1,40) AS name, substring(state_desc,1,20) AS STATE,
substring(recovery_model_desc,1,20) AS RECOVERY_MODEL
FROM sys.databases
order by name;

———————————————————-Database data and log size with total

print N’Data and Log Size with Total’;
with fs
select database_id, type, size * 8.0 / 1024 size
from sys.master_files
(select sum(size) from fs where type = 0 and fs.database_id = db.database_id) DataFileSizeMB,
(select sum(size) from fs where type = 1 and fs.database_id = db.database_id) LogFileSizeMB,
(select sum(size) from fs where type = 0 and fs.database_id = db.database_id) +
(select sum(size) from fs where type = 1 and fs.database_id = db.database_id) AS Total_MB
from sys.databases db;

–Grand total of ALL data and log files as one value
select SUM(size*8.0)/1024 AS Total_MB
from sys.master_files;
———————————————————-Database file size and growth settings

print N’Size and Growth’;
select substring(,1,40) AS DB_Name, substring(,1,40) AS Logical_name,
substring(a.filename,1,100) AS File_Name,
cast((a.size * 8.00) / 1024 as numeric(12,2)) as DB_Size_in_MB,
case when a.growth > 100 then ‘In MB’ else ‘In Percentage’ end File_Growth,
cast(case when a.growth > 100 then (a.growth * 8.00) / 1024
else (((a.size * a.growth) / 100) * 8.00) / 1024
end as numeric(12,2)) File_Growth_Size_in_MB,
case when ( maxsize = -1 or maxsize=268435456 ) then ‘AutoGrowth Not Restricted’ else ‘AutoGrowth Restricted’ end AutoGrowth_Status
from sysaltfiles a
join sysdatabases b on a.dbid = b.dbid
order by;
———————————————————-Is this a Cluster Node?

SELECT ‘Clustered’, case when SERVERPROPERTY(‘IsClustered’) = 0 then ‘No’
else ‘Yes’ end;

———————————————————-Nodes in Cluster

print N’Cluster Nodes’;

SELECT * FROM fn_virtualservernodes();
———————————————————-Is AlwaysOn enabled (2012 and above)?

SELECT ‘AlwaysOn’, case when SERVERPROPERTY(‘IsHadrEnabled’) = 0 then ‘No’
when SERVERPROPERTY(‘IsHadrEnabled’) = 1 then ‘Yes’
else SERVERPROPERTY(‘IsHadrEnabled’) end;
———————————————————-AlwaysOn status

declare @c int;
declare @rd nvarchar(60);
declare @osd nvarchar(60);
declare @rhd nvarchar(60);
declare @shd nvarchar(60);
declare @csd nvarchar(60);
select @c = COUNT(name)
from sys.all_objects
where name = ‘dm_hadr_availability_replica_states’;
if @c = 0
print N’No AlwaysOn Status’;
select @rd = role_desc, @osd= case when operational_state_desc is null then ‘Replica is not local’
else operational_state_desc end,
@rhd = recovery_health_desc, @shd = synchronization_health_desc,
@csd = connected_state_desc
from sys.dm_hadr_availability_replica_states;
print @rd
print @osd
print @rhd
print @shd
print @csd
———————————————————-Memory usage per database

print N’Memory Usage per User Database’;
substring(DB_NAME(database_id),1,40) AS [Database Name]
,COUNT(*) * 8/1024.0 AS [Cached Size (MB)]
database_id > 4
AND database_id <> 32767
AND db_name(database_id) <> ‘SSISDB’
GROUP BY DB_NAME(database_id)

———————————————————-Memory usage of in-memory OLTP tables

print N’In-memory OLTP table usage’;
SELECT object_name(object_id) AS Name, *
FROM sys.dm_db_xtp_table_memory_stats;
———————————————————-Last backup time per database

SELECT substring(sdb.Name,1,40) AS DatabaseName,
COALESCE(CONVERT(VARCHAR(12), MAX(bus.backup_finish_date), 101),’-‘) AS LastBackUpTime
FROM sys.sysdatabases sdb
LEFT OUTER JOIN msdb.dbo.backupset bus ON bus.database_name =
where sdb.Name <> ‘tempdb’
GROUP BY sdb.Name;
———————————————————-No log backups for FULL or BULK_LOGGED recovery model databases in last 30 days
print N’Databases with FULL or BULK_LOGGED recovery model and no log backups in last 30 days’;
SELECT name AS at_risk_database
FROM sys.databases
where recovery_model_desc in(‘FULL’,’BULK_LOGGED’)
and name not in(
msdb.dbo.backupset.database_name AS DBName
FROM msdb.dbo.backupmediafamily
INNER JOIN msdb.dbo.backupset ON msdb.dbo.backupmediafamily.media_set_id = msdb.dbo.backupset.media_set_id
WHERE (CONVERT(datetime, msdb.dbo.backupset.backup_start_date, 102) >= GETDATE() – 30)
and msdb..backupset.type = ‘L’
group by msdb.dbo.backupset.database_name

———————————————————-Databases with no backups at all in the last 30 says

print N’Databases with NO backups in last 30 days’;

SELECT name AS at_risk_database
FROM sys.databases
where name <> ‘tempdb’
and name not in(
substring(msdb.dbo.backupset.database_name,1,40) AS DBName
FROM msdb.dbo.backupmediafamily
INNER JOIN msdb.dbo.backupset ON msdb.dbo.backupmediafamily.media_set_id = msdb.dbo.backupset.media_set_id
WHERE (CONVERT(datetime, msdb.dbo.backupset.backup_start_date, 102) >= GETDATE() – 30)
group by msdb.dbo.backupset.database_name
———————————————————-Backups for the previous week per database

print N’All backups for previous week’;

CONVERT(CHAR(40), SERVERPROPERTY(‘Servername’)) AS Server,
substring(msdb.dbo.backupset.database_name,1,40) AS DBName,
CASE msdb..backupset.type
WHEN ‘D’ THEN ‘Database’
WHEN ‘F’ THEN ‘File’
WHEN ‘P’ THEN ‘Partial’
WHEN ‘I’ THEN ‘Differential database’
WHEN ‘G’ THEN ‘Differential file’
WHEN ‘Q’ THEN ‘Differential partial’
WHEN NULL THEN msdb..backupset.type
END AS backup_type,
substring(msdb.dbo.backupmediafamily.logical_device_name,1,50) AS logical_device_name,
substring(msdb.dbo.backupmediafamily.physical_device_name,1,50) AS physical_device_name,
substring(,1,50) AS backupset_name,
substring(msdb.dbo.backupset.description,1,50) AS description
FROM msdb.dbo.backupmediafamily
INNER JOIN msdb.dbo.backupset ON msdb.dbo.backupmediafamily.media_set_id = msdb.dbo.backupset.media_set_id
WHERE (CONVERT(datetime, msdb.dbo.backupset.backup_start_date, 102) >= GETDATE() – 7)
———————————————————-Jobs that failed in the last 24 hours

print N’Jobs Failing in last 24 hours’;

———————————————————-Variable Declarations
DECLARE @PreviousDate datetime
———————————————————-Initialize Variables
SET @PreviousDate = DATEADD(dd, -1, GETDATE()) –Last 1 day
SET @Year = DATEPART(yyyy, @PreviousDate)
SELECT @MonthPre = CONVERT(VARCHAR(2), DATEPART(mm, @PreviousDate))
SELECT @Month = RIGHT(CONVERT(VARCHAR, (@MonthPre + 1000000000)),2)
SELECT @DayPre = CONVERT(VARCHAR(2), DATEPART(dd, @PreviousDate))
SELECT @Day = RIGHT(CONVERT(VARCHAR, (@DayPre + 1000000000)),2)
SET @FinalDate = CAST(@Year + @Month + @Day AS INT)
–Final Logic
SELECT substring(j.[name],1,40) AS JOB,
substring(s.step_name,1,40) AS Step,
substring(h.step_name,1,40) AS Step,
substring(h.message,1,100) AS Message,
FROM msdb.dbo.sysjobhistory h
INNER JOIN msdb.dbo.sysjobs j
ON h.job_id = j.job_id
INNER JOIN msdb.dbo.sysjobsteps s
ON j.job_id = s.job_id AND h.step_id = s.step_id
WHERE h.run_status = 0 –Failure
AND h.run_date > @FinalDate
ORDER BY h.instance_id DESC;

———————————————————-Missing indexes

print N’Missing Indexes’;
SELECT substring(,1,40) AS Name
, (avg_total_user_cost * avg_user_impact) * (user_seeks + user_scans) as Impact
, ddmid.equality_columns
, ddmid.inequality_columns
, ddmid.included_columns
FROM sys.dm_db_missing_index_group_stats AS ddmigs
INNER JOIN sys.dm_db_missing_index_groups AS ddmig
ON ddmigs.group_handle = ddmig.index_group_handle
INNER JOIN sys.dm_db_missing_index_details AS ddmid
ON ddmig.index_handle = ddmid.index_handle
INNER JOIN sys.objects so WITH (nolock)
ON ddmid.object_id = so.object_id
WHERE ddmigs.group_handle IN (
SELECT TOP (5000) group_handle
FROM sys.dm_db_missing_index_group_stats WITH (nolock)
ORDER BY (avg_total_user_cost * avg_user_impact)*(user_seeks+user_scans)DESC);

———————————————————-Duplicate indexes

print N’Duplicate Indexes’;

WITH ind_list AS(
select o.schema_id, i.object_id, i.index_id,, i.type_desc,
i.is_unique, i.is_primary_key,
FROM sys.index_columns ic
JOIN sys.columns tc
ON tc.column_id = ic.column_id AND
tc.object_id = ic.object_id
WHERE ic.object_id = i.object_id AND
ic.index_id = i.index_id
AND ic.is_included_column = 0
ORDER BY ic.index_column_id
FOR XML PATH (”) ),1,1,” ) index_columns,
FROM sys.index_columns ic
JOIN sys.columns tc
ON tc.column_id = ic.column_id AND
tc.object_id = ic.object_id
WHERE ic.object_id = i.object_id AND
ic.index_id = i.index_id
AND ic.is_included_column = 1
ORDER BY ic.index_column_id
FOR XML PATH (”) ),1,1,” ) include_columns
FROM sys.indexes i
JOIN sys.objects o ON o.object_id = i.object_id
WHERE i.index_id > 0 AND i.type_desc <> ‘XML’
AND object_name(i.object_id) LIKE @TABLENAME
AND i.is_disabled = 0
AND schema_name(o.schema_id) LIKE @SCHEMANAME )
SELECT substring(schema_name(included_indexes.schema_id),1,30) AS owner,
object_name(included_indexes.object_id) table_name,
(SELECT SUM(st.row_count) FROM sys.dm_db_partition_stats st
WHERE st.object_id = included_indexes.object_id
AND st.index_id < 2 ) num_rows, included_index_name,
included_indexes.index_columns included_index_columns,
included_indexes.type_desc included_index_type,
included_indexes.is_unique included_index_uniqueness,
included_indexes.is_primary_key included_index_PK,
(SELECT SUM(a.total_pages) * 8 FROM sys.allocation_units a
JOIN sys.partitions p ON a.container_id = p.partition_id
WHERE p.object_id = included_indexes.object_id AND
p.index_id = included_indexes.index_id
) included_index_size_kb, including_index_name,
including_indexes.index_columns including_index_columns,
including_indexes.type_desc including_index_type,
including_indexes.is_unique including_index_uniqueness,
including_indexes.is_primary_key including_index_PK,
(SELECT SUM(a.total_pages) * 8 FROM sys.allocation_units a
JOIN sys.partitions p ON a.container_id = p.partition_id
WHERE p.object_id = including_indexes.object_id AND
p.index_id = including_indexes.index_id
) including_index_size_kb
FROM ind_list included_indexes
JOIN ind_list including_indexes
ON including_indexes.object_id = included_indexes.object_id
JOIN sys.partitions ing_p
ON ing_p.object_id = including_indexes.object_id AND
ing_p.index_id = including_indexes.index_id
JOIN sys.allocation_units ing_a
ON ing_a.container_id = ing_p.partition_id
WHERE including_indexes.index_id <> included_indexes.index_id
AND LEN(included_indexes.index_columns) <=
AND included_indexes.index_columns + ‘,’ =
LEN(included_indexes.index_columns + ‘,’))

———————————————————-High index fragmentation check

print N’Index with HIGH Fragmentation’;

EXEC sp_MSforeachdb ‘
USE [?]
QUOTENAME( AS [index_name],
FROM sys.dm_db_index_physical_stats (DB_ID(), NULL, NULL, NULL, ”LIMITED”)
AS indstat
INNER JOIN sys.indexes sysind ON indstat.object_id = sysind.object_id AND
indstat.index_id = sysind.index_id
where avg_fragmentation_in_percent >= 30
ORDER BY avg_fragmentation_in_percent DESC;

use master
———————————————————-Wait stats

print N’Wait Stats’;

FROM sys.dm_os_wait_stats
where wait_time_ms > 10000
ORDER BY wait_time_ms DESC;
———————————————————-Users and roles

print N’Users and Roles’;

WITH Roles_CTE(Role_Name, Username)
User_Name(sm.[groupuid]) as [Role_Name],
user_name(sm.[memberuid]) as [Username]
FROM [sys].[sysmembers] sm

[DatabaseUserName] = princ.[name],
[UserType] = CASE princ.[type]
WHEN ‘U’ THEN ‘Windows User’
WHEN ‘G’ THEN ‘Windows Group’
WHEN ‘A’ THEN ‘Application Role’
WHEN ‘R’ THEN ‘Database Role’
WHEN ‘C’ THEN ‘User mapped to a certificate’
WHEN ‘K’ THEN ‘User mapped to an asymmetric key’
sys.database_principals princ
JOIN Roles_CTE on Username =
where princ.type in (‘S’, ‘U’, ‘G’, ‘A’, ‘R’, ‘C’, ‘K’)
———————————————————-Job information

print N’Job Information’;
SELECT [JobName] = [jobs].[name]
,[Category] = [categories].[name]
,[Owner] = SUSER_SNAME([jobs].[owner_sid])
,[Enabled] = CASE [jobs].[enabled] WHEN 1 THEN ‘Yes’ ELSE ‘No’ END
,[Scheduled] = CASE [schedule].[enabled] WHEN 1 THEN ‘Yes’ ELSE ‘No’ END
,[Description] = [jobs].[description]
,[Occurs] =
CASE [schedule].[freq_type]
WHEN 1 THEN ‘Once’
WHEN 4 THEN ‘Daily’
WHEN 8 THEN ‘Weekly’
WHEN 16 THEN ‘Monthly’
WHEN 32 THEN ‘Monthly relative’
WHEN 64 THEN ‘When SQL Server Agent starts’
WHEN 128 THEN ‘Start whenever the CPU(s) become idle’
,[Occurs_detail] =
CASE [schedule].[freq_type]
WHEN 4 THEN ‘Every ‘ + CONVERT(VARCHAR, [schedule].[freq_interval]) + ‘ day(s)’
WHEN 8 THEN ‘Every ‘ + CONVERT(VARCHAR, [schedule].[freq_recurrence_factor]) + ‘ weeks(s) on ‘ +
CASE WHEN [schedule].[freq_interval] & 1 = 1 THEN ‘Sunday, ‘ ELSE ” END +
CASE WHEN [schedule].[freq_interval] & 2 = 2 THEN ‘Monday, ‘ ELSE ” END +
CASE WHEN [schedule].[freq_interval] & 4 = 4 THEN ‘Tuesday, ‘ ELSE ” END +
CASE WHEN [schedule].[freq_interval] & 8 = 8 THEN ‘Wednesday, ‘ ELSE ” END +
CASE WHEN [schedule].[freq_interval] & 16 = 16 THEN ‘Thursday, ‘ ELSE ” END +
CASE WHEN [schedule].[freq_interval] & 32 = 32 THEN ‘Friday, ‘ ELSE ” END +
CASE WHEN [schedule].[freq_interval] & 64 = 64 THEN ‘Saturday, ‘ ELSE ” END ,
CASE WHEN [schedule].[freq_interval] & 1 = 1 THEN ‘Sunday, ‘ ELSE ” END +
CASE WHEN [schedule].[freq_interval] & 2 = 2 THEN ‘Monday, ‘ ELSE ” END +
CASE WHEN [schedule].[freq_interval] & 4 = 4 THEN ‘Tuesday, ‘ ELSE ” END +
CASE WHEN [schedule].[freq_interval] & 8 = 8 THEN ‘Wednesday, ‘ ELSE ” END +
CASE WHEN [schedule].[freq_interval] & 16 = 16 THEN ‘Thursday, ‘ ELSE ” END +
CASE WHEN [schedule].[freq_interval] & 32 = 32 THEN ‘Friday, ‘ ELSE ” END +
CASE WHEN [schedule].[freq_interval] & 64 = 64 THEN ‘Saturday, ‘ ELSE ” END
) – 1
WHEN 16 THEN ‘Day ‘ + CONVERT(VARCHAR, [schedule].[freq_interval]) + ‘ of every ‘ + CONVERT(VARCHAR, [schedule].[freq_recurrence_factor]) + ‘ month(s)’
WHEN 32 THEN ‘The ‘ +
CASE [schedule].[freq_relative_interval]
WHEN 1 THEN ‘First’
WHEN 2 THEN ‘Second’
WHEN 4 THEN ‘Third’
WHEN 8 THEN ‘Fourth’
WHEN 16 THEN ‘Last’
CASE [schedule].[freq_interval]
WHEN 1 THEN ‘ Sunday’
WHEN 2 THEN ‘ Monday’
WHEN 3 THEN ‘ Tuesday’
WHEN 4 THEN ‘ Wednesday’
WHEN 5 THEN ‘ Thursday’
WHEN 6 THEN ‘ Friday’
WHEN 7 THEN ‘ Saturday’
WHEN 8 THEN ‘ Day’
WHEN 9 THEN ‘ Weekday’
WHEN 10 THEN ‘ Weekend Day’
END + ‘ of every ‘ + CONVERT(VARCHAR, [schedule].[freq_recurrence_factor]) + ‘ month(s)’
,[Frequency] =
CASE [schedule].[freq_subday_type]
WHEN 1 THEN ‘Occurs once at ‘ +
STUFF(STUFF(RIGHT(‘000000’ + CONVERT(VARCHAR(8), [schedule].[active_start_time]), 6), 5, 0, ‘:’), 3, 0, ‘:’)
WHEN 2 THEN ‘Occurs every ‘ +
CONVERT(VARCHAR, [schedule].[freq_subday_interval]) + ‘ Seconds(s) between ‘ +
STUFF(STUFF(RIGHT(‘000000’ + CONVERT(VARCHAR(8), [schedule].[active_start_time]), 6), 5, 0, ‘:’), 3, 0, ‘:’) + ‘ and ‘ +
STUFF(STUFF(RIGHT(‘000000’ + CONVERT(VARCHAR(8), [schedule].[active_end_time]), 6), 5, 0, ‘:’), 3, 0, ‘:’)
WHEN 4 THEN ‘Occurs every ‘ +
CONVERT(VARCHAR, [schedule].[freq_subday_interval]) + ‘ Minute(s) between ‘ +
STUFF(STUFF(RIGHT(‘000000’ + CONVERT(VARCHAR(8), [schedule].[active_start_time]), 6), 5, 0, ‘:’), 3, 0, ‘:’) + ‘ and ‘ +
STUFF(STUFF(RIGHT(‘000000’ + CONVERT(VARCHAR(8), [schedule].[active_end_time]), 6), 5, 0, ‘:’), 3, 0, ‘:’)
WHEN 8 THEN ‘Occurs every ‘ +
CONVERT(VARCHAR, [schedule].[freq_subday_interval]) + ‘ Hour(s) between ‘ +
STUFF(STUFF(RIGHT(‘000000’ + CONVERT(VARCHAR(8), [schedule].[active_start_time]), 6), 5, 0, ‘:’), 3, 0, ‘:’) + ‘ and ‘ +
STUFF(STUFF(RIGHT(‘000000’ + CONVERT(VARCHAR(8), [schedule].[active_end_time]), 6), 5, 0, ‘:’), 3, 0, ‘:’)
,[AvgDurationInSec] = CONVERT(DECIMAL(10, 2), [jobhistory].[AvgDuration])
,[Next_Run_Date] =
CASE [jobschedule].[next_run_date]
ELSE CONVERT(DATETIME, CONVERT(CHAR(8), [jobschedule].[next_run_date], 112) + ‘ ‘ +
STUFF(STUFF(RIGHT(‘000000’ + CONVERT(VARCHAR(8), [jobschedule].[next_run_time]), 6), 5, 0, ‘:’), 3, 0, ‘:’))
FROM [msdb].[dbo].[sysjobs] AS [jobs] WITh(NOLOCK)
LEFT OUTER JOIN [msdb].[dbo].[sysjobschedules] AS [jobschedule] WITh(NOLOCK)
ON [jobs].[job_id] = [jobschedule].[job_id]
LEFT OUTER JOIN [msdb].[dbo].[sysschedules] AS [schedule] WITh(NOLOCK)
ON [jobschedule].[schedule_id] = [schedule].[schedule_id]
INNER JOIN [msdb].[dbo].[syscategories] [categories] WITh(NOLOCK)
ON [jobs].[category_id] = [categories].[category_id]
( SELECT [job_id], [AvgDuration] = (SUM((([run_duration] / 10000 * 3600) +
(([run_duration] % 10000) / 100 * 60) +
([run_duration] % 10000) % 100)) * 1.0) / COUNT([job_id])
FROM [msdb].[dbo].[sysjobhistory] WITh(NOLOCK)
WHERE [step_id] = 0
GROUP BY [job_id]
) AS [jobhistory]
ON [jobhistory].[job_id] = [jobs].[job_id];
———————————————————-Existing linked server listing
print N’Linked Server Information’;
declare @x int;
select @x = COUNT(name)
from sys.all_objects
where name = ‘Servers’;
if @x <> 0
–, provider, data_source, is_remote_login_enabled, b.modify_date
FROM sys.Servers a
LEFT OUTER JOIN sys.linked_logins b ON b.server_id = a.server_id
LEFT OUTER JOIN sys.server_principals c ON c.principal_id = b.local_principal_id
where a.server_id <> 0;
exec sp_linkedservers;
———————————————————-User statistics identification

print N’User statistics’;

FROM sys.stats s
INNER JOIN sys.Objects ob ON ob.Object_id = s.object_id
WHERE SCHEMA_NAME(ob.Schema_id) <> ‘sys’
AND Auto_Created = 0 AND User_Created = 1;

Evaluating a SQL Server environment

This is a topic with more than one meaning, so I will narrow the focus to the aspect of just looking at an existing SQL Server environment for the first time. If you are experiencing performance issues with an existing environment, I recommend you start with the Microsoft (MS) SQL Server performance monitor to concentrate your effort instead of looking at the overall SQL Server. This is especially true if there are only a few problematic processes.

It is good to start asking basic questions for better understanding of the environment to evaluate. A short checklist of the items to inquire about before proceeding includes:

  • Is this a 24×7 environment?
  • What are the peak times of usage and critical availability?
  • Are there any current performance concerns?
  • What is the current backup configuration?
  • When was the last time a recovery was performed? Practice recovery? Cloning using backups?

This information can help you determine the best approach for your evaluation like the best times to execute any of the more resource-intensive examinations. There are many tools to choose from when evaluating a SQL Server environment. Some of these are MS native tools and some are third-party. As a precaution, verify the presence of a recent backup or even execute a one-off full backup before evaluation.

The most recognized MS built-in tool for evaluation is DBCC CHECKDB which is included with every SQL Server version. DBCC CHECKDB is a Transact SQL based utility that checks the logical and physical integrity of all the objects in the specified database. This process also calls other DBCC programs and performs these steps while executing:

  • Runs DBCC CHECKALLOC on the database to check the consistency of disk space allocation structures.
  • Runs DBCC CHECKTABLE on every table and view in the database to check the integrity of all the pages and structures that make table or indexed views.
  • Runs DBCC CHECKCATALOG on the database to check for catalog consistency within the database.
  • Validates the contents of every indexed view in the database.
  • Validates link-level consistency between table metadata and file system directories and files when storing varbinary(max) data in the file system using FILESTREAM.
  • Validates the Service Broker data in the database.

The DBCC CHECKDB utility includes options to repair issues it finds, but I do not recommend the use of these options until any existing issues have been identified and understood. All of these checks and validations can take a long time to execute against a sizeable database, so there are options to exclude one or more of the above steps for time and resource considerations. MS SQL Server versions 2014 through 2017 even offer the ability to execute this utility with a parallel degree. Consult the MS DBCC CHECKDB documentation for the version you are evaluating for more specific information.

Basic execution (from a SQL Server Management Studio prompt):

use <db_name>


dbcc checkdb



Note – For larger database executions, use the “Results To” option under the Query menu item in SQL Server Management Studio to spool the results for later review.

Brent Ozar, a well-regarded SQL Server expert, created a tool called the First Responder Kit, or sp_Blitz for short.   It is available for free from and provides a list of its findings prioritized for immediate action or informational purposes. The lower the number a finding is given, the higher the priority. This tool installs required stored procedures and an output table in the database. However, it is free and you can use parameters at execution to avoid checking certain objects or ignore findings over a certain priority level.

Basic execution (from a SQL Server Management Studio prompt):

After downloading the First Responder’s Kit compressed archive and placing it in an accessible location, execute the sp_blitz.sql file to install the database objects. Then, from a SQL Server command prompt, execute:



Note – By default, this script examines ALL databases present on the current SQL Server and groups the findings by order of priority, sorting by database name.

My own SQL Server Health Check script (, which is also free, may not have the maturity of some other tools, but it does not require any objects to be installed and provides a good overall picture of a database environment. It is a single script that can be executed the same as a standard SQL script.


Backups triggered automatically

Database availability is the number one goal of the database administrator.  Backup and recovery should be the second goal.  An internet search on establishing database backup and recovery processes will produce many examples.  I won’t go into my personal preferences, just emphasize the points I think are important:

  • Backup and recovery processes MUST be documented for reference by personnel responsible for maintenance and execution. I find that the lack of backup and recovery documentation is fairly common out in the world. No DBA likes to try to do an undocumented recovery at 3AM.
  • Backups processes that do not include a check for successful completion are INVALID. If the backup has been completing with an error for a month then guess what, you last valid backup was a month ago and if you had to do a recovery today, you just lost a month’s worth of data.
  • Recovery processes that have not been tested by performing a practice recovery or a clone are INVALID. You may have a proven backup method, but it has not been proven until you do a recovery in your own environment.
  • If there is available time and resources, add an export to your backup process. Exports are good for granular recovery of metadata and objects such as indexes and PL/SQL code.

That said, even solid backup processes can get blind-sided by unexpected activity in the database that produces vastly more redo activity then is normally seen.  It only takes one instance of the dreaded “ORA-00257: archiver error” to send personnel from support through management looking for or demanding answers and remediation steps.

The work around for these situations is usually increasing either the amount of space in the archived log destination or increasing the frequency of backups.  While either of these are fairly easy to implement, they each have areas of concern that should be considered prior to implementation.

Increasing the amount of space in the archived log location could be costly based on amount and type of storage used.  Also, if the surge of redo is not a common occurrence, all the additional space will be unused, essentially wasted.  Increasing the backup frequency may not cost in terms of additional storage, but in database resources needed to perform the additional backups, especially when they occur during normal business hours.

When a database is configured to use the fast recovery area (FRA) for backups and flashback, Oracle automatically monitors and alerts on its space usage.  Oracle 11.2 and 12.2 documentation both have the same information on these alerts:

The database issues a warning alert when reclaimable space is 
less than 15% and a critical alert when reclaimable space is 
less than 3%. To warn the DBA of this condition, an entry is 
added to the alert log and to the DBA_OUTSTANDING_ALERTS table 
(used by Enterprise Manager). Nevertheless, the database 
continues to consume space in the fast recovery area until 
there is no reclaimable space left.

When the recovery area is completely full, the error displayed 
is as follows, where nnnnn is the number of bytes required and 
mmmmm is the disk quota:

ORA-19809: limit exceeded for recovery files ORA-19804: cannot 
reclaim nnnnn bytes disk space from mmmmm limit

From <>
The database issues a warning alert when reclaimable space is 
less than 15% and a critical alert when reclaimable space is 
less than 3%. To warn the DBA of this condition, an entry is 
added to the alert log and to the DBA_OUTSTANDING_ALERTS table 
(used by Enterprise Manager). Nevertheless, the database 
continues to consume space in the fast recovery area until 
there is no reclaimable space left.

When the recovery area is completely full, the error displayed 
is as follows, where nnnnn is the number of bytes required and 
mmmmm is the disk quota:

ORA-19809: limit exceeded for recovery files ORA-19804: cannot 
reclaim nnnnn bytes disk space from mmmmm limit

From <>

A practical database administrator should already be monitoring the alert log for errors and warnings.  Why not configure these monitoring processes to automatically trigger backups if this error is encountered?  One of my favorite alert log monitor scripts sends an email when an ORA- error is found in the alert log and sends a page or text message to designated parties if the error involves some type of corruption:


ERRMSG="Missing command line parameters"

if [ "$1" ]; then
        export ORACLE_SID=$1
        echo $ERRMSG
        exit 1

if [ "$2" ]; then
        export ALERT_LOG_DIR=$2
        echo $ERRMSG
        exit 1

            CHECK_TIMEB="`date +%m/%d/%y-%H:%M`"
            SAVFILE="${ALERT_LOG_DIR}/alert_${ORACLE_SID}_`date '+%y%m%d-%H%M'`.log"
            echo "  "
            echo "**"
            echo "**** $ORACLE_SID"
            echo "**"
            echo " "
            if [ -f $LOGFILE ]; then
               if [ `egrep -i "ORA-|corrupt" "${LOGFILE}" | wc -l` -eq 0 ]; then
                  echo "No errors found in alert log "
                  egrep -i "ORA-|corrupt" $LOGFILE > $ERRFILE
                  echo " " >> $ERRFILE
                  echo "  Found the above error(s) in the alert log for ${ORACLE_SID}" >> $ERRFILE
                  echo "  Alert Log with error(s) saved as ${SAVFILE}" >> $ERRFILE
                  echo " " >> $ERRFILE
                  echo " " >> $ERRFILE
                  mailx -s "ALERT-Errors found in ${ORACLE_SID} Alert Log" < $ERRFILE
                  if [ `egrep -i "corrupt" "${ERRFILE}" | wc -l` -gt 0 ]; then
                  mv $LOGFILE $SAVFILE
               echo "No alert log found for ${ORACLE_SID} "

You could easily add a line to call a separate backup script from this script.  I usually set my alert log monitor to execute once an hour, but increasing the frequency of execution will help catch redo space problems and not cost in terms of resource or space usage.  Also,  having an automatically-triggered backup will save time diagnosing the issue when an archiver ‘out of space’ problem occurs.

The drawback here is that you must have FRA enabled to utilize the alert functionality.  What if usage of the FRA is not preferred?  You could use the alert log monitor script above to trigger a backup when an archiver error is seen, but that would mean the database is already in the process of halting activity when the backup is triggered.  It is better to be pre-emptive in these types of alerts by setting up monitoring to alert of a problem before an alert log message is generated.

For example, here is a snippet of code for a Linux/Unix shell script that determines how much space is available in GB for a certain filesystem.  If that amount of space falls below a certain value, a message is generated:


space_avail="`df -k | grep /archive | awk '{print $4}'`"

if [ `expr $space_avail / 1048576` -lt 40  ]; then
echo Backup needed

You could easily replace the echo command with a call to a backup script.  After testing, this script could be called at an interval of your choosing from the crontab.

If you have your backup location in ASM, you can use this next piece of code to accomplish the same thing by querying the desired diskgroup:


free_mb=`$CRS_HOME/bin/sqlplus -s "/ as sysasm"<<EOF
set heading off
select free_mb
from v\\$asm_diskgroup
where name = 'DATA'
order by name;

if [ `expr $free_mb / 1024` -lt 40  ]; then
echo Backup needed

With a few simple steps, you can be proactive about excessive redo activity in your database.

To output in real time or not?

Just yesterday I encountered a wrong conclusion based on a misunderstanding about the use of the dbms_output.put_line function.  The writer of a SQL script that spooled its output to a log file used the function to generate ‘start’ and ‘end’ messages within a PL/SQL block and was confused when the ‘start’ message was not seen in the log file while the main section of the PL/SQL block was executing.  He was under the assumption that calls to this function were carried out immediately.

This is a common misconception regarding dbms_output.put_line.  This example of the same functionality they were attempting reinforces this conclusion.

set serveroutput on

spool ./output.log


  dbms_output.put_line('Start - '||to_char(sysdate,'MM/DD/YYYY HH24:MI:SS'));

  execute immediate 'select * from tab';


  dbms_output.put_line('End - '||to_char(sysdate,'MM/DD/YYYY HH24:MI:SS'));

spool off

In this example, the spooling of the output is started inside the PL/SQL block.  This code will execute for about 30 seconds because of the dbms_lock.sleep function call.  During execution, the output.log file will be empty until the code finishes, either successfully or unsuccessfully, and ‘spool off’ is executed.  You can verify this with a ‘tail -f’ command in Linux/Unix or the ‘type’ command in Windows.

What happens if you place the ‘start’ and ‘end’ messages outside the PL/SQL block?

set serveroutput on

spool ./output.log

select 'Start - '||to_char(sysdate,'MM/DD/YYYY HH24:MI:SS')
from dual;


  execute immediate 'select * from tab';


select 'End - '||to_char(sysdate,'MM/DD/YYYY HH24:MI:SS')
from dual;

spool off

Even though you see the ‘start’ message at the onset in your SQL session, the log file will still not be populated until the PL/SQL block execution is finished and the ‘spool off’ command runs.

If you truly want to see the ‘start’ message before ANY code executes, then using OS commands to start spooling a log file and generate the ‘start’ message is probably your only alternative.  Here is an example for a Linux-based system:

echo "Start - $(date)" > ./output.log

This command, executed before the script is called, will create the log file and write a ‘start’ message to the file with the current date and time in the current directory.  This information will show up in the log file immediately.   Next, call the script to be executed:

$ORACLE_HOME/bin/sqlplus -S username/password @script.sql >> ./output.log

This command will start a SQL*Plus session in silent mode (-S), suppressing header and prompt output, call the script.sql script, and send the output to the log file opened earlier.  Using this method, you don’t need to use any ‘spool’ commands in the script.  Notice I used “>>” to redirect the output in this command and not “>” for the command when I opened the log file.  The “>>” redirect appends to an existing file, opening it if it does not exist.  If you use “>” for the redirect, an existing file will be overwritten and you will lose any information previously written to the file.

For a Windows-based system, the command is almost the same:

echo Start - %date% %time% > ./output.log

This command, executed before the script is called, will create the log file and write a ‘start’ message to it with the date and time in the current directory.  This information will show up in the log file immediately.  The “>>” and “>” redirect options even work the same as a Linux/Unix environment.

From here there a few different ways to execute a SQL script from a Windows command prompt.  If this is going to be an ongoing process in Windows, then I recommend writing the scripts in Perl.  It will save time, especially if you have to port the code between Windows and Linux/Unix environments.

One last point.  You should not use this ‘start’ and ‘end’ message method in a production system.  While it is fine for time tracking while testing the script, it should not be relied on outside of a test environment.  Even when these messages are sent to a log file, that file needs to be parsed for error messages at the conclusion of the process.  A better solution is to send the messages to a table used for auditing and monitoring.  Just don’t forget to commit each time you send a message to the table and remember to purge the information from the table based on audit requirements.



Get in the Penalty Box! (Redux version)

Note – The commands and processes listed have been tested and verified in a Oracle 11g release 2 ( environment.

Sometimes, there exist database users with no regard for what type of queries they run or how long they run for.  These users typically kick off these ill-behaved queries and then go for coffee, lunch, or leave for the day.  Their query sits out there spinning and consuming resources that you know could be better used elsewhere.  This can be especially problematic in a production system if end-of-the-month aggregation processes or reports are being executed at the same time.

Maybe you have talked to them before about this behavior or even helped them tune a query or two, but they don’t seem to get the message.  In extreme cases, these users even terminate sessions when the query does not come back in a timely manner only to reconnect and retry it in a new session, leaving the old one (possibly) still executing.

It’s time to let these users know that their behavior is not acceptable in an environment where resources are finite and they need to share with other higher priority processes.  They need to spend some time in the penalty box.  For those unfamiliar with this term, it is a term used in ice hockey where misbehaving players have to sit out the game in a segregated area for a certain length of time, depending on the seriousness of their misdeeds.  The team with the misbehaving player is not allowed to send in a substitute player during the penalty time, so it leaves them one player short and vulnerable to be scored against.

The Oracle Resource Manager is the keeper of the database penalty box.  It has the ability to enforce limits on resources such CPU and I/O consumption, as well as execution time limits.  We are going to use this last feature to create our own penalty box.  I realize that most database managers don’t want to put constraints on users, especially developers, but in all honesty, if they are writing queries that take an hour or more to execute on a regular basis, then the managers need to put their foot down and the users or developers may need a remedial SQL tuning course.

A common misconception about resource plans is that they will limit all users in the database.  While it is true that only one resource plan can be active at any time, users can be segregated into consumer groups under the plan and each consumer group can be tailored with the use of directives to make sure that users or applications do not suffer due to the actions of a few users.

Here is a simple process to create a resource plan that will limit execution time to 60 seconds.  This limit is just for testing purposes as even I have been known to write queries that run for over a minute ;).  Administering the resource manager requires the ADMINISTER_RESOURCE_MANAGER privilege, go figure.  A SYSDBA user has this privilege already, but if you need to assign the privilege to a non-SYSDBA user, you can use this command (as a properly privileged user):

exec dbms_resource_manager_privs.grant_system_privilege( -

grantee_name => '<USER_NAME>', -

admin_option => true);

Check for a currently active resource plan:

SQL> show parameter resource_manager_plan

NAME                     TYPE     VALUE

-------------------------- ----------- ------------------------------

resource_manager_plan   string

Since no resource plan is active, we can continue with the rest of the penalty box steps to create one.  Later on, we will look at our options if there is an actual resource plan already in place.

Create a new pending area for plan, consumer group, and directives creation.  The pending area is where our resource plan work will be kept until it is validated and submitted.





After the pending area is created, if your session abruptly ends or you make a mistake, simply run this next command to clear the pending area and start again.





Create the penalty box resource plan:



  plan => 'PENALTY_BOX_PLAN',

  comment => 'Resource plan to limit execution time');



Create the penalty box resource consumer group:



  consumer_group => 'PENALTY_BOX_GROUP',

  comment => 'Resource consumer group to limit execution time');




Create the penalty box resource plan directives.  Every resource plan must contain a directive to OTHER_GROUPS.  This consumer group contains all sessions that have not been assigned to a consumer group.  If you try to submit the resource plan without the OTHER_GROUPS option, you will get this error message when attempting to validate the pending area:



ERROR at line 1:

ORA-29377: consumer group OTHER_GROUPS is not part of top-plan PENALTY_BOX_PLAN

ORA-06512: at "SYS.DBMS_RMIN_SYS", line 3640

ORA-06512: at "SYS.DBMS_RMIN_SYS", line 3691

ORA-06512: at "SYS.DBMS_RMIN_SYS", line 3703

ORA-06512: at "SYS.DBMS_RESOURCE_MANAGER", line 746

ORA-06512: at line 2


As mentioned previously, we are setting an execution limit of 60 seconds.





    COMMENT=>'Kill statement after exceeding total execution time',








    COMMENT=>'leave others alone',




The CANCEL_SQL group that will be switched to when the SWITCH_TIME_IN_CALL of 60 seconds is reached is actually an internal call introduced in 11g that will just cancel the SQL.  The KILL_SESSION group can also be used, but that would be a bit extreme in this case.

Validate the pending area for the new plan, consumer group, and directives.  If there is a problem with the plan, it will appear during validation






Submit the pending area for plan, consumer group, and directives






Now we have to grant privileges in order to assign consumer groups to users.

Create a new pending area for the users we want to put in our newly-created penalty box:






Grant the switch privilege for resource consumer groups to users or roles.  We need to execute this command for every user we want to be eligible to put into the penalty box.



   grantee_name => 'SCOTT',

   consumer_group => 'PENALTY_BOX_GROUP',

   grant_option => FALSE);




Assign the desired users to the new resource consumer group



  user => 'SCOTT',

  consumer_group => 'PENALTY_BOX_GROUP');




Validate the pending area before submitting it:






Submit the pending area:






Finally, specify the new plan to be used by the instance:


System altered.

To clear the resource plan from the instance, use this command.  Note the command uses two single quotation marks, not a double-quote.  If you just want to switch to a different plan and not disable resource plans altogether, you can just supply the resource plan name within the quotation marks.


Once our user’s behavior has been adjusted, we can remove them from the plan and put them back into their original plan:



  user => 'SCOTT',

  consumer_group => 'DEFAULT_CONSUMER_GROUP');




To place other users in the penalty box, follow these abbreviated steps:



   grantee_name => 'USER',

   consumer_group => 'PENALTY_BOX_GROUP',

   grant_option => FALSE);





  user => 'USER',

  consumer_group => 'PENALTY_BOX_GROUP');




Now to test out our penalty box.  This query was executed as user SCOTT:

NOTE – This is my “hammer” query which will execute until it runs out of TEMP space:

set timing on

select distinct owner

from dba_constraints

where owner = (select owner

from dba_constraints

where constraint_type = 'C'

and owner in(select owner

from dba_constraints

where constraint_type = 'P'

and owner in(select owner

from dba_constraints

where constraint_type = 'R'

and owner in(select owner

from dba_constraints

where constraint_type = 'U'))))


If the penalty box is working properly, you should see this error if the query exceeds the time limit set in the resource plan directive.  The elapsed time output was the result of executing ‘set timing on’ to enable this SQL*Plus option.

ERROR at line 14:

ORA-00040: active time limit exceeded - call aborted

Elapsed: 00:01:31.20

Notice that the query did not end EXACTLY when the sixty-second time limit had been reached.  The time limit is an approximation.  The query will look to end after the time limit at the next convenient interrupt as determined by the database, much like times when you kill a database session and instead of ending immediately, it gets ‘marked for kill’ instead.  The database is waiting for the next convenient interrupt.  As a side note, you can add the word ‘immediate’ to the end of your ‘alter system kill session’ to speed things along.

If you want to make changes to the resource plan, consumer group, or resource plan directives, you can use the following commands.  Please note that you will need to create a pending area, execute the desired command, then submit the pending area in order for the command to take affect.

To remove a consumer group:





To remove a resource plan:





To remove a resource plan and all associated sub-plans and consumer groups:





To delete plan directives:








In the event that there is an active resource plan already in place, there are two options to consider.  Either you can create a new consumer group and assign the offending user to the new group, or you can add a plan directive to the offending user’s assigned consumer group.

The DBA_RSRC_CONSUMER_GROUP_PRIVS view displays the consumer groups granted to users or roles. Specifically, it displays the groups to which a user or role is allowed to belong or be switched.  In the example shown below, user SCOTT always starts in the SALES consumer group (INITIAL_GROUP is YES) and can switch to the MARKETING group through a specific grant, and can also switch to the DEFAULT_CONSUMER_GROUP (OTHER_GROUPS) and LOW_GROUP groups because they are granted to PUBLIC.  SCOTT also can grant the SALES group but not the MARKETING group to other users (GRANT_OPTION is YES for SALES, NO for MARKETING).

SELECT * FROM dba_rsrc_consumer_group_privs;

------------------ ------------------------------ ------------ -------------
PUBLIC             DEFAULT_CONSUMER_GROUP         YES          YES
PUBLIC             LOW_GROUP                      NO           NO
SCOTT              MARKETING                      NO           NO
SCOTT              SALES                          YES          YES
SYSTEM             SYS_GROUP                      NO           YES

Using this information, let’s look at our database after the PENALTY_BOX_PLAN has been created and enabled:


select *
from dba_rsrc_consumer_group_privs
order by grantee;

GRANTEE                        GRANTED_GROUP                  GRA INI
------------------------------ ------------------------------ --- ---
PUBLIC                         DEFAULT_CONSUMER_GROUP         YES YES
PUBLIC                         LOW_GROUP                      NO  NO
SCOTT                          PENALTY_BOX_GROUP              NO  YES
SYSTEM                         SYS_GROUP                      NO  YES

Of the two options I mentioned earlier, it may be safer to create a new consumer group for the particular user with all the directives of their currently-assigned initial group.  The script in My Oracle Support (MOS) note 1388634.1 has the ability to reverse-engineer a complete resource plan including all included consumer groups.

With this information, you can create a new consumer group with the addition of the execution time limiting directive mentioned earlier and then assign it to the user as their initial consumer group.

The second option – adding the execution time limiting directive to their initial consumer group, is more risky because the directive would affect ALL the users assigned to that consumer group.

This post has been a small sampling of what the Oracle Resource Manager can do.  Over the years, I have seen nearly no shops where this feature was actively used.  Using this feature in non-production environment where resources are usually more limited can be beneficial.  I have also seen instances where using it in a production environment where some users are allowed to execute ad-hoc queries would have also removed some problems.


Update 2/1/19

Upcoming Oracle version 19c takes the penalty box concept one step further:

Quarantine for SQL statements consuming excessive system resources

You can now configure quarantine for SQL statements terminated by the Resource
Manager due to their excessive consumption of system resources, such as CPU and I/O.
The quarantined SQL statements are not allowed to run again, thus protecting the
database from performance degradation.

The return of the Recyclebin

Last Friday, I received an email concerning a problem a DBA was having migrating objects from one tablespace to another.  They thought they had migrated all the objects, but they found some indexes when querying the DBA_SEGMENTS view.  These indexes were not visible in the DBA_INDEXES view, which was my first clue to the problem.  The next clue was when I found out all the indexes had a prefix of “BIN$”.  The DBA did not realize that these were objects that had been dropped previously and were sitting in the recyclebin.

The reyclebin  is a little-mentioned feature of the Oracle database that I blogged about back in 2011 when it saved me from an embarrassing and tricky recovery.  In honor of this unsung feature, I found the old blog, dusted it off, and updated it where needed.  Here we go…

What is recyclebin?

The Oracle recyclebin has been around since version 10g release 1 as part of a feature called Flashback Drop and in my opinion, continues to be a little-known and little-understood feature. It has saved me from potential data loss on at least one occasion, though, so I think it deserves some consideration.

The recyclebin is actually a data dictionary table containing information about dropped objects. Each named user account has their own recyclebin. Unless a user has the SYSDBA privilege, the only objects that the user has access to in the recyclebin are those that the user owns. A user can view his objects in the recyclebin using the following statement:


When the recyclebin is enabled, dropped tables and any associated objects such as indexes, constraints, and nested tables are removed from the DBA_<objects> views that track database objects.  During this process, the table is renamed.  The new name has a telltale “BIN$” prefix placed on it.  These objects still occupy space in the tablespace where they originated. They continue to count against user space quotas, until purged from the recyclebin either manually or by the database because of tablespace space constraints.

When you drop an entire tablespace including its contents, the objects in the tablespace are NOT placed in the recycle bin and the database purges any entries in the recycle bin for objects located in the dropped tablespace. The database also purges any recycle bin entries for objects in a tablespace you drop. Likewise:

  • When you drop a user, any objects belonging to the user are NOT placed in the recyclebin and any objects in the recyclebin owned by that user are purged.
  • When you drop a cluster, its member tables are NOT placed in the recyclebin and any former member tables in the recyclebin are purged.
  • When you drop a type, any dependent objects such as subtypes are NOT placed in the recyclebin and any former dependent objects in the recyclebin are purged.

Recyclebin Settings

In 10g release 1, the recyclebin was enabled by default and there was only an undocumented parameter to disable it.  This parameter was _recyclebin which defaulted to a value of TRUE.  The recyclebin initialization parameter was introduced in 10g release 2 and had a default setting of ON It was a dynamic parameter and could be enabled or disabled (ON or OFF) with either an ‘alter session’ or an ‘alter system’ command.  Oracle 11g release 1 made no changes to the basic feature, except to add the ‘deferred’ option to the ‘alter session’ command to allow subsequent, but not the current, session to be affected.  Release 2 remained unchanged from release 1.  These features remain unchained into 12c release 2.  Also, 12c includes more table-associated objects that is moves to the recyclebin when a table is dropped.  The full list of these objects are:

Nested tables

LOB segments


Constraints (excluding foreign key constraints)



To find out if the recyclebin is enabled, except for 10g release 1, you can execute ‘show parameter recyclebin’ from a SQL prompt as a user with sysdba privileges.

SQL> show parameter recyclebin

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
recyclebin                           string      on

I would recommend keeping the recyclebin enabled at all times, except during an upgrade, to add an extra layer of recovery options to the existing environment.

Using Recyclebin

Once the recyclebin is enabled, you don’t have to give much thought to its use. Every time a ‘drop table’ command is executed, the table and its associated objects are placed in the recyclebin and renamed with a system-generated naming standard. Again, the objects actually remain in their native tablespaces.  Because objects age out of the recyclebin based on demands for space, it is always a good idea to back up a table with an export or execute create table as select (CTAS) to create a backup table before dropping it. This is especially true for dropping very large tables which may not fit in the recyclebin.

I was unable to find any information on exactly how much space is allocated to the recyclebin, but suspect that it can consume all available space in the respective tablespace where the objects reside.

To find out how much space per owner and tablespace recyclebin data is using, you can use this query:

select owner, tablespace_name, round(sum(bytes)/1024/1024/1024,1) "RECYCLE_GB"

from dba_segments

where segment_name like 'BIN$%'

group by owner, tablespace_name 

order by owner, tablespace_name;

In the event that you do not want a table to be preserved in the recyclebin, the addition of the ‘purge’ option at the end of the ‘drop table’ statement ensures that the recyclebin bin is bypassed and the table as well as the associated objects are dropped from the database.  Just ensure you have a backup of the table, just in case.


Recovering tables from the recyclebin

To recover a table from the recyclebin, you need to know if it is available. Use the following query to get this information (the owner was designated to limit the output):

select owner, object_name, original_name, droptime

from dba_recyclebin

where owner = ‘CMN’;

OWNER                          OBJECT_NAME                    ORIGINAL_NAME                    DROPTIME

------------------------------ ------------------------------ -------------------------------- -------------------

CMN                            BIN$o5lg4yWjH3ngRAAUT0rhuQ==$0 CMN_BASE_CONTACT                 2011-05-18:23:12:42

CMN                            BIN$o5lg4yWiH3ngRAAUT0rhuQ==$0 CMN_BASE_CONTACT_PK              2011-05-18:23:12:42

CMN                            BIN$o5icV6rtByXgRAAUT0rhuQ==$0 CMN_BASE_CONTACT                 2011-05-18:22:51:07

CMN                            BIN$o5icV6rfByXgRAAUT0rhuQ==$0 CMN_BASE_CONTACT                 2011-05-18:22:17:45

CMN                            BIN$o5lg4yWvH3ngRAAUT0rhuQ==$0 CMN_BASE_CONTACT                 2011-05-18:23:15:11

CMN                            BIN$o5icV6rgByXgRAAUT0rhuQ==$0 EXECUTE_TABLE                    2011-05-18:22:49:58

CMN                            BIN$o5mU5fo6KF/gRAAUT0rhuQ==$0 EXECUTE_TABLE                    2011-05-18:23:30:15

CMN                            BIN$o5icV6ruByXgRAAUT0rhuQ==$0 EXECUTE_TABLE                    2011-05-18:22:53:23

CMN                            BIN$o5mU5foqKF/gRAAUT0rhuQ==$0 CMN_BASE_CONTACT                 2011-05-18:23:27:15

CMN                            BIN$o5mU5fo5KF/gRAAUT0rhuQ==$0 CMN_BASE_CONTACT                 2011-05-18:23:30:08

CMN                            BIN$o5mU5fo4KF/gRAAUT0rhuQ==$0 CMN_BASE_CONTACT_PK              2011-05-18:23:30:08

CMN                            BIN$o5mU5fpGKF/gRAAUT0rhuQ==$0 CMN_BASE_CONTACT                 2011-05-18:23:32:02

CMN                          BIN$o5mU5fpHKF/gRAAUT0rhuQ==$0 EXECUTE_TABLE                    2011-05-18:23:32:02

Here you can see which tables for the CMN user are in the recyclebin and when they were dropped.  You may have noticed that there are several versions of the BASE_CONTACT table in this recyclebin differentiated by their droptime value.

To also help determine which table holds the data you want recovered, you can query recyclebin tables like regular tables.

 select count(*)

from cmn."BIN$o5mU5fpGKF/gRAAUT0rhuQ==$0";




It is important to use both the owner identifier (CMN in this case) and double-quotation marks because of the special characters in the recyclebin table name.

Once the table you want to recover has been identified, there are a few different methods by which you can recover the table or the individual data. The ‘flashback table <table_name> to before drop’ command will recover the latest copy of the table in the recyclebin. You can also specify the ‘rename to <alt_table_name>’ at the end of the command to recover the table to a different table name to recover specific data. In the case where there are several copies of the table in the recyclebin and you are not interested in the latest copy, you can do a simple create table as select (CTAS) operation using the recyclebin table name, not the original name. Remember to use double-quotation marks around the recyclebin table name due to the special characters.

When you restore a table from the recyclebin, dependent objects such as indexes DO NOT get their original names back; they retain their system-generated recyclebin names. You must manually rename dependent objects to restore their original names. If you plan to manually restore original names for dependent objects, ensure that you make note of each dependent object’s system-generated recyclebin name before you restore the table.

Purging the recyclebin

You can purge the recyclebin completely or for objects belonging to individual users. As stated earlier, each user account in the database will automatically have a recyclebin assigned to it. A user can simply execute the ‘purge recyclebin’ command to purge their own recyclebin. Users with the sysdba privilege can purse the entire recyclebin with the ‘purge dba_recyclebin’ command. Be careful, though, as this command also purges ALL users recyclebins as well. Purging the entire reyclebin should be reserved for upgrade situations, or unless otherwise directed by Oracle Support.

You can also perform selective purges of the recyclebin as a sysdba-privileged user. For example ‘purge tablespace tmpdata’ only purges those objects in the recyclebin belonging to the tmpdata tablespace regardless of which user owns them. Also, ‘purge tablespace tmpdata user jon’ will purge only those objects in the recyclebin belonging to the tmpdata tablespace owned by the named user jon. You can also purge individual tables from the recyclebin with the command ‘purge table <table_name>’. The table name in this command can be either the system-generated recyclebin name or the original name of the table.  As of Oracle version 12c release 2, you need to purge individual user’s objects out of the recyclebin by tablespace as a sysdba-privileged user.

The recyclebin is one of those features that you forget, but are very thankful when it is available and you need it.

Update 6/21/21

Oracle bug 19264323 prevents the recycle bin from purging index segments which enforce a constraint automatically when under space pressure. This bug only affects Oracle version and is fixed in Oracle version 12.2. The workaround is to purge the recycle bin manually if this bug is encountered and there is a patch available to fix this bug. Reach out through your Oracle support channel for more details.

Don’t modify your spfile (unless you are sure)

I have seen it twice in the past week, so I need to get the word out.  You need to test parameter changes BEFORE modifying your spfile.  In at least one of the cases I have seen in the past week, the DBA created a backup of their spfile before modifying it.  This is smart, but they did not realize they could use this backup copy for testing changes.

The environment was an Oracle 11g release 2 ( three-node RAC system.  The DBA needed to change a non-modifiable (non-dynamic) parameter and restart the database to make the change permanent.  They created a copy of the spfile, which is always a GREAT idea before modifying it, brought down the three instances, but was unable to bring them back up due to a problem with the parameter setting.  Now the spfile is bad and the database won’t start.  Now they need to recreate the spfile with the backup and start the process over.

What they could have done is make the parameter changes to the spfile backup and try to start one more of the instances using it instead of the spfile. In that case, if the parameter setting is bad, the spfile is unaffected and corrections can be made more swiftly with a regular editor.

In order to start the instance with the backup spfile (pfile), this command could have been used while connected to one of the instances through SQL*Plus as SYSDBA:

startup pfile=<path_and_name_of_pfile>

Once the instance starts with the parameter change, you can shut it down, modify the spfile by either changing the parameter with an ‘alter system’ command or simply replacing the spfile with the contents of the modified pfile with the ‘create spfile from pfile’ command and you are done.



Database ‘shutdown immediate’ is slow or hangs

I was contacted by a DBA on a project I am on and they told me that a database they had executed a ‘shutdown immediate’ command on had been hanging for 20 minutes without any progress.

This problem is typical in an environment where a remote application maintains a pool of connections to the database.  It is these remote connection processes that are preventing the database from closing immediately.

I have not seen this issue in an Oracle-on-Windows environment, so I will just concentrate on resolving this issue in a Unix-type environment.  First, you need to see if you do indeed have remote connections:

$ >ps -ef | grep -v grep | grep LOCAL=NO
 oracle 29257 18960   0 05:31:03 ?     0:00 oracleDUMMY (LOCAL=NO)
 oracle 13255 18960   0   Sep 04 ?     0:00 oracleDUMMY (LOCAL=NO)
 oracle 29187 18960   0 05:31:01 ?     0:00 oracleDUMMY (LOCAL=NO)
 oracle 13277 18960   0   Sep 04 ?     0:01 oracleDUMMY (LOCAL=NO)
 oracle 29283 18960   0 05:31:07 ?     0:00 oracleDUMMY (LOCAL=NO)
 oracle 13285 18960   0   Sep 04 ?     0:00 oracleDUMMY (LOCAL=NO)
 oracle 29191 18960   0 05:31:02 ?     0:01 oracleDUMMY (LOCAL=NO)
 oracle 29290 18960   0 05:31:18 ?     0:05 oracleDUMMY (LOCAL=NO)
 oracle 29182 18960   0 05:31:00 ?     0:00 oracleDUMMY (LOCAL=NO)
 oracle 13269 18960   0   Sep 04 ?     0:00 oracleDUMMY (LOCAL=NO)
 oracle 13249 18960   0   Sep 04 ?     0:01 oracleDUMMY (LOCAL=NO)
 oracle 13251 18960   0   Sep 04 ?     0:01 oracleDUMMY (LOCAL=NO)
 oracle 29288 18960   0 05:31:17 ?     0:17 oracleDUMMY (LOCAL=NO)
 oracle 29292 18960   0 05:31:18 ?     0:01 oracleDUMMY (LOCAL=NO)
 oracle 29281 18960   0 05:31:07 ?     0:00 oracleDUMMY (LOCAL=NO)
 oracle 13283 18960   0   Sep 04 ?     0:01 oracleDUMMY (LOCAL=NO)
 oracle 29285 18960   0 05:31:07 ?     0:00 oracleDUMMY (LOCAL=NO)
 oracle 17496 18960   0 10:27:26 ?     0:01 oracleDUMMY (LOCAL=NO)
 oracle 13267 18960   0   Sep 04 ?     0:01 oracleDUMMY (LOCAL=NO)
 oracle 13288 18960   0   Sep 04 ?     0:00 oracleDUMMY (LOCAL=NO)
 oracle 10164 18960   0 12:41:18 ?     0:00 oracleDUMMY (LOCAL=NO)
 oracle  3579 18960   0 09:00:08 ?     0:00 oracleDUMMY (LOCAL=NO)
 oracle 12425 18960   0 12:56:18 ?     0:00 oracleDUMMY (LOCAL=NO)
 oracle 13275 18960   0   Sep 04 ?     0:00 oracleDUMMY (LOCAL=NO)
 oracle 29185 18960   0 05:31:01 ?     0:00 oracleDUMMY (LOCAL=NO)

Now that we know there are remote connections, they need to be removed so the database can continue the shutdown process.  It would take some time to issue ‘kill -9’ commands individually for each of these processes, so I came up with method to kill them automatically.  You just need to isolate the OS process for each remote connection:

$ >ps -ef | grep -v grep | grep LOCAL=NO | awk '{print $2}'

Once we isolate the OS process ID’s, we can plug them into a command that will loop through and perform a ‘kill -9’ on each:

for i in $(`ps -ef | grep -v grep | grep LOCAL=NO | awk '{print $2}'`)
 kill -9 $i

The command will execute immediately after pressing <ENTER> after the word ‘done’.  I gave this process to the DBA who contacted me and it resolved their issue.  Try this the next time your database will not shutdown immediately.

Generating test data sets with Swingbench

Reading about new features is great, but finding data to test them out on is  sometimes troublesome.  As DBAs, we’ve created test databases countless times, but the majority of the time they are empty.

Oracle does provide sample schemas that can be installed automatically by using the database configuration assistant (DBCA) when creating a database or manually by executing scripts under the Oracle software home directory.  Consult the documentation of your particular version for more details.  While these sample schemas do provide a framework that mimics a common layout, they lack the volume of data that typical production systems have.  If your employer allows “sandboxes” to be created to test new versions and features then you can stop reading here because you are one of the fortunate ones.  However, if you do not have this opportunity but would still like to create a sizeable data set to work with, there is a utility called Swingbench that fills this need very well.

Swingbench was created by Dominic Giles ( as an Oracle database load generator, but has since become a very useful suite of tools including Data Generator, Trace Analyzer, and several monitoring tools.  Best of all, these tools are free to use.  I would recommend that you provide a small donation if you find them useful, though.

Quoting the creator, “Data Generator is a utility to populate, create, and load tables with semi random data. Users can specify sequences, random text, random numbers, database columns, and files as sources for tables.”  This sounds pretty good.

I recently downloaded Swingbench version 2.5 which includes the Data Generator utility and worked with it on a Oracle VirtualBox virtual machine running Oracle Enterprise Linux (OEL) version 6 update 6 with and Oracle version non-container database.  Here are the steps I followed to create my test data set:

  • Swingbench requires Java 8 JDK.  Download the jdk-8u111-linux-x64.rpm and install it (as the root user) with the command: “rpm -Uvh jdk-8u111-linux-x64.rpm”. Note that this file is for the 64-bit version of OEL.
  • Download the file for version 2.5 from the website mentioned previously and place it in a location where the oracle user account has read, write, and execute privileges.
  • Unzip the file.  it creates a “swingbench” directory in this location
  •  As the oracle user, set the following environment variables if they are not already set (actual values depend on your environment):
    • JAVA_HOME=/usr/bin/java
  • Go to the swingbench/bin directory.  You have the option to create the sales history schema (shwizard), the calling center schema (ccwizard), or the order entry schema (oewizard).
  • Once you execute the desired wizard file, the interface appears



  • After clicking Next, you have the option to create or drop your desired schema.


  • For the connection string, use //<servername>/<database_sid>. Make sure you can ping the host by the <servername> used and that there is a listener present providing service for the <database_sid>.  Leave the connection type as the default (jdbc thin).  Leave DBA username as ‘sys as sysdba’ and make sure the password is correct.  Press Next to connect.


  • If the connection is successful, you will see the next screen where you can choose the username, password, schema tablespace, and tablespace datafile of your data set schema.  I like to keep my data separated for easier management, so I elect to create a separate tablespace and datafile.


  • The next screen allows you to choose database options such as partitioning (range or none), compression (yes or no), tablespace type (bigfile or normal), and indexing (all, primary key, or none).  Be careful here.  Do not select options that you are not licensed for in systems that are eligible for license audits.


  • Sizing options come next.  I don’t suppose I need to tell you not to size your schema any bigger than you have space available.  What is good here is that two counters at the bottom will let you know the size of the schema tablespace and the size of the temp tablespace required based on your selection.  Click next when you are done.


  • Finally, select the level of parallelism and click finish to create your schema.


  • The wizard log popup appears and the main wizard updates to tell you the progress of the creation process


  • Once the build is over, you should see a “successful” message popup window.  Click Ok to acknowledge the message and close the window.  You then have the option to review the wizard log in the wizard log window or save it to a file.  Close both the wizard log and main wizard windows after you are done.


At this point, you have the option to generate other data sets if you want to.  Enjoy.