Get in the Penalty Box! (Redux version)

Note – The commands and processes listed have been tested and verified in a Oracle 11g release 2 ( environment.

Sometimes, there exist database users with no regard for what type of queries they run or how long they run for.  These users typically kick off these ill-behaved queries and then go for coffee, lunch, or leave for the day.  Their query sits out there spinning and consuming resources that you know could be better used elsewhere.  This can be especially problematic in a production system if end-of-the-month aggregation processes or reports are being executed at the same time.

Maybe you have talked to them before about this behavior or even helped them tune a query or two, but they don’t seem to get the message.  In extreme cases, these users even terminate sessions when the query does not come back in a timely manner only to reconnect and retry it in a new session, leaving the old one (possibly) still executing.

It’s time to let these users know that their behavior is not acceptable in an environment where resources are finite and they need to share with other higher priority processes.  They need to spend some time in the penalty box.  For those unfamiliar with this term, it is a term used in ice hockey where misbehaving players have to sit out the game in a segregated area for a certain length of time, depending on the seriousness of their misdeeds.  The team with the misbehaving player is not allowed to send in a substitute player during the penalty time, so it leaves them one player short and vulnerable to be scored against.

The Oracle Resource Manager is the keeper of the database penalty box.  It has the ability to enforce limits on resources such CPU and I/O consumption, as well as execution time limits.  We are going to use this last feature to create our own penalty box.  I realize that most database managers don’t want to put constraints on users, especially developers, but in all honesty, if they are writing queries that take an hour or more to execute on a regular basis, then the managers need to put their foot down and the users or developers may need a remedial SQL tuning course.

A common misconception about resource plans is that they will limit all users in the database.  While it is true that only one resource plan can be active at any time, users can be segregated into consumer groups under the plan and each consumer group can be tailored with the use of directives to make sure that users or applications do not suffer due to the actions of a few users.

Here is a simple process to create a resource plan that will limit execution time to 60 seconds.  This limit is just for testing purposes as even I have been known to write queries that run for over a minute ;).  Administering the resource manager requires the ADMINISTER_RESOURCE_MANAGER privilege, go figure.  A SYSDBA user has this privilege already, but if you need to assign the privilege to a non-SYSDBA user, you can use this command (as a properly privileged user):

exec dbms_resource_manager_privs.grant_system_privilege( -

grantee_name => '<USER_NAME>', -

admin_option => true);

Check for a currently active resource plan:

SQL> show parameter resource_manager_plan

NAME                     TYPE     VALUE

-------------------------- ----------- ------------------------------

resource_manager_plan   string

Since no resource plan is active, we can continue with the rest of the penalty box steps to create one.  Later on, we will look at our options if there is an actual resource plan already in place.

Create a new pending area for plan, consumer group, and directives creation.  The pending area is where our resource plan work will be kept until it is validated and submitted.





After the pending area is created, if your session abruptly ends or you make a mistake, simply run this next command to clear the pending area and start again.





Create the penalty box resource plan:



  plan => 'PENALTY_BOX_PLAN',

  comment => 'Resource plan to limit execution time');



Create the penalty box resource consumer group:



  consumer_group => 'PENALTY_BOX_GROUP',

  comment => 'Resource consumer group to limit execution time');




Create the penalty box resource plan directives.  Every resource plan must contain a directive to OTHER_GROUPS.  This consumer group contains all sessions that have not been assigned to a consumer group.  If you try to submit the resource plan without the OTHER_GROUPS option, you will get this error message when attempting to validate the pending area:



ERROR at line 1:

ORA-29377: consumer group OTHER_GROUPS is not part of top-plan PENALTY_BOX_PLAN

ORA-06512: at "SYS.DBMS_RMIN_SYS", line 3640

ORA-06512: at "SYS.DBMS_RMIN_SYS", line 3691

ORA-06512: at "SYS.DBMS_RMIN_SYS", line 3703

ORA-06512: at "SYS.DBMS_RESOURCE_MANAGER", line 746

ORA-06512: at line 2


As mentioned previously, we are setting an execution limit of 60 seconds.





    COMMENT=>'Kill statement after exceeding total execution time',








    COMMENT=>'leave others alone',




The CANCEL_SQL group that will be switched to when the SWITCH_TIME_IN_CALL of 60 seconds is reached is actually an internal call introduced in 11g that will just cancel the SQL.  The KILL_SESSION group can also be used, but that would be a bit extreme in this case.

Validate the pending area for the new plan, consumer group, and directives.  If there is a problem with the plan, it will appear during validation






Submit the pending area for plan, consumer group, and directives






Now we have to grant privileges in order to assign consumer groups to users.

Create a new pending area for the users we want to put in our newly-created penalty box:






Grant the switch privilege for resource consumer groups to users or roles.  We need to execute this command for every user we want to be eligible to put into the penalty box.



   grantee_name => 'SCOTT',

   consumer_group => 'PENALTY_BOX_GROUP',

   grant_option => FALSE);




Assign the desired users to the new resource consumer group



  user => 'SCOTT',

  consumer_group => 'PENALTY_BOX_GROUP');




Validate the pending area before submitting it:






Submit the pending area:






Finally, specify the new plan to be used by the instance:


System altered.

To clear the resource plan from the instance, use this command.  Note the command uses two single quotation marks, not a double-quote.  If you just want to switch to a different plan and not disable resource plans altogether, you can just supply the resource plan name within the quotation marks.


Once our user’s behavior has been adjusted, we can remove them from the plan and put them back into their original plan:



  user => 'SCOTT',

  consumer_group => 'DEFAULT_CONSUMER_GROUP');




To place other users in the penalty box, follow these abbreviated steps:



   grantee_name => 'USER',

   consumer_group => 'PENALTY_BOX_GROUP',

   grant_option => FALSE);





  user => 'USER',

  consumer_group => 'PENALTY_BOX_GROUP');




Now to test out our penalty box.  This query was executed as user SCOTT:

NOTE – This is my “hammer” query which will execute until it runs out of TEMP space:

set timing on

select distinct owner

from dba_constraints

where owner = (select owner

from dba_constraints

where constraint_type = 'C'

and owner in(select owner

from dba_constraints

where constraint_type = 'P'

and owner in(select owner

from dba_constraints

where constraint_type = 'R'

and owner in(select owner

from dba_constraints

where constraint_type = 'U'))))


If the penalty box is working properly, you should see this error if the query exceeds the time limit set in the resource plan directive.  The elapsed time output was the result of executing ‘set timing on’ to enable this SQL*Plus option.

ERROR at line 14:

ORA-00040: active time limit exceeded - call aborted

Elapsed: 00:01:31.20

Notice that the query did not end EXACTLY when the sixty-second time limit had been reached.  The time limit is an approximation.  The query will look to end after the time limit at the next convenient interrupt as determined by the database, much like times when you kill a database session and instead of ending immediately, it gets ‘marked for kill’ instead.  The database is waiting for the next convenient interrupt.  As a side note, you can add the word ‘immediate’ to the end of your ‘alter system kill session’ to speed things along.

If you want to make changes to the resource plan, consumer group, or resource plan directives, you can use the following commands.  Please note that you will need to create a pending area, execute the desired command, then submit the pending area in order for the command to take affect.

To remove a consumer group:





To remove a resource plan:





To remove a resource plan and all associated sub-plans and consumer groups:





To delete plan directives:








In the event that there is an active resource plan already in place, there are two options to consider.  Either you can create a new consumer group and assign the offending user to the new group, or you can add a plan directive to the offending user’s assigned consumer group.

The DBA_RSRC_CONSUMER_GROUP_PRIVS view displays the consumer groups granted to users or roles. Specifically, it displays the groups to which a user or role is allowed to belong or be switched.  In the example shown below, user SCOTT always starts in the SALES consumer group (INITIAL_GROUP is YES) and can switch to the MARKETING group through a specific grant, and can also switch to the DEFAULT_CONSUMER_GROUP (OTHER_GROUPS) and LOW_GROUP groups because they are granted to PUBLIC.  SCOTT also can grant the SALES group but not the MARKETING group to other users (GRANT_OPTION is YES for SALES, NO for MARKETING).

SELECT * FROM dba_rsrc_consumer_group_privs;

------------------ ------------------------------ ------------ -------------
PUBLIC             DEFAULT_CONSUMER_GROUP         YES          YES
PUBLIC             LOW_GROUP                      NO           NO
SCOTT              MARKETING                      NO           NO
SCOTT              SALES                          YES          YES
SYSTEM             SYS_GROUP                      NO           YES

Using this information, let’s look at our database after the PENALTY_BOX_PLAN has been created and enabled:


select *
from dba_rsrc_consumer_group_privs
order by grantee;

GRANTEE                        GRANTED_GROUP                  GRA INI
------------------------------ ------------------------------ --- ---
PUBLIC                         DEFAULT_CONSUMER_GROUP         YES YES
PUBLIC                         LOW_GROUP                      NO  NO
SCOTT                          PENALTY_BOX_GROUP              NO  YES
SYSTEM                         SYS_GROUP                      NO  YES

Of the two options I mentioned earlier, it may be safer to create a new consumer group for the particular user with all the directives of their currently-assigned initial group.  The script in My Oracle Support (MOS) note 1388634.1 has the ability to reverse-engineer a complete resource plan including all included consumer groups.

With this information, you can create a new consumer group with the addition of the execution time limiting directive mentioned earlier and then assign it to the user as their initial consumer group.

The second option – adding the execution time limiting directive to their initial consumer group, is more risky because the directive would affect ALL the users assigned to that consumer group.

This post has been a small sampling of what the Oracle Resource Manager can do.  Over the years, I have seen nearly no shops where this feature was actively used.  Using this feature in non-production environment where resources are usually more limited can be beneficial.  I have also seen instances where using it in a production environment where some users are allowed to execute ad-hoc queries would have also removed some problems.




The return of the Recyclebin

Last Friday, I received an email concerning a problem a DBA was having migrating objects from one tablespace to another.  They thought they had migrated all the objects, but they found some indexes when querying the DBA_SEGMENTS view.  These indexes were not visible in the DBA_INDEXES view, which was my first clue to the problem.  The next clue was when I found out all the indexes had a prefix of “BIN$”.  The DBA did not realize that these were objects that had been dropped previously and were sitting in the recyclebin.

The reyclebin  is a little-mentioned feature of the Oracle database that I blogged about back in 2011 when it saved me from an embarrassing and tricky recovery.  In honor of this unsung feature, I found the old blog, dusted it off, and updated it where needed.  Here we go…

What is recyclebin?

The Oracle recyclebin has been around since version 10g release 1 as part of a feature called Flashback Drop and in my opinion, continues to be a little-known and little-understood feature. It has saved me from potential data loss on at least one occasion, though, so I think it deserves some consideration.

The recyclebin is actually a data dictionary table containing information about dropped objects. Each named user account has their own recyclebin. Unless a user has the SYSDBA privilege, the only objects that the user has access to in the recyclebin are those that the user owns. A user can view his objects in the recyclebin using the following statement:


When the recyclebin is enabled, dropped tables and any associated objects such as indexes, constraints, and nested tables are removed from the DBA_<objects> views that track database objects.  During this process, the table is renamed.  The new name has a telltale “BIN$” prefix placed on it.  These objects still occupy space in the tablespace where they originated. They continue to count against user space quotas, until purged from the recyclebin either manually or by the database because of tablespace space constraints.

When you drop an entire tablespace including its contents, the objects in the tablespace are NOT placed in the recycle bin and the database purges any entries in the recycle bin for objects located in the dropped tablespace. The database also purges any recycle bin entries for objects in a tablespace you drop. Likewise:

  • When you drop a user, any objects belonging to the user are NOT placed in the recyclebin and any objects in the recyclebin owned by that user are purged.
  • When you drop a cluster, its member tables are NOT placed in the recyclebin and any former member tables in the recyclebin are purged.
  • When you drop a type, any dependent objects such as subtypes are NOT placed in the recyclebin and any former dependent objects in the recyclebin are purged.

Recyclebin Settings

In 10g release 1, the recyclebin was enabled by default and there was only an undocumented parameter to disable it.  This parameter was _recyclebin which defaulted to a value of TRUE.  The recyclebin initialization parameter was introduced in 10g release 2 and had a default setting of ON It was a dynamic parameter and could be enabled or disabled (ON or OFF) with either an ‘alter session’ or an ‘alter system’ command.  Oracle 11g release 1 made no changes to the basic feature, except to add the ‘deferred’ option to the ‘alter session’ command to allow subsequent, but not the current, session to be affected.  Release 2 remained unchanged from release 1.  These features remain unchained into 12c release 2.  Also, 12c includes more table-associated objects that is moves to the recyclebin when a table is dropped.  The full list of these objects are:

Nested tables

LOB segments


Constraints (excluding foreign key constraints)



To find out if the recyclebin is enabled, except for 10g release 1, you can execute ‘show parameter recyclebin’ from a SQL prompt as a user with sysdba privileges.

SQL> show parameter recyclebin

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
recyclebin                           string      on

I would recommend keeping the recyclebin enabled at all times, except during an upgrade, to add an extra layer of recovery options to the existing environment.

Using Recyclebin

Once the recyclebin is enabled, you don’t have to give much thought to its use. Every time a ‘drop table’ command is executed, the table and its associated objects are placed in the recyclebin and renamed with a system-generated naming standard. Again, the objects actually remain in their native tablespaces.  Because objects age out of the recyclebin based on demands for space, it is always a good idea to back up a table with an export or execute create table as select (CTAS) to create a backup table before dropping it. This is especially true for dropping very large tables which may not fit in the recyclebin.

I was unable to find any information on exactly how much space is allocated to the recyclebin, but suspect that it can consume all available space in the respective tablespace where the objects reside.

To find out how much space per owner and tablespace recyclebin data is using, you can use this query:

select owner, tablespace_name, round(sum(bytes)/1024/1024/1024,1) "RECYCLE_GB"

from dba_segments

where segment_name like 'BIN$%'

group by owner, tablespace_name 

order by owner, tablespace_name;

In the event that you do not want a table to be preserved in the recyclebin, the addition of the ‘purge’ option at the end of the ‘drop table’ statement ensures that the recyclebin bin is bypassed and the table as well as the associated objects are dropped from the database.  Just ensure you have a backup of the table, just in case.


Recovering tables from the recyclebin

To recover a table from the recyclebin, you need to know if it is available. Use the following query to get this information (the owner was designated to limit the output):

select owner, object_name, original_name, droptime

from dba_recyclebin

where owner = ‘CMN’;

OWNER                          OBJECT_NAME                    ORIGINAL_NAME                    DROPTIME

------------------------------ ------------------------------ -------------------------------- -------------------

CMN                            BIN$o5lg4yWjH3ngRAAUT0rhuQ==$0 CMN_BASE_CONTACT                 2011-05-18:23:12:42

CMN                            BIN$o5lg4yWiH3ngRAAUT0rhuQ==$0 CMN_BASE_CONTACT_PK              2011-05-18:23:12:42

CMN                            BIN$o5icV6rtByXgRAAUT0rhuQ==$0 CMN_BASE_CONTACT                 2011-05-18:22:51:07

CMN                            BIN$o5icV6rfByXgRAAUT0rhuQ==$0 CMN_BASE_CONTACT                 2011-05-18:22:17:45

CMN                            BIN$o5lg4yWvH3ngRAAUT0rhuQ==$0 CMN_BASE_CONTACT                 2011-05-18:23:15:11

CMN                            BIN$o5icV6rgByXgRAAUT0rhuQ==$0 EXECUTE_TABLE                    2011-05-18:22:49:58

CMN                            BIN$o5mU5fo6KF/gRAAUT0rhuQ==$0 EXECUTE_TABLE                    2011-05-18:23:30:15

CMN                            BIN$o5icV6ruByXgRAAUT0rhuQ==$0 EXECUTE_TABLE                    2011-05-18:22:53:23

CMN                            BIN$o5mU5foqKF/gRAAUT0rhuQ==$0 CMN_BASE_CONTACT                 2011-05-18:23:27:15

CMN                            BIN$o5mU5fo5KF/gRAAUT0rhuQ==$0 CMN_BASE_CONTACT                 2011-05-18:23:30:08

CMN                            BIN$o5mU5fo4KF/gRAAUT0rhuQ==$0 CMN_BASE_CONTACT_PK              2011-05-18:23:30:08

CMN                            BIN$o5mU5fpGKF/gRAAUT0rhuQ==$0 CMN_BASE_CONTACT                 2011-05-18:23:32:02

CMN                          BIN$o5mU5fpHKF/gRAAUT0rhuQ==$0 EXECUTE_TABLE                    2011-05-18:23:32:02

Here you can see which tables for the CMN user are in the recyclebin and when they were dropped.  You may have noticed that there are several versions of the BASE_CONTACT table in this recyclebin differentiated by their droptime value.

To also help determine which table holds the data you want recovered, you can query recyclebin tables like regular tables.

 select count(*)

from cmn."BIN$o5mU5fpGKF/gRAAUT0rhuQ==$0";




It is important to use both the owner identifier (CMN in this case) and double-quotation marks because of the special characters in the recyclebin table name.

Once the table you want to recover has been identified, there are a few different methods by which you can recover the table or the individual data. The ‘flashback table <table_name> to before drop’ command will recover the latest copy of the table in the recyclebin. You can also specify the ‘rename to <alt_table_name>’ at the end of the command to recover the table to a different table name to recover specific data. In the case where there are several copies of the table in the recyclebin and you are not interested in the latest copy, you can do a simple create table as select (CTAS) operation using the recyclebin table name, not the original name. Remember to use double-quotation marks around the recyclebin table name due to the special characters.

When you restore a table from the recyclebin, dependent objects such as indexes DO NOT get their original names back; they retain their system-generated recyclebin names. You must manually rename dependent objects to restore their original names. If you plan to manually restore original names for dependent objects, ensure that you make note of each dependent object’s system-generated recyclebin name before you restore the table.

Purging the recyclebin

You can purge the recyclebin completely or for objects belonging to individual users. As stated earlier, each user account in the database will automatically have a recyclebin assigned to it. A user can simply execute the ‘purge recyclebin’ command to purge their own recyclebin. Users with the sysdba privilege can purse the entire recyclebin with the ‘purge dba_recyclebin’ command. Be careful, though, as this command also purges ALL users recyclebins as well. Purging the entire reyclebin should be reserved for upgrade situations, or unless otherwise directed by Oracle Support.

You can also perform selective purges of the recyclebin as a sysdba-privileged user. For example ‘purge tablespace tmpdata’ only purges those objects in the recyclebin belonging to the tmpdata tablespace regardless of which user owns them. Also, ‘purge tablespace tmpdata user jon’ will purge only those objects in the recyclebin belonging to the tmpdata tablespace owned by the named user jon. You can also purge individual tables from the recyclebin with the command ‘purge table <table_name>’. The table name in this command can be either the system-generated recyclebin name or the original name of the table.  As of Oracle version 12c release 2, you need to purge individual user’s objects out of the recyclebin by tablespace as a sysdba-privileged user.

The recyclebin is one of those features that you forget, but are very thankful when it is available and you need it.

Don’t modify your spfile (unless you are sure)

I have seen it twice in the past week, so I need to get the word out.  You need to test parameter changes BEFORE modifying your spfile.  In at least one of the cases I have seen in the past week, the DBA created a backup of their spfile before modifying it.  This is smart, but they did not realize they could use this backup copy for testing changes.

The environment was an Oracle 11g release 2 ( three-node RAC system.  The DBA needed to change a non-modifiable (non-dynamic) parameter and restart the database to make the change permanent.  They created a copy of the spfile, which is always a GREAT idea before modifying it, brought down the three instances, but was unable to bring them back up due to a problem with the parameter setting.  Now the spfile is bad and the database won’t start.  Now they need to recreate the spfile with the backup and start the process over.

What they could have done is make the parameter changes to the spfile backup and try to start one more of the instances using it instead of the spfile. In that case, if the parameter setting is bad, the spfile is unaffected and corrections can be made more swiftly with a regular editor.

In order to start the instance with the backup spfile (pfile), this command could have been used while connected to one of the instances through SQL*Plus as SYSDBA:

startup pfile=<path_and_name_of_pfile>

Once the instance starts with the parameter change, you can shut it down, modify the spfile by either changing the parameter with an ‘alter system’ command or simply replacing the spfile with the contents of the modified pfile with the ‘create spfile from pfile’ command and you are done.



Database ‘shutdown immediate’ is slow or hangs

I was contacted by a DBA on a project I am on and they told me that a database they had executed a ‘shutdown immediate’ command on had been hanging for 20 minutes without any progress.

This problem is typical in an environment where a remote application maintains a pool of connections to the database.  It is these remote connection processes that are preventing the database from closing immediately.

I have not seen this issue in an Oracle-on-Windows environment, so I will just concentrate on resolving this issue in a Unix-type environment.  First, you need to see if you do indeed have remote connections:

$ >ps -ef | grep -v grep | grep LOCAL=NO
 oracle 29257 18960   0 05:31:03 ?     0:00 oracleDUMMY (LOCAL=NO)
 oracle 13255 18960   0   Sep 04 ?     0:00 oracleDUMMY (LOCAL=NO)
 oracle 29187 18960   0 05:31:01 ?     0:00 oracleDUMMY (LOCAL=NO)
 oracle 13277 18960   0   Sep 04 ?     0:01 oracleDUMMY (LOCAL=NO)
 oracle 29283 18960   0 05:31:07 ?     0:00 oracleDUMMY (LOCAL=NO)
 oracle 13285 18960   0   Sep 04 ?     0:00 oracleDUMMY (LOCAL=NO)
 oracle 29191 18960   0 05:31:02 ?     0:01 oracleDUMMY (LOCAL=NO)
 oracle 29290 18960   0 05:31:18 ?     0:05 oracleDUMMY (LOCAL=NO)
 oracle 29182 18960   0 05:31:00 ?     0:00 oracleDUMMY (LOCAL=NO)
 oracle 13269 18960   0   Sep 04 ?     0:00 oracleDUMMY (LOCAL=NO)
 oracle 13249 18960   0   Sep 04 ?     0:01 oracleDUMMY (LOCAL=NO)
 oracle 13251 18960   0   Sep 04 ?     0:01 oracleDUMMY (LOCAL=NO)
 oracle 29288 18960   0 05:31:17 ?     0:17 oracleDUMMY (LOCAL=NO)
 oracle 29292 18960   0 05:31:18 ?     0:01 oracleDUMMY (LOCAL=NO)
 oracle 29281 18960   0 05:31:07 ?     0:00 oracleDUMMY (LOCAL=NO)
 oracle 13283 18960   0   Sep 04 ?     0:01 oracleDUMMY (LOCAL=NO)
 oracle 29285 18960   0 05:31:07 ?     0:00 oracleDUMMY (LOCAL=NO)
 oracle 17496 18960   0 10:27:26 ?     0:01 oracleDUMMY (LOCAL=NO)
 oracle 13267 18960   0   Sep 04 ?     0:01 oracleDUMMY (LOCAL=NO)
 oracle 13288 18960   0   Sep 04 ?     0:00 oracleDUMMY (LOCAL=NO)
 oracle 10164 18960   0 12:41:18 ?     0:00 oracleDUMMY (LOCAL=NO)
 oracle  3579 18960   0 09:00:08 ?     0:00 oracleDUMMY (LOCAL=NO)
 oracle 12425 18960   0 12:56:18 ?     0:00 oracleDUMMY (LOCAL=NO)
 oracle 13275 18960   0   Sep 04 ?     0:00 oracleDUMMY (LOCAL=NO)
 oracle 29185 18960   0 05:31:01 ?     0:00 oracleDUMMY (LOCAL=NO)

Now that we know there are remote connections, they need to be removed so the database can continue the shutdown process.  It would take some time to issue ‘kill -9’ commands individually for each of these processes, so I came up with method to kill them automatically.  You just need to isolate the OS process for each remote connection:

$ >ps -ef | grep -v grep | grep LOCAL=NO | awk '{print $2}'

Once we isolate the OS process ID’s, we can plug them into a command that will loop through and perform a ‘kill -9’ on each:

for i in $(`ps -ef | grep -v grep | grep LOCAL=NO | awk '{print $2}'`)
 kill -9 $i

The command will execute immediately after pressing <ENTER> after the word ‘done’.  I gave this process to the DBA who contacted me and it resolved their issue.  Try this the next time your database will not shutdown immediately.

Generating test data sets with Swingbench

Reading about new features is great, but finding data to test them out on is  sometimes troublesome.  As DBAs, we’ve created test databases countless times, but the majority of the time they are empty.

Oracle does provide sample schemas that can be installed automatically by using the database configuration assistant (DBCA) when creating a database or manually by executing scripts under the Oracle software home directory.  Consult the documentation of your particular version for more details.  While these sample schemas do provide a framework that mimics a common layout, they lack the volume of data that typical production systems have.  If your employer allows “sandboxes” to be created to test new versions and features then you can stop reading here because you are one of the fortunate ones.  However, if you do not have this opportunity but would still like to create a sizeable data set to work with, there is a utility called Swingbench that fills this need very well.

Swingbench was created by Dominic Giles ( as an Oracle database load generator, but has since become a very useful suite of tools including Data Generator, Trace Analyzer, and several monitoring tools.  Best of all, these tools are free to use.  I would recommend that you provide a small donation if you find them useful, though.

Quoting the creator, “Data Generator is a utility to populate, create, and load tables with semi random data. Users can specify sequences, random text, random numbers, database columns, and files as sources for tables.”  This sounds pretty good.

I recently downloaded Swingbench version 2.5 which includes the Data Generator utility and worked with it on a Oracle VirtualBox virtual machine running Oracle Enterprise Linux (OEL) version 6 update 6 with and Oracle version non-container database.  Here are the steps I followed to create my test data set:

  • Swingbench requires Java 8 JDK.  Download the jdk-8u111-linux-x64.rpm and install it (as the root user) with the command: “rpm -Uvh jdk-8u111-linux-x64.rpm”. Note that this file is for the 64-bit version of OEL.
  • Download the file for version 2.5 from the website mentioned previously and place it in a location where the oracle user account has read, write, and execute privileges.
  • Unzip the file.  it creates a “swingbench” directory in this location
  •  As the oracle user, set the following environment variables if they are not already set (actual values depend on your environment):
    • JAVA_HOME=/usr/bin/java
  • Go to the swingbench/bin directory.  You have the option to create the sales history schema (shwizard), the calling center schema (ccwizard), or the order entry schema (oewizard).
  • Once you execute the desired wizard file, the interface appears



  • After clicking Next, you have the option to create or drop your desired schema.


  • For the connection string, use //<servername>/<database_sid>. Make sure you can ping the host by the <servername> used and that there is a listener present providing service for the <database_sid>.  Leave the connection type as the default (jdbc thin).  Leave DBA username as ‘sys as sysdba’ and make sure the password is correct.  Press Next to connect.


  • If the connection is successful, you will see the next screen where you can choose the username, password, schema tablespace, and tablespace datafile of your data set schema.  I like to keep my data separated for easier management, so I elect to create a separate tablespace and datafile.


  • The next screen allows you to choose database options such as partitioning (range or none), compression (yes or no), tablespace type (bigfile or normal), and indexing (all, primary key, or none).  Be careful here.  Do not select options that you are not licensed for in systems that are eligible for license audits.


  • Sizing options come next.  I don’t suppose I need to tell you not to size your schema any bigger than you have space available.  What is good here is that two counters at the bottom will let you know the size of the schema tablespace and the size of the temp tablespace required based on your selection.  Click next when you are done.


  • Finally, select the level of parallelism and click finish to create your schema.


  • The wizard log popup appears and the main wizard updates to tell you the progress of the creation process


  • Once the build is over, you should see a “successful” message popup window.  Click Ok to acknowledge the message and close the window.  You then have the option to review the wizard log in the wizard log window or save it to a file.  Close both the wizard log and main wizard windows after you are done.


At this point, you have the option to generate other data sets if you want to.  Enjoy.


Sequences are not gap free!

mind the gap

Yesterday I encountered a trace file on a “poorly-performing” process with some interesting waits.  The trace was incomplete because the trace process was started once the process was identified as “poorly-performing”.  Even so, it was interesting.


call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        0      0.00       0.00          0          0          0           0
Execute      0      0.00       0.00          0          0          0           0
Fetch        0      0.00       0.00          0          0          0           0
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total        0      0.00       0.00          0          0          0           0


call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        0      0.00       0.00          0          0          0           0
Execute     78      0.03       0.05          0        106        106          53
Fetch       25      0.01       0.03          0         75          0          25
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total      103      0.05       0.08          0        181        106          78

Trace file: PROD2_ora_2406.trc
Trace file compatibility:
Sort options: exeela  fchela  prsela  
       1  session in tracefile.
       0  user  SQL statements in trace file.
       2  internal SQL statements in trace file.
       2  SQL statements in trace file.
       2  unique SQL statements in trace file.
  384089  lines in trace file.
    1507  elapsed seconds in trace file.


Ok, we have 1507 seconds of tracing, or slightly over 25 minutes.  Of that, only .08 seconds are accounted for between the recursive (system) and non-recursive (user) statements.  Where did the time go?  There were no event waits under the non-recursive statements section, but the wait events for the recursive statements was very telling.  Note that this is not a complete listing, I just filtered out any waits that did not add up to more than 1 second.

Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  enq: SV -  contention                      289471        0.48       1210.91
  gc current grant 2-way                      22811        0.13         17.21
  db file sequential read                     57139        0.53         83.05
  gc current block 2-way                       2094        0.02          1.94

We now see that about 1211 seconds, or about 20 minutes of the trace, was spent on “enq: SV – contention”.  But what is that?  This event is sequence value (SV) contention and appears when there is a high demand for sequence values across multiple instances in a RAC environment.  I typically see “log file sync” waits that point to sequence contention issues in non-RAC environments.

When sequences are created they have a default cached value of 20.  This means that every time a sequence is accessed, the next 20 values are cached in memory for quick access.  This value needs to be much higher when the sequence is used to generate values for inserts when hundreds of thousands or millions of records are inserted in one process or routine.

I always recommend increasing the size of the sequence cache values when I see these type of situations, but I am often met with one of two very weak arguments.  One is that the client does not want to have to recreate the sequences to increase the cache value while the other is that the client wants to keep the cache small in order to avoid gaps in sequence values.  Let’s take these arguments one at a time.

First,  altering the sequence cache value dynamically has been available since Oracle database version 10g release 1.  In fact, there is a quote in the Oracle 10g release 1 SQL reference guide which states “Oracle recommends using the CACHE setting to enhance performance if you are using sequences in a Real Application Clusters environment.”  The rest of this argument is disproved by creating a test sequence and changing the cache value with a simple “alter sequence” command.

The second argument is the most exasperating for me because the client usually won’t budge from their position.  Although the client is correct when they state that a smaller cache size reduces the risk of sequence value gaps, they are mistaken when they assume that a reduced cache will ensure no sequence gaps.

If you are relying on sequences to generate primary keys for newly-inserted records, there is NO guarantee that there will be no sequence value gaps.  Even Tom Kyte is in agreement here (

The closest you can get to no gaps using a sequence in this fashion is creating the sequence with the “INCREMENT BY 1”,  “NOCACHE”, and “ORDER” options together.  There will be a MAJOR performance cost using a sequence in this fashion for generating values during large insert activity.  Even with these values set, there is still the possibility of sequence value gaps.  Here is a version 11.2.04 demonstration of that fact:

SQL> create table jontab2 (jonkey number, jonstring varchar2(20));

Table created.

SQL> create sequence jonseq start with 1 increment by 1 nomaxvalue nocycle order nocache;

Sequence created.

SQL> insert into jontab2 values(jonseq.nextval, 'String');

1 row created.

SQL> insert into jontab2 values(jonseq.nextval, 'String');

1 row created.

SQL> commit;

Commit complete.

SQL> insert into jontab2 values(jonseq.nextval, 'String');

1 row created.

SQL> insert into jontab2 values(jonseq.nextval, 'String');

1 row created.

SQL> insert into jontab2 values(jonseq.nextval, 'String');

1 row created.

SQL> insert into jontab2 values(jonseq.nextval, 'String');

1 row created.

SQL> insert into jontab2 values(jonseq.nextval, 'String');

1 row created.

SQL> insert into jontab2 values(jonseq.nextval, 'String');

1 row created.

<I crash the instance here by killing the PMON process>

SQL> insert into jontab2 values(jonseq.nextval, 'String');
insert into jontab2 values(jonseq.nextval, 'String')
ERROR at line 1:
ORA-03135: connection lost contact
Process ID: 3657
Session ID: 125 Serial number: 7

SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

<I restart the instance and login>

[oracle@rac1 ~]$ sqlplus jontest/semperf1

SQL*Plus: Release Production on Fri Aug 4 09:10:01 2017

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> insert into jontab2 values(jonseq.nextval, 'String');

1 row created.

SQL> commit;

Commit complete.

SQL> select *
  2  from jontab2
  3  order by jonkey;

---------- --------------------
         1 String
         2 String
         9 String                        --------GAAAAAAP!!!!!!!

SQL> select jonseq.nextval
  2  from dual;



Because I did not commit some newly-inserted records before the instance crashed, they were not written to the table even though the sequence incremented and created a gap in the key values of the table.  Even if you have a batch process that commits after every record inserted, you can’t account for instance crashes.

If you STILL intend to use a sequence alone to generate a “no gap” primary key, what happens if you have to delete records from the table?  Guess what, you now have gaps in your primary key values.  What are you going to do, re-index all the remaining records to remove the gaps?  I doubt it.  The best bet is to use the sequence and one or more other values in the table, perhaps even a timestamp, as the basis for a primary key value.  This way, you can increase the sequence cache and still have solid primary keys.  You just need to get over the “no gap sequence values” idea.


Oracle Multitenant – Part 2 – Creating, connecting to, removing, and querying CDBs and PDBs

Now that we have an understanding of the multitenant architecture from part 1, we want to add some practical experience starting with the creation of CDBs and PDBs.  My test environment will be Oracle VirtualBox v.5.1.22 guest running an Oracle Enterprise Linux 7.3 guest with Oracle version software installed.

Creating CDBs and PDBs

NOTE – If you create a CDB under Oracle 12c release 1, the following is not supported:

  • Flashback Transaction Query
  • Flashback Transaction Back out
  • Database Change Notification
  • Continuous Query Notification
  • Client Side Cache
  • Heat map
  • Automatic Data Optimization
  • Oracle Streams
  • Oracle Fail Safe release 4.1.0 that ships with Oracle Database

The preferred CDB creation method is through the Database Configuration Assistant (DBCA).  The DBCA wizard will guide you through the process and automatically execute the proper catalog scripts once the CDB is created.  The DBCA can also be used to generate all the scripts you need to create the CDB manually based on choices made through the DBCA create database wizard.  The DBCA can also generate a CDB creation template that you can use to run DBCA in other environments if you copy the template over.

Once you login to the OS as the Oracle software owner and set your environment, execute $ORACLE_HOME/bin/dbca


The obligatory Oracle splash screen appears


Choose the ‘create database’ option and proceed to the next screen


We are going to choose ‘typical configuration’ here to save time.  On this screen, we can select the storage type, DB file location, character set, and designate the administrative (SYS and SYSTEM) passwords.  More importantly, we have the option to create the new database as a CDB or non-CDB database.  Since CDB is our goal here, we check the CDB box and designate the name of our inclusive pluggable database, then proceed.


The summary screen appears to confirm our choices.


If you like, you can watch the progress bar and the progress of the individual steps as the database is being built


Acknowledge the finish screen when the process completes to exit the DBCA


What if we had selected the ‘advanced configuration’ option earlier?  All the options under Typical configuration are automatically disabled.


Now we need to select the type of database (single-instance, RAC, or RAC one-node) and the characteristics (custom, general purpose, or data warehouse).  We’ll select single-instance and general purpose, then proceed.


There are some interesting choices on the next screen.  After providing a global database name and SID, you can choose a CDB or non-CDB database.  If you choose a CDB, you can opt for an empty CDB with no PDBs (except the PDB$SEED) or you can choose to create one or more PDBs at the same time.  Another option that is newly available in 12c release 2 is the option to create a separate UNDO tablespace for each PDB.


Storage options come next


Recovery options on this screen.  I chose to create a flash recovery area with the recommended size and enable archiving.


Listener selection is next.  There are no existing database listeners in the system, so I opt to create one with a default name and port number.  It is recommended to avoid using the default 1521 port for security reasons, but since this is a closed system, there should be no concerns.


At the next screen, I chose not to install Database Vault or Label Security


The next screen is multi-tabbed with options for memory allocation, sizing of processes, selection of character sets,  connection mode, and installation of sample schemas.  I choose the defaults in each section.


I opted to configure Enterprise Manager database express.  We may want to view our CDB options for backup and recovery later in a separate post.


Specify the administrative user credentials at the next screen


This screen prompts you with options to create the database, create a template of your choices for additional CDBs of this type, or the generation of scripts for manual database creation.  I opt for all three.  Why not?


Confirm your choices on the summary screen


If you like, you can watch the progress bar and the progress of the individual steps as the database is being built


Acknowledge the finish screen when the process completes to exit the DBCA


Creating a PDB

If you have a CDB and want to create a PDB or add a PDB, the DBCA is also the preferred method.

To create a new PDB, start DBCA. Once you get past the 12c splash screen, select ‘Manage pluggable databases’ on the first screen, then proceed.


This screen contains all the PDB options you have in your current configuration such as create, delete, unplug, or configure.  The delete, unplug, and configure options will only by available if there is an existing PDB.  Select ‘create a pluggable database’, then proceed.


Designate the CDB that will host your new PDB and give the admin account credentials for the designated CDB, then proceed.


Here you can designate whether to create a new PDB from the ‘seed’ or create a new PDB from an existing PDB.  We are going to use the seed since we don’t have an existing PDB.


Give a name to the new PDB and the credentials of the new administrative account for the PDB.  It is interesting to note here that the wizard is not looking for a typical SYS or SYSTEM account since these accounts already exist at the CDB level.


Storage options are next as well as opting to create a default user tablespace.  The default user tablespace will be called USERS if you choose this option.  it is interesting to note here that even though the option is not given, the new PDB will have its own UNDO tablespace by default.


Confirm your choices on the summary screen, then proceed.


Watch the progress of the PDB creation.


Acknowledge the finish screen when the process completes to exit the DBCA


Other methods to create a PDB include:

  • Using the ‘create pluggable database’ command to create the PDB while connected to the CDB.
  • Plug an existing PDB into the CDB.
  • Using Oracle Cloud Control to guide you through the creation of the PDB.
  • Cloning an existing PDB to a new PDB with, or without data.
  • Creating an empty PDB and using Oracle Data Pump to import data from one database source into a PDB.

Starting and stopping the CDB

Starting and stopping a CDB is just like starting and stopping a non-CDB database.  Use the SQL*Plus utility (or Oracle Enterprise Manager) and issue the startup and shutdown commands as usual.  All of the options still exist such as startup force, shutdown abort and shutdown immediate.

Starting and stopping the PDB

First, determine the state of the PDB before issuing start or stop commands by connecting to the CDB with an account having SYSDBA, SYSOPER, SYSBACKUP or SYSDG privileges:

SQL> select name, open_mode from v$pdbs;

 --------------------       ------------------

The PDB$SEED database is ALWAYS read only.  As the template database, it needs to maintain its integrity.  Since the PDB is already mounted, you just need to open it:

SQL> alter pluggable database JON12PDB1 open;

Pluggable database altered.

You can execute the commands directly without the ‘pluggable database’ syntax by changing to the PDB within the session first:

SQL> alter session set container=JON12PDB1;

Session altered.

SQL> shutdown
 Pluggable Database closed.

I would not recommend this method, however, since it would be easy to forget what container you are pointing to and issue a shutdown mistakenly against the CDB could make you think about updating your resume.

Other methods:

  • Use the ‘alter pluggable database’ command to start and stop the PDB when connected to the CDB with an account having SYSDBA, SYSOPER, SYSBACKUP or SYSDG privileges.
  • Using the SQL*Plus ‘startup pluggable database’ and ‘shutdown pluggable database’ commands when connected to the CDB with an account having SYSDBA, SYSOPER, SYSBACKUP or SYSDG privileges.

PDB startup options (make sure you are connected to the PDB using the ‘alter session set container’ command while in the CDB in the example above):

Open (default) – Open the PDB normally (startup pluggable database JON12PDB1 😉

Force – Force the PDB open (startup pluggable database JON12PDB1 force;)

Upgrade – Open the PDB in upgrade mode (startup pluggable database JON12PDB1 upgrade;)

Restricted – Open the PDB in restricted mode (startup pluggable database JON12PDB1 restricted;)

Read only – Open the PDB in read only mode (startup pluggable database JON12PDB1 open read only;)

For those who don’t want to have to execute separate startup commands for the PDBs after starting the CDB, there is a means by which you can set the default state per PDB after the CDB is started.

For example, after starting the CDB, set the state of your desired PDB, albeit OPEN (read/write) or OPEN READ ONLY, execute the SAVE STATE command from the CDB:

alter pluggable database <PDB_NAME> save state;

Now when the CDB starts, the respective PDB will automatically go to the saved state.  You can view the saved state of each PDB in the DBA_PDB_SAVED_STATES view.  Please note that the saved state feature for PDBs is only on Oracle software versions or higher.

To clear the saved state of the PDB, issue the DISCARD STATE command:

alter pluggable database <PDB_NAME> discard state;

This command will set the state of the particular PDB back to the MOUNTED default.  Changing the saved state of the desired PDB is a simple matter of placing the PDB in the desired state and issuing the SAVE STATE command again.

PDB open options:

Open (default) – Open the PDB in read write mode (alter pluggable database JON12PDB1 open;)

Open read only – Open the PDB in read only mode (alter pluggable database JON12PDB1 open read only;)

Open resetlogs – Open the PDB after a point-in-time-recovery (PITR) (alter pluggable database JON12PDB1 open resetlogs;)

Open restricted – Open the PDB in restricted mode (alter pluggable database JON12PDB1 open restricted;)

Force – Force shutdown and restart (alter pluggable database JON12PDB1 open force;)

Upgrade  – Open in upgrade mode (alter pluggable database JON12PDB1 open upgrade;)

PDB shutdown options (make sure you are connected to the PDB using the ‘alter session set container’ command while in the CDB in the example above):

Normal (default) – Close the PDB normally, waiting for transactions to end (shutdown)

Shutdown Immediate – Close the PDB. Rollback any pending transaction. Flush the SGA (shutdown immediate)

Abort – Shutdown the PDB without delay. Dirty buffers may be written to disk (shutdown abort)

Transactional – Shutdown the PDB after all active transactions have completed (shutdown transactional)

PDB close options:

Close (default) – Close the PDB normally (alter pluggable database JON12PDB1 close;)

Immediate – Force the PDB closed (alter pluggable database JON12PDB1 close immediate;)


Connecting to the CDB/PDB

Accessing the CDB is the same as accessing a non-CDB database and we have already discussed accessing the PDB while connected to the CDB, but what about accessing the PDB directly from outside the CDB?  As I mentioned in my last post, listener services are automatically created for the CDB and each PDB on creation.  Here you can see the JON12CDB and JON12PDB1 services when checking the listener status.


Make sure the CDB is started and the PDB is open.  You can query the v$pdbs view from the CDB to see the status of all the enclosed PDBs.  Once you are sure the PDB is open, you can add an entry to your tnsnames.ora file to access the PDB directly (The entry for the CDB should already be present):

 (ADDRESS = (PROTOCOL = TCP)(HOST = localhost)(PORT = 1521))

You should then be able to access your PDB directly:


Dropping a CDB and PDB

Dropping or removing a CDB is the same as dropping a non-CDB database.  The only difference in the case of dropping a CDB is that all PDB databases beneath it will be removed as well.

Dropping or removing a PDB is a bit more complicated when performing the task manually, but not by much.  When connected to the CDB that contains the PDB, use the ‘alter pluggable database’ command to first close the PDB, then issue the ‘drop pluggable database <PDB_name> including datafiles;’ to complete the process.  Using the DBCA to drop a PDB is even easier as it performs the closing of the PDB automatically, if necessary.

Other PDB removal options:

  • The SQL prompt, using the drop pluggable database command.
  • Using Oracle Enterprise Manager Cloud Control to remove the PDB
  • Unplugging the PDB
  • Deleting the entire CDB.


Querying the CDB and PDB

Once you are connected to the CDB or PDB, queries function the same as they have in non-CDB databases.  What is more important in this stage of our multitenant experience is knowing about multitenant-specific commands and objects.  For instance, when you are connected to a CDB, you can use the command ‘show pdbs’ to get all associated PDBs listed with their current state:

SQL> show pdbs

 ---------- ------------------------------ ---------- ----------

Remember the PDB$SEED PDB is ALWAYS open in read only mode.

The container ID (CON_ID) is an important identifier in the multitenant environment.  The ‘show con_id’ command displays the CON_ID of the current CDB or PDB you are in.  Notice the value changes when I switch from the CDB to the PDB with the ‘alter session set container’ command:

SQL> show con_id

 SQL> alter session set container=jon12pdb1;

Session altered.

SQL> show con_id


NOTE – Once you alter your session to switch from the CDB to a PDB, you cannot switch back to the CDB and must exit your session and open a new one or open a different session to access the CDB once more.

The ‘show con_name’ command is similar to the ‘show con_id’ except that the name and not the ID is displayed:

SQL> show con_name


SQL> alter session set container=jon12pdb1;

Session altered.

SQL> show con_name



If you need a list of the PDB IDs and names, query the v$pdbs view:
SQL> select con_id, name
2  from v$pdbs;

 ---------- ----------
 3 JON12PDB1

In some views, the CON_ID column has been added for multitenant identification as in the case of v$tablespace:

select a.con_id,,
 from v$tablespace a, v$pdbs b
 where a.con_id=b.con_id order by 1;

 ---------- ---------- ----------
 2 TEMP       PDB$SEED
 3 TEMP       JON12PDB1
 3 USERS      JON12PDB1

NOTE – This is a listing from the CDB.  If you are connected to a PDB, you will only see the tablespaces for that particular PDB.

In other views, some filtering has been enabled, such as dba_tables.  In this case, you will see only tables associated with the particular CDB or PDB you are connected to:

SQL> select count(*)
 2  from dba_tables;


SQL> alter session set container=jon12pdb1;

Session altered.

SQL> select count(*)
 2  from dba_tables;


The difference in the counts here is that when you are connected to the PDB, you cannot see the CDB-specific objects.  The CDB_TABLES view allows you to identify tables as belonging to a CDB or PDB:

select a.con_id,, count(*)
 from cdb_tables a, v$containers b
 where a.con_id=b.con_id
 group by a.con_id,;

 ---------- ---------- ----------
 1 CDB$ROOT     2106
 3 JON12PDB1     2105

This view lets you see the CDB and enclosed PDB tables when connected to the CDB.  However, it does not work the same when connected to the PDB:

SQL> alter session set container=jon12pdb1;

Session altered.

SQL> select a.con_id,, count(*)
 from cdb_tables a, v$containers b
 where a.con_id=b.con_id
 group by a.con_id,;
 ---------- ---------- ----------
 3 JON12PDB1     2105


Now that you are on the PDB, the view only lets you see tables associated with that particular PDB.

Here are some other admin views available:

V$CONTAINERS – Contains information on the containers in the database including the ROOT and all PDBs.

V$PDB_INCARNATION – Provides information on incarnations specific to given PDBs.

V$PDBS – Provides information on PDBs within the database

DBA_CONTAINER_DATA – Displays various container data for container data objects.

DBA_HIST_PDB_INSTANCE – Contains workload repository records for PDBs

DBA_PDB_HISTORY – Contains the history of a PDB. Only visible from within a specific PDB.

DBA_PDB_SAVED_STATES – Contains the saved state of a PDB.

DBA_PDBS Contains a list of the PDBs in the database. Only visible from the ROOT container.

PDB_ALERTS Provides information on specific PDB alerts.

PDB_PLUG_IN_VIOLATIONS Provides information on incompatibilities between the CDB and the PDB.


We’ve covered quite a bit of territory in this post.  Next up, cloning and migrating PDBs