PSUB Jobs

Check the Failure Text in the Submitted Batch Jobs Window

If a problem arises while you are running PSUB, you should first review the Failure Text field of the Submitted Batch Jobs window.

To check this field for your batch job:

Open the Submitted Batch Jobs window: select Admin, then PSUB/Reports Jobs, and then Batch Jobs.
Locate the relevant Batch Job ID number.
Check the Execution Status of the job. If there is an entry in the Failure Text field, make a note of its contents

If the failure text does not help you to resolve the problem, see Check the PSUB Log Files. If your batch job is hanging because the batch queue is full, see If Batch Jobs Hang and the Batch Queue Is Full.

Parent topic: PSUB Jobs

Check the PSUB Log Files

The PSUB process log files are cumulative, text-based descriptions of PSUB activity. These files are very helpful when you are troubleshooting problems with PSUB Process log files can include time stamped entries for:

Error messages returned by the PSUB process
All jobs submitted by the user; the entry may include each job's:
- Message ID
- Batch_job_ID
- User name

Naming Convention

On both UNIX and Windows systems, PSUB process log file names are in the form:

rxcpsd_product_instance code_environment_1.log

On UNIX systems, there is a second process log file. Whenever you examine the "*_1.log" process log file on UNIX systems, you should also check this second file to see if it contains relevant entries. Its name is identical to the first log file, except that it has an "_2" suffix, rather than "_1". So the second UNIX process log file name is in the form:

rxcpsd_product_instance code_environment_2.log

The "*_2.log" process log files contain error and warning messages that are generated by certain UNIX commands that the PSUB process executes (e.g., non-background commands). These commands are not present in the PSUB service on Windows. Therefore, Windows systems only generate "*_1.log" process files.

Verbose vs. Nonverbose Mode

The [verbose|nonverbose] argument must be included when the PSUB startup command, rxcpsdps, is executed. Oracle recommends that you start PSUB in verbose mode because the process logs that are generated:

Contribute to efficient troubleshooting
Do not pose significant disk space concerns

On UNIX systems, rxcpsdps is 'wrapped' in the OPA script start_psub. By default, the start_psub script executes rxcpsdps in verbose mode.

On Windows systems, the PSUB service requires that you explicitly provide the [verbose|nonverbose] argument. See Managing the PSUB Process.

If you cannot check the Failure text or the .out and .log files because the batch queue is hung, see If Batch Jobs Hang and the Batch Queue Is Full.

Verify that the process log files for the relevant Batch Job ID exist.

Review the .out and .log files. The following table summarizes these files. Make a note of any error messages.

Name	Description	Directory
l`batch_job_id`.log	Log file of a specific job	RXC_LOG
o`batch_job_id`.out	Output file of a specific job	RXC_LOG
`*`.log	Log file of the PSUB process (see Viewing Log and Output Files on the Screen for naming convention)	RXC_CENTRAL_LOG

Check the job-specific log and output files first, then the central log file. In the central log file, search for the batch job ID number to find the relevant entry. See if the database and code environment settings are correct.

Parent topic: PSUB Jobs

If Batch Jobs Hang and the Batch Queue Is Full

If all PSUB jobs hang (that is, they do not reach a completed execution status), and the batch queue is full, attempt to clear the queue and submit a single job to PSUB. If a problem then occurs with a single job, it may be clearer which area is causing the problem. The method for clearing the queue is either: stop all of the hung batch jobs (on UNIX systems) or stop the PSUB service (on Windows systems).

The queue may become full and PSUB jobs may hang under the following circumstances:

PSUB is waiting, either for an operating system resource, or a database resource
the operating system is overloaded, for example, a built-in limitation, such as maximum number of processes, is exceeded.

For more information , see:

Parent topic: PSUB Jobs

Stopping Batch Jobs on UNIX Systems

This section describes how to stop batch jobs on UNIX systems. See also Starting and Stopping PSUB Manually in UNIX.

Stopping an Individual Job

To stop an individual batch job:

Navigate to Admin, PSUB/Reports Jobs, and Batch Jobs).
Locate and select the row associated with the relevant Batch Job ID.
Click the Stop button.

Stopping all Jobs

On UNIX servers, a series of hanging jobs can cause the batch queue to become full. When the queue fills and is backlogged with hanging jobs, all jobs are eventually given an execution status of SUBMIT_FAILED or SUBMITTED. If this type of problem occurs frequently, it may be advantageous to fine-tune the existing queues or add more queues.

Administrator-level Troubleshooting

If, after trying all relevant solutions, you are unable to stop the jobs on a UNIX server, contact your Administrator so that she may use the solutions described here.

Only Administrator-level personnel should attempt to stop PSUB jobs using these solutions. Use these strategies, in the order they are listed, to stop PSUB jobs.

Use the stop_psub utility.
Identify and then stop the processes that are hanging:
1. To identify the process that is hanging, use either:
  
  ps -ef|grep opapps
  
  or
  
  ps -ef|grep userid
2. To stop all of the hanging processes that you identified in Step 2a, use this command:
  
  kill -9 pid
Log in as opapps and, at a command prompt, enter:

at -l [-q]

This command lists all of the jobs that are currently in all of the queues. Each job has a unique ID number.

If there are jobs pending in the queue, the following command, which uses the unique ID number to remove specific jobs from the queue, may be of use:

at -r id

If you are able to stop all PSUB jobs, stop and then restart the PSUB process and submit one job. If it hangs, try to isolate whether one particular module is the cause or if any PSUB job hangs, regardless of module.
If you are unable to determine a module that is causing the problem and jobs are still hanging, the only recourse is to reboot the computer.

Parent topic: If Batch Jobs Hang and the Batch Queue Is Full

Stopping Batch Jobs on Windows

On Windows systems, Oracle recommends that you:

Stop the PSUB service.
Shut down any databases, if any, that are on the computer.
Reboot the computer.
Start the PSUB service.

See Starting and Stopping PSUB Manually in Windows.

Parent topic: If Batch Jobs Hang and the Batch Queue Is Full

Determining if PSUB Is Running for a Database

To find out if a PSUB process is listening to a particular database, and if it is, what code environment it is running in, enter this query:

SQL> select host, code_environment, stop_ts
  2  from psub_process_log
  3  where start_ts = (
  4  select max (start_ts) from psub_process_log;

This query returns the:

computer on which PSUB was last started against the database
code environment
state of the process:
- if stop_ts is null, the PSUB process is currently active
- if stop_ts is not null, the PSUB process is stopped.
- new code list

Note:

As an alternative, to find out on which computer PSUB is set up, query one of the following:

OCL_STATE local reference codelist, SERVER_NAME setting
RXC.PSUB_PROCESS_LOG table

Use this information, in conjunction with the operating system-specific instructions, below, to determine if PSUB is running on a given computer.

For more information , see:

Parent topic: PSUB Jobs

What PSUB Processes Are Running on a Given UNIX Server?

Use this command to find out if PSUB is running on particular UNIX server:

% ps -ef | grep -i rxcpsdps

The process search command, ps, returns descriptions of the PSUB processs that are currently running. Each row that is returned represents one PSUB process running on the server. Each process has a unique product_instance and code_environment pair. The format of the response to the process search command listed above is:

rxcpsdps [verbose|nonverbose]product_instance code_environment

Example C-1 Using the ps Command

Two examples of ps command usage:

opapps 15685 1 0 Apr 04 ? 0:00 rxcpsdps verbose example_db ssuneja_oc40_sun
opapps  4143 1 0 Apr 02 ? 0:00 rxcpsdps verbose example_db 40102_8163

Parent topic: Determining if PSUB Is Running for a Database

Is PSUB Running on a Given Windows Server?

Use this procedure to find out if the PSUB service is running on a given Windows server.

Open the Control Panel.
Double-click the Services icon.
In the Services window, note the status of the PSUB service with the relevant database name. The status will be "Started" if the service is running.

Parent topic: Determining if PSUB Is Running for a Database

Troubleshooting PSUB Based on the Batch Job's Execution Status

Execution status as reported in the Submitted Batch Jobs form is shown below. You can take various actions depending on execution status.

Value	Meaning
ENTERED	The user has requested a job submission.
SUBMITTED	The process submitted the job to the batch queue; it may be pending.
SUBMIT_FAILED	The process attempted to submit the job to the batch queue but failed.
STARTED	The job is executing on the batch queue.
SUCCESS	The job completed with SUCCESS status.
FAILURE	The job completed with FAILURE status. Look at the Failure Text in the Submitted Batch Jobs window for possible reasons.

Examine the Submitted Batch Jobs window, look for the Execution Status and Failure Text for your Batch Job ID, and take one of the following actions, depending on the circumstances.

Parent topic: PSUB Jobs

ENTERED

If the Execution Status of your batch job remains at ENTERED, perhaps:

The PSUB process is not running on the server, or it is not receiving the request from the client.
The corresponding Oracle user's operating system account does not exist.
The Advanced Queue is not started; see PSUB Remains at Entered Status

Parent topic: Troubleshooting PSUB Based on the Batch Job's Execution Status

SUBMITTED

If the Execution Status of your batch job remains at SUBMITTED, perhaps:

The job is pending in the batch queue, or the batch queue is stopped.
The opapps account does not have Write permission for the user's RXC_LOG directory on the PSUB server.
A log file exists with the same number as the one for the submitted job. This is a rare situation. Delete the old log file and resubmit the job.
In the case of a PSUB job that stays in SUBMITTED status even though the PSUB process is up and running, if your .log file says:
```
ERROR:Error while connecting:
      ORA-01017: invalid username/password; logon denied

Exiting...
```
Edit sqlnet.ora (in drive:\app\oracle\product\11.2.0.2.0\NETWORK\ADMIN):
- UNIX: Comment out the following line (add # at the beginning of the line) and save.
```
# SQLNET.AUTHENTICATION_SERVICES = (NTS)
```
- Windows: Make sure the same line in sqlnet.ora is not commented out (if there is a pound sign (#) at the beginning of the line, remove it and save):
```
SQLNET.AUTHENTICATION_SERVICES = (NTS)
```
If the PSUB process is still running you can now resubmit the PSUB jobs. Otherwise, stop and then start the PSUB process.

Parent topic: Troubleshooting PSUB Based on the Batch Job's Execution Status

SUBMIT_FAILED

If the Execution Status of your batch job is SUBMIT_FAILED, examine the Failure Text. If this action gives no possible cause, perhaps:

The ssh command cannot be executed by the OPAPPS user. Check that the host name in the /etc/hosts.equiv file is the official name of the host as specified in /etc/hosts.
The user's password is not correct.
The batch queue does not exist. Check the Long Value of the BATCH QUEUE NAME local reference codelist.
The batch queue is in a stopped state.

Parent topic: Troubleshooting PSUB Based on the Batch Job's Execution Status

STARTED

If the Execution Status of your batch job remains at STARTED, perhaps:

The job is executing and waiting for some resource.
The job is hung.

Parent topic: Troubleshooting PSUB Based on the Batch Job's Execution Status

FAILED

If the Execution Status is FAILED, examine the Failure Text. If this action gives no possible cause, perhaps:

The report or command exited with error status.
The report or executable file does not exist.
The print command exited with failure status, because, for example, the specified print queue does not exist.

Parent topic: Troubleshooting PSUB Based on the Batch Job's Execution Status

Other Items to Check

Make sure the Long Value of the SERVER_OS entry in the OCL_STATE local reference codelist is correct for your operating system. You can enter one of the following values:

NT — Indicates the server is running one of the Windows operating systems currently supported by Oracle Clinical.
UNIX — Indicates the server is running the UNIX operating system.

Check that the SERVER_NAME in OCL_STATE is set to the database/PSUB server.

Note:

On UNIX systems, the Long Value of the SERVER_NAME entry (also in OCL_STATE) must be in lowercase letters.

Check that your RXC_LOG is correctly defined or modify the user's log directory via the menu path Admin, then Users, and Oracle Accounts.

Note:

You may get an error message on the Windows server about the Kernel32.DLL initialization because of too many jobs running at the same time. Stop the unwanted processes, including cmd.exe and pslaunch.exe, using the Task Manager. If the error happens frequently, stop the PSUB service, reboot the Windows server, and restart the PSUB service. This should fix the problem.

Parent topic: Troubleshooting PSUB Based on the Batch Job's Execution Status

Handling PSUB Failures that Return "Fatal two-task communication protocol" Error

If you submit a PSUB job that fails and returns a "Fatal two-task communication protocol" error (this failure is sometimes followed by the "End-of-communication-channel" in the core dump information on your console), you might have the environment variable NLS_LANG set inconsistently with the settings in the database.

To verify that the environment variable NLS_LANG matches the actual database settings:

Execute the following query:

SQL> select parameter, value from V$NLS_PARAMETERS
     where parameter in ('NLS_LANGUAGE','NLS_TERRITORY','NLS_CHARACTERSET');

Open opa_settings and search for the following string:
```
db_env_setting:database_name:NLS_LANG
```
1. If you do not find this string, add a line with the following syntax:
```
db_env_setting:database_name:NLS_LANG:NLS_LANGUAGE_NLS_TERRITORY.NLS_CHARACTERSET
```
  where NLS_LANGUAGE_NLS_TERRITORY.NLS_CHARACTERSET are the values returned in Step 1.
2. If you find the string, correct the values to match the values in Step 1 in the following syntax:
```
db_env_setting:database_name:NLS_LANG:NLS_LANGUAGE_NLS_TERRITORY.NLS_CHARACTERSET
```
  where NLS_LANGUAGE_NLS_TERRITORY.NLS_CHARACTERSET are the values returned in Step 1.

Parent topic: PSUB Jobs

Handling PSUB Failure that Returns "Illegal use of PSLAUNCH…" Error

In a UNIX environment, you may see the following error when you submit a PSUB job (3GL or PLSQL):

Illegal use of PSLAUNCH by user. Job ID=batch_job_ID. Exiting...

For more information , see:

Parent topic: PSUB Jobs

Verify PSUB Account Uses C Shell

This error can occur if the PSUB user is not using the C Shell (csh). The default shell gets set up when you create the user account. For example, on Linux, the bash shell is set by default.

Verify that all user accounts that run PSUB jobs are configured to use the C Shell (csh).

Parent topic: Handling PSUB Failure that Returns "Illegal use of PSLAUNCH…" Error

Modify launch.ps

If the error continues to occur after you verify that PSUB account uses C Shell, modify $RXC_PSUB/launchps.sh as follows:

Log on to the UNIX computer on which the PSUB process is running, as owner of the file launchps.sh. (The owner is usually OPAPPS.)
Set environment variables for the database name and code environment; see Setting Environment Variables on the Command Line
Change to the $RXC_PSUB directory.
Edit launchps.sh by adding the following line immediately before the pslaunch command:
```
sleep 2 
pslaunch $4 $5 $6 $7 $3 $8
```
This command introduces a 2-second delay before the system calls pslaunch. You may increase the delay if the error continues to occur.

Parent topic: Handling PSUB Failure that Returns "Illegal use of PSLAUNCH…" Error

Tracking Previous PSUB Process Connections

To find out specific information about PSUB connections to a given database, query the table RXC.PSUB_PROCESS_LOG. This will return the:

instance
environment
time a PSUB process started
time a PSUB process stopped.

Example C-2 Host and Code Environment

This query will return the host and code environment for the last time PSUB was started against the database.

SQL> SELECT start_ts, host, code_environment, server_os
  2  from psub_process_log
  3  where start_ts= (select max(start_ts) from psub_process_log);

Example C-3 Start and Stop Time Stamps

This example lists, in chronological order, all start and stop time stamps of PSUB processes.

SQL> SELECT start_ts, stop_ts, host, code_environemnt
  2  from psub_process_log order by 1;

Parent topic: PSUB Jobs

PSUB Fails to Start

If PSUB does not start; for example, after installing or upgrading:

Check one line in sqlnet.ora (in drive:\app\oracle\product\11.2.0.2.0\NETWORK\ADMIN):
- UNIX: Edit sqlnet.ora by commenting out the following line (add # at the beginning of the line) and save it.
```
# SQLNET.AUTHENTICATION_SERVICES = (NTS)
```
- Windows: Make sure the same line in sqlnet.ora is not commented out (if there is a pound sign (#) at the beginning of the line, remove it and save):
```
SQLNET.AUTHENTICATION_SERVICES = (NTS)
```
Locate and ensure that these lines in init.ora are not commented out and the values are as specified:

remote_os_authent=null

os_authent_prefix=""

(a null string)
Shutdown the database.
Start the database.
Start PSUB.

Parent topic: PSUB Jobs

PSUB Remains at Entered Status

If PSUB remains at Entered status, you may need to manually restart the AQ process:

Stop PSUB, see Starting and Stopping PSUB.
Log in as sysdba.

Enter:

EXEC DBMS_AQADM.START_QUEUE('RXC.PSUB_REPLY_QUEUE',TRUE,TRUE); 
EXEC DBMS_AQADM.START_QUEUE('RXC.PSUB_SEND_QUEUE',TRUE,TRUE);

Restart PSUB.
Rerun the job.

Parent topic: PSUB Jobs

Troubleshooting PSUB on a Windows Database

If you have difficulty starting PSUB on a Windows database after upgrading to or installing Oracle Clinical:

Open the sqlnet.ora file and confirm that following line is not commented (that is, there is no '#' at the beginning of the line). If there is, uncomment the line (remove the #):
```
sqlnet.authentication_service=(NTS)
```
Attempt to start PSUB.

If PSUB fails to start:

Open the init.ora file. Ensure that the following lines are not commented out and have the values specified. If not, uncomment and/or change the values.
```
remote_os_authen=NULL
os_authent_prefix="OPS$"
```
Shut down any databases on the Windows server, then start the databases.

Parent topic: PSUB Jobs