C File Conversions

This appendix contains these topics:

Section C.1, "Scheduler Conversions,"
Section C.2, "Main Conversions,"
Section C.3, "JDEINSTAL Subsystem,"
Section C.4, "JDEINSTAL Job Queues,"
Section C.5, "File Conversion Job Queues,"
Section C.6, "Disk Utilization - Work With Disk Status,"
Section C.7, "CPU Utilization - Work With System Status,"
Section C.8, "Summary."

There are two types of conversions for A9.4:

Scheduler Conversions
Main File Conversions

C.1 Scheduler Conversions

There are two reasons for scheduler conversions.

Program conversions where the data in the file needs to be manipulated.
Data files that must convert first because the file is used in another conversion later in the process. A schedule is set up and delivered in the JDEINSTAL library that determines the order in which scheduler conversions run.

C.1.1 Scheduler Types

There are four types of scheduler conversions:

Control File Conversions (CTF) are for files that are the technical foundation of JD Edwards World software such as Data Dictionary and Menus. These conversions occur first in the process as other conversions and the control file merges are dependent on the files already being converted.
Application Control File Conversions (ACF) are for files considered application control files such as Company Constants (F0010). Some other application conversion programs (BIG, LTL) are dependent on these files already being converted.
Large File Conversions (BIG) are for master and transaction files that contain large volumes of data. An example is the General Ledger File (F0911). Most of the large file conversions have been converted to SQL which considerably reduces the conversion time.
Small File Conversions (LTL) are for secondary data files where the files generally contain small amounts of data.

C.2 Main Conversions

Main conversions are run for files where one of the following has changed and there is no scheduler record for the file:

record format
file keys
select/omit criteria

These files do not require special processing of data; have no dependency on other files. Physical files, logical files that have key or select/omit criteria changes and join logical files with changes can run through the main conversion process.

Main file conversions use CHGPF which is considerably faster than CPYF which was used in previous upgrades.

As a default, the main conversions run as single threaded jobs. Batch job performance of file conversions is not an issue for most customers. However, if many of your files have millions of records, the following information may help you fine tune the main conversion process.

Many articles, etc. have been written through the years about machine tuning to optimize batch performance and throughput. Experts and distinguished IBM engineers, such as Rick Turner and Dick Bains, and folks in the IBM Teraplex Center make careers from just this one topic. The articles, discussions, and range of performance monitoring tools are aimed at data collection, analysis, and reconfiguring machine resources to optimize batch performance for repeatable jobs on a specific machine configuration. A software upgrade (control file merges and database conversions) presents a different perspective to the performance discussion because it is a one time job, expected to run across the full range of machine models, configurations, and over databases ranging from minimal in size, to those that exceed terabytes.

Within that context, every plausible question regarding performance factors of an upgrade can be answered with - it depends. This paper explains the subsystem definition, job queues, and methodology of file conversion execution, and one adjustable factor you can make determined by a couple observations.

C.3 JDEINSTAL Subsystem

The JDEINSTAL library contains the work management components, subsystem, job queues, job descriptions, etc. used by the batch drivers, and individual file conversion drivers during an upgrade. The definition and activity levels of the components are depicted here.

JDEINSTAL Subsystem

Maximum Active Jobs in subsystem - 20

Storage Pool ID: 1

Storage Size: *BASE

JdeMonitor Job

Step-1/2 DriversMaxAct-1Each Step-2 job loads up the file conversion job queues.	"Big" FileConversionsMaxAct-1	"Little" FileConversionsMaxAct-3	"Main" FileConversionsMaxAct-1	Logical File ConversionsMaxAct-6	Join FileConversionsMaxAct-3
JDEINSTAL	JDEBIGQ	JDELTLQ	JDEFILCNV	JDECRTLF	JDECRTJLF
Step-1 Job	BigFile1	LtlFile1	MainFile1	LglFile1	JoinFile1
Step-2 Job	BigFile2	LtlFile2	MainFile2	LglFile2	JoinFile2
Step-2 Job	BigFile3	LtlFile3	MainFile3	LglFile3	JoinFile3
	END_BIGS	LtlFile4	MainFile4	LglFile4	JoinFile4
	BigFile1	END_LTLS	MainFile5	LglFile5	END_JOINS
	BigFile2	LtlFile1	MainFile6	LglFile6	JoinFile1
	BigFile3	LtlFile2	END_MAINS	LglFile7	JoinFile2
	END_BIGS	LtlFile3	MainFile1	LglFile8	JoinFile3
		LtlFile4	MainFile2		JoinFile4
		END_LTLS	MainFile3		END_JOINS
			MainFile4
			MainFile5
			MainFile6
			END_MAINS

The maximum activity level for the subsystem is 20 jobs. The subsystem contains six job queues. The JDEMONITOR job is always running throughout the upgrade process and is submitted through the JDEINSTAL job queue. It is the message dispatcher for various parts of the Step-1 and Step-2 batch jobs, as well as the message handler for all the various file conversion drivers, and also updates the final conversion status for each spawned conversion job. It is not a controller or sequencer, but only a message dispatcher.

C.4 JDEINSTAL Job Queues

The JDEINSTAL job queue is also where the Step-1 and Step-2 drivers are submitted. The other job queues run the file conversions.

C.5 File Conversion Job Queues

Job queues JDEBIGQ, JDELTLQ, and JDEFILCNV are referred to as the "data pumps" - they convert the physical structure of the physical files. JDEBIGQ and JDELTLQ run scheduler conversion jobs which normally manipulate the files data as part of the file conversion process. Job queue JDEFILCNV is where physical files are converted that require no additional data manipulation or processing.

Job queue JDECRTLF rebuilds any changed logical files (indexes) after the physical file has been processed, and job queue JDECRTJLF is held until the data pump queues have completed all conversions for the current Step-2 job. The END_xxxx are the final jobs in the data pump queues and when all three indicate completion, the JDECRTJLF job queue is released. Join logical file rebuilds begin and when its END_xxxx comes active, holds the job queue. This provides the needed boundary control for multi-plan upgrades. It also performs a look ahead at the next plan to determine if all three data pumps have completed or not. If not, the queue is held, is the next Step-2 job has completed all physical file conversions, the JDECRTJLF continues processing join rebuilds for the next environment.

The JDEFILCNV job queue can be increased to run multiple conversions simultaneously depending upon available system resources.

The rule of thumb is one for each CPU, or until disk utilization and activity levels are achieved as described below.

C.6 Disk Utilization - Work With Disk Status

The data pumps use little CPU as file conversions, while using system memory, are mostly I/O intensive. The command WRKDSKSTS shows statistics on disk activity. Observe the "%busy" column. Maximum disk arm utilization is achieved around 40%. The recommendation is to achieve a level close to 30% over an elapsed time of around 30 seconds. Use F10 to restart the statistics use F5 to refresh the display until you reach an elapsed time of 30 seconds.

Description of image137.gif follows
Description of the illustration image137.gif

Note:

This display is just representation of the Disk Status Panel and is not a representative example of actual system activity.

If the %busy column is low, you can increase the activity level of the "MAIN" conversion job queue and allowing more conversion jobs to run. Use the following command to do so:

CHGJOBQE SBSD(JDFINS/JDEINSTAL) JOBQ(JDFINS/JDEFILCNV)

MAXACT(2)

Increasing the activity level for the job queue will allow additional conversion jobs to run simultaneously. Increment in small graduations, and continue checking the disk activity as stated above.

It important to note that the maximum active jobs allowed in the subsystem is configured at 20. This translates into a maximum activity level for job queue JDEFILCNV to 8, after all Step-2 jobs have completed. (job queue JDEINSTAL is empty).

C.7 CPU Utilization - Work With System Status

The command WRKSYSSTS shows a range of statistical information on CPU utilization, DASD, memory pool sizes, system activity and page faults. If you are knowledgeable, and comfortable with machine tuning you can adjust memory storage pool sizes and allocate more memory to *BASE.

Description of image138.gif follows
Description of the illustration image138.gif

Remember that four conversion "programs" could be active at any point in time, one from job queue JDEBIGQ and three from job queue JDELTLQ. All remaining activity will be system commands (CHGPF/CPYF) and index rebuilds (CRTLF). Although each CHGPF/CPYF and index build job has a driver program, the resources of those drivers are minimal - as is the overhead of the JDEMONITOR itself.

C.8 Summary

In summary, when migrating large amounts of data, the bottleneck will be disk speed and utilization. So throwing more memory at the process may result in little if any gain at all, unless your batch pools are already undersized.

Performance tuning is an art geared toward maximizing job throughput on a machine with fixed, known resources. The upgrade batch jobs must run on all System i models comprised of unlimited variations of memory, DASD, etc. and resource allocations. Therefore, the assumptions and configurations we make are conservative.