Varying Priority Scheduling Example

This example assumes default preferences. It also assumes that each job executes in full without yielding.

The map/reduce deployment in the first diagram has a Queues value of 2 and 3. The map/reduce deployment in the second and third diagrams has a Concurrency Limit value of 2.

Note:

With the introduction of SuiteCloud Processors, the Queues field is replaced with the Concurrency Limit field on map/reduce deployments.

The second and third diagrams vary from Default Priority Scheduling Example as follows:

• In the second diagram, jobs 15 and 18 are high priority. Job 4 is low priority. The remaining jobs are standard priority.

• In the third diagram, job 4 is low priority. Jobs 15 and 18 in the third diagram are standard priority.

Diagram Key for Varying Priority Scheduling Example

• Black circles are scheduled script jobs. Each scheduled script is processed by a single job.

• Red circles are map/reduce jobs. In this example, all of the map/reduce jobs belong to a single map/reduce task. The map/reduce jobs include a label that indicates the map/reduce stage being processed.

• Circles with blue fill are high priority. The circles with gray fill are low priority. The circles with white fill are standard priority.

• Jobs in columns labeled Q1, Q2, and Q3 are for deployments that still use queues. The jobs in columns without a label are for deployments that no longer use queues.

• P1, P2, and P3 represent the processors that are available in the processor pool.

  Note:

  The processors are labeled for this example. Although technically, processors aren't distinguished as individual entities. Also, a real account would never have only three processors available to it. The minimum number of processors available to an account with SuiteCloud Plus is listed at SuiteCloud Plus Settings.

• Numbers within each circle represent the time stamp on the job submission, with 1 being the first job submitted. The example also uses this number as a unique identifier for each job.

• Arrows show dependencies. For example, in the Scheduling Queues diagram, job 2 can't start until job 1 is complete.

  • Black arrows represent queue dependencies. Jobs within a queue must be processed in the order they were submitted.

  • Red arrows represent dependencies imposed by the map/reduce algorithm.

  • Green arrows indicate access to processors.

Scheduling Queues Only	SuiteCloud Processors – All Queues Removed	SuiteCloud Processors – Some Queues Removed

Time Slot	Scheduling Queues Only	SuiteCloud Processors – All Queues Removed	SuiteCloud Processors – Some Queues Removed
101	Jobs 1, 4, and 6 start.	Jobs 15, 18, and 1 start. Jobs 15 and 18 are designated as high priority so they are assigned to processors first (in the order of submission). Job 1 is the first standard priority job submitted so it is the next job assigned to a processer.	Jobs 1, 5, and 15 start. Job 4 is designated low priority, so it is processed after all other available standard priority jobs. All other standard priority jobs submitted before job 15 have dependencies that are blocking them.
102	Job 6 cancels job 5. The getInput stage can't be processed by multiple jobs, so the first getInput job to start (job 6) automatically cancels all others in the queues.	—	—
103	Job 6 completes. The next job in Q3, job 8, starts.	Job 18 completes. Job 2 starts.	Job 5 completes. There's no job 6 with SuiteCloud Processors. In the Scheduling Queues example, jobs 5 and 6 are both getInputstage jobs. The getInput stage can't be processed by multiple jobs. Since this map/reduce deployment isn't using queues, multiple getInput stage jobs are no longer needed. Job 7 starts.
104	Job 8 completes. The map stage can be processed by multiple jobs. But since the other map stage job, job 7, is unable to start (the job utilizing its processor, job 4, isn't yet complete), job 8 completes the entire map stage. Job 7 is no longer needed, so it is canceled. The next job in Q3, job 10, starts.	Job 15 completes. Job 3 starts.	Jobs 7 and 15 complete. Since job 7 completes the entire map stage, job 8 is no longer needed. Job 8 is canceled. Jobs 9 and 16 start.
105	Job 10 cancels job 9. The shuffle stage can't be processed by multiple jobs, so the first shuffle job to start (job 10) automatically cancels all others in the queues.	Job 2 completes. Job 5 starts. Job 4 is designated low priority, so it is processed after all other available standard priority jobs.	—
106	Job 1 completes. The next job in Q1, job 2, starts.	Jobs 1 and 3 complete. Jobs 16 and 17 start.	Job 1 completes. The next job in Q1, job 2, starts.
107	Job 4 completes. Q2 is blocked. The next job in Q2, job 11, can't start. It is dependent on job 10, and job 10 isn't complete.	Job 5 completes. There's no job 6 with SuiteCloud Processors. In the Scheduling Queues example, jobs 5 and 6 are both getInputstage jobs. The getInput stage can't be processed by multiple jobs. Since this map/reduce deployment isn't using queues, multiple getInput stage jobs are no longer needed. Job 7 starts.	Job 16 completes. Job 17 starts.
108	Job 2 completes. The next job in Q1, job 3, starts.	Job 7 completes. Since job 7 completes the entire map stage, job 8 is no longer needed. Job 8 is canceled. Job 9 starts.	Job 2 completes. The next job in Q1, job 3, starts.
109	Job 10 completes. Job 11 can now start and Q2 is no longer blocked. The next job in Q3, job 12, starts. These are both reduce stage jobs. The reduce stage can be processed by multiple jobs.	Jobs 16 and 17 complete. Job 4 starts. Job 4 is designated as low priority, but all the remaining standard priority jobs are dependent on job 9.	Job 9 completes. There's no job 10 with SuiteCloud Processors. In the Scheduling Queues example, jobs 9 and 10 are both shuffle stage jobs. The shuffle stage can't be processed by multiple jobs. Since this map/reduce deployment isn't using queues, multiple shuffle stage jobs are no longer needed. Job 11 starts.
110	Job 3 completes. The next job in Q1, job 16, starts.	—	Jobs 3 and 17 complete. There are no additional jobs to process in Q1. Job 12 starts. The next job in Q3, Job 18, starts.
111	Job 11 completes. Q2 is blocked. The next job in Q2, job 13, can't start. It is dependent on job 12, and job 12 isn't complete.	—	—
112	Job 12 completes. Job 13 can now start and Q2 is no longer blocked. The next job in Q3, job 14, can also start. But these are both summarize stage jobs, and the summarize stage can't be processed by multiple jobs. Job 13 starts first, so job 14 is canceled. Since job 14 is canceled, the next job in Q3, job 15, starts	—	Jobs 11, 12 and 18 complete. There are no additional jobs to process in Q3. Jobs 13 and 4 start.
113	Job 16 completes. The next job in Q1, job 17, starts.	Job 9 completes. There's no job 10 with SuiteCloud Processors. In the Scheduling Queues example, jobs 9 and 10 are both shuffle stage jobs. The shuffle stage can't be processed by multiple jobs. Since this map/reduce deployment isn't using queues, multiple shuffle jobs are no longer needed. Jobs 11 and 12 start.	—
114	Job 13 completes. There are no additional jobs to process in Q2.	—	Job 13 completes. There's no job 14 with SuiteCloud Processors. In the Scheduling Queues example, jobs 13 and 14 are both summarize stage jobs. The summarize stage can't be processed by multiple jobs. Since this map/reduce deployment isn't using queues, multiple summarize stage jobs are no longer needed.
115	Job 15 completes. The next job in Q3, job 18, starts.	Jobs 4 and 12 complete.	—
116	Job 17 completes. There are no additional jobs to process in Q1.	Job 11 completes. Job 13 starts.	—
117	Job 18 completes. There are no additional jobs to process in Q3.	—	—
118	—	Job 13 completes. There's no job 14 with SuiteCloud Processors. In the Scheduling Queues example, jobs 13 and 14 are both summarize stage jobs. The summarize stage can't be processed by multiple jobs. Since this map/reduce deployment isn't using queues, multiple summarize stage jobs are no longer needed. There are no additional jobs to process.	Job 4 completes. There are no additional jobs to process.

Related Topics

• SuiteCloud Processors Priority Scheduling Examples
• Default Priority Scheduling Example

General Notices