Default Priority Scheduling Example

This example assumes default priority settings (standard priority) and default preferences. It also assumes that each job executes in full without yielding.

The map/reduce deployment in the first diagram has a Queues value of 2 and 3. The map/reduce deployment in the second and third diagrams has a Concurrency Limit value of 2.

Note:

With the introduction of SuiteCloud Processors, the Queues field is replaced with the Concurrency Limit field on map/reduce deployments.

Diagram Key for Default Priority Scheduling Example

• Black circles are scheduled script jobs. Each scheduled script is processed by a single job.

• Red circles are map/reduce jobs. In this example, all of the map/reduce jobs belong to a single map/reduce task. The map/reduce jobs include a label that indicates the map/reduce stage being processed.

• Jobs in columns labeled Q1, Q2, and Q3 are for deployments that still use queues. The jobs in columns without a label are for deployments that no longer use queues.

• P1, P2, and P3 represent the processors that are available in the processor pool.

  Note:

  Although processors are labeled in this example, they aren't technically individual entities. Also, a real account would never have only three processors available to it. The minimum number of processors available to an account with SuiteCloud Plus is listed at SuiteCloud Plus Settings.

• Numbers within each circle represent the time stamp on the job submission, with 1 being the first job submitted. The example also uses this number as a unique identifier for each job.

• Arrows show dependencies. For example, in the Scheduling Queues diagram, job 2 can't start until job 1 is complete.

  • Black arrows represent queue dependencies. Jobs within a queue must be processed in the order they were submitted.

  • Red arrows represent dependencies imposed by the map/reduce algorithm.

  • Green arrows indicate access to processors.

Scheduling Queues Only	SuiteCloud Processors – All Queues Removed	SuiteCloud Processors – Some Queues Removed

Time Slot	Scheduling Queues Only	SuiteCloud Processors – All Queues Removed	SuiteCloud Processors – Some Queues Removed
101	Jobs 1, 4, and 6 start.	Jobs 1, 2, and 3 start.	Jobs 1, 4, and 5 start.
102	Job 6 cancels job 5. The getInput stage can't be processed by multiple jobs, so the first getInput job to start (job 6) automatically cancels all others in the queues.	—	—
103	Job 6 completes. The next job in Q3, job 8, starts.	Jobs 2 and 3 complete. Jobs 4 and 5 start.	Job 5 completes. There's no job 6 with SuiteCloud Processors. In the Scheduling Queues example, jobs 5 and 6 are both getInputstage jobs. The getInput stage can't be processed by multiple jobs. Since this map/reduce deployment isn't using queues, multiple getInput stage jobs are no longer needed. Job 7 starts.
104	Job 8 completes. The map stage can be processed by multiple jobs. But since the other map stage job, job 7, is unable to start (the job utilizing its processor, job 4, isn't yet complete), job 8 completes the entire map stage. Job 7 is no longer needed, so it is canceled. The next job in Q3, job 10, starts.	—	Job 7 completes. Since job 7 completes the entire map stage, job 8 is no longer needed. Job 8 is canceled. Job 9 starts.
105	Job 10 cancels job 9. The shuffle stage can't be processed by multiple jobs, so the first shuffle job to start (job 10) automatically cancels all others in the queues.	Job 5 completes. There's no job 6 with SuiteCloud Processors. In the Scheduling Queues example, jobs 5 and 6 are both getInputstage jobs. The getInput stage can't be processed by multiple jobs. Since this map/reduce deployment isn't using queues, multiple getInput stage jobs are no longer needed. Job 7 starts.	—
106	Job 1 completes. The next job in Q1, job 2, starts.	Job 7 completes. Since job 7 completes the entire map stage, job 8 is no longer needed. Job 8 is canceled. Job 1 completes. Jobs 9 and 15 start. Jobs 11 - 13 are dependent on job 9. There's no job 10 or job 14 with SuiteCloud Processors. In the Scheduling Queues example, jobs 9 and 10 are both shuffle stage jobs and jobs 13 and 14 are both summarize stage jobs. The shuffle and summarize stages can't be processed by multiple jobs. Since this map/reduce deployment isn't using queues, multiple shuffle and summarize stage jobs are no longer needed.	Job 1 completes. The next job in Q1, job 2, starts.
107	Job 4 completes. Q2 is blocked. The next job in Q2, job 11, can't start. It is dependent on job 10, and job 10 isn't complete.	—	Job 4 completes. The first job in Q3, Job 15, starts.
108	Job 2 completes. The next job in Q1, job 3, starts.	—	Job 2 completes. The next job in Q1, job 3, starts.
109	Job 10 completes. Job 11 can now start and Q2 is no longer blocked. The next job in Q3, job 12, starts. These are both reduce stage jobs. The reduce stage can be processed by multiple jobs.	Jobs 4 and 15 complete. Jobs 16 and 17 start.	Job 9 completes. There's no job 10 with SuiteCloud Processors. In the Scheduling Queues example, jobs 9 and 10 are both shuffle stage jobs. The shuffle stage can't be processed by multiple jobs. Since this map/reduce deployment isn't using queues, multiple shuffle stage jobs are no longer needed. Job 11 starts.
110	Job 3 completes. The next job in Q1, job 16, starts.	—	Jobs 3 and 15 complete. There are no additional jobs to process in Q1. Jobs 12 and 16 start.
111	Job 11 completes. Q2 is blocked. The next job in Q2, job 13, can't start. It is dependent on job 12, and job 12 isn't complete.	Job 9 completes. Job 11 starts.	—
112	Job 12 completes. Job 13 can now start and Q2 is no longer blocked. The next job in Q3, job 14, can also start. But these are both summarize stage jobs, and the summarize stage can't be processed by multiple jobs. Job 13 starts first, so job 14 is canceled. Since job 14 is canceled, the next job in Q3, job 15, starts	Jobs 16 and 17 complete. Jobs 12 and 18 start.	Jobs 11 and 12 complete. Jobs 13 and 17 start.
113	Job 16 completes. The next job in Q1, job 17, starts.	—	Job 16 completes. The next job in Q3, Job 18, starts.
114	Job 13 completes. There are no additional jobs to process in Q2.	Jobs 11, 12, and 18 complete. Job 13 starts	Job 13 completes. There's no job 14 with SuiteCloud Processors. In the Scheduling Queues example, jobs 13 and 14 are both summarize stage jobs. The summarize stage can't be processed by multiple jobs. Since this map/reduce deployment isn't using queues, multiple summarize stage jobs are no longer needed.
115	Job 15 completes. The next job in Q3, job 18, starts.	—	Jobs 17 and 18 complete. There are no additional jobs to process.
116	Job 17 completes. There are no additional jobs to process in Q1.	Job 13 completes. There are no additional jobs to process.	—
117	Job 18 completes. There are no additional jobs to process in Q3.	—	—

Related Topics

• SuiteCloud Processors Priority Scheduling Examples
• Varying Priority Scheduling Example

General Notices