Configuration Options for Handling Map/Reduce Interruptions
A map/reduce script can be interrupted at any time. For example, a disruption to the application server immediately stops the script’s execution. Additionally, an uncaught error, although it does not cause the script to stop executing, stops the current function invocation, even if it is not complete.
The system includes two configuration options that let you fine-tune the overall response of your script to interruptions. For details, see the following sections:
retryCount
The retryCount option affects the map and reduce stages only. This option lets you configure your script to restart the map or reduce function for any key/pairs that were left in an uncertain state following an interruption. including application server restarts and uncaught errors.
The effect of this setting varies slightly depending on whether the script was disrupted by application server restart or an uncaught error. For example:
-
When an application server restart occurs, the script cannot identify the exact key-value pair that was being processed when the interruption occurred. However, the script can identify the pairs that had been flagged for processing but were not yet marked as complete. In the event of a retry, the script retries processing for all of these pairs.
-
When an uncaught error occurs, the script can identify the exact key-value pair that was being processed when the interruption occurred. When the retryCount option is being used, the script invokes the map or reduce function again for that specific key-value pair.
Valid values are 0 to 3. The number you choose for the retryCount setting is the number of retries permitted for each key-value pair that was left in an uncertain state after an interruption. For example, suppose you have retryCount set to 2, and an error interrupts a map job. In this case, the script would restart the function for the key-value pair that was being processed when the error was thrown. If the second function invocation for that pair was also interrupted by an error, the script would invoke the function for that key-value pair another time. However, if an error was thrown during the third attempt, the script would not retry processing again, because the retryCount setting permits only two retries. Later, when the job starts processing a different key-value pair, two retries are again available.
The retryCount setting is optional. You can set a value for it in the return statement of your map/reduce script, as described in Adding a Configuration Option to a Script.
If you do not add the retryCount option to your script, the behavior varies depending on whether the script was disrupted by an uncaught error or by an application server restart. For example:
-
When retryCount is not used and an application server restart interrupts the script, the system restarts processing for all key-value pairs that were left in an uncertain state.
-
When retryCount is not used and an error interrupts the map or reduce function, the system does not restart processing for the key-value pair that was left in an uncertain state.
See also System Response After an Uncaught Error and System Response After an Application-Server Disruption.
In the case of an uncaught error, the system’s overall response is also affected by the value of the exitOnError option. However, the script evaluates and reacts to the retryCount setting first.
exitOnError
The exitOnError option is used when an uncaught error occurs in the map or reduce stage. It is evaluated after the retryCount option.
When exitOnError is set to true, the script exits the stage after both of the following occur:
-
An error is thrown and not caught.
-
All retries permitted by the retryCount option have been exhausted.
This setting has no impact on the system’s behavior after an application server restart. It is used only when an uncaught error is thrown.
The exitOnError setting is optional. You can set a value for it by adding it in the return statement of your map/reduce script, as described in Adding a Configuration Option to a Script. Possible values are:
-
false — The script continues processing in the current stage. (This is also the behavior used when the option is not added to the script.)
-
true — The script exits the stage and goes immediately to the summarize stage.
See also System Response After an Uncaught Error.
Adding a Configuration Option to a Script
If you want to use the retryCount or exitOnError option, add them to the script’s return statement in a config block. For example:
// Add additional code.
...
return {
config:{
retryCount: 3,
exitOnError: true
},
getInputData: myGetInputData,
map: myMap,
reduce: myReduce,
summarize: mySummarize
};
...
// Add additional code.