Map/Reduce Governance
As with all script types, NetSuite imposes usage limits on map/reduce scripts. Governance rules for map/reduce scripts fall into two main categories:
-
Certain limits, if violated, cause an interruption to the current function invocation. These limits are known as hard limits.
-
Other limits are managed automatically by the system. These limits never interrupt a function invocation. Rather, after the completion of a function invocation, these limits can cause a job to yield and its work to be rescheduled for later. These limits are known as soft limits.
See the following sections for more details:
Be aware that the system does not impose any limit on the full duration of a map/reduce script deployment instance. It also is not possible for a user to impose such a limit. Rather, the system’s limits regulate isolated components of the deployment, such as the duration of a single function invocation.
One way that NetSuite measures a script’s activity is through usage units. For more information about usage units, see SuiteScript Governance and Limits.
Hard Limits on Total Persisted Data
The total persisted data used by a map/reduce script cannot exceed 200MB at any one time. If your script exceeds this limit at any point during its processing, the system throws a PERSISTED_DATA_LIMIT_FOR_MAPREDUCE_SCRIPT_EXCEEDED error. Additionally, the script ends its current function invocation, exits the current stage, and goes immediately to the summarize stage. (This error does not occur in the summarize stage, because the total persisted data cannot be increased during that stage.)
Note that persisted data is computed by a total sum of the following:
-
Total size of all keys and values not yet mapped
-
Total size of all keys and values not yet reduced
-
Total size of all keys and values written as results in reduce
After a particular key and value(s) have been processed by map or reduce, they are no longer counted toward the total storage size. If you have data that is preserved for internal usage, such as troubleshooting or analytics, the data will not be counted toward the total user-facing storage size enforced by the script.
The system takes into account any search results retrieved and returned by the input function. Note that a large number of columns in a result set can significantly increase the amount of data used.
During the map and reduce stages, the total size is a measure of the keys and values yet to be processed. After a key or value is processed, it does not count toward the limit.
Hard Limits on Function Invocations
The following table describes the limits applied to a map/reduce script’s function invocations. If your script exceeds any of these limits, the system throws an SSS_USAGE_LIMIT_EXCEEDED error. The way the system responds to this error varies depending on the stage and the configuration of your script, as shown in the following table.
|
Stage |
Limits per function invocation |
Response to SSS_USAGE_LIMIT_EXCEEDED error |
|---|---|---|
|
getInputData |
|
The script ends the function invocation and exits the stage. It proceeds directly to the summarize stage. |
|
map |
|
The response includes two parts. Note that you can configure the second part:
|
|
reduce |
|
|
|
summarize |
|
The script stops executing. |
Script governance is applied to each invocation in a script, instead of the overall execution.
As shown in the table above, a single execution of the map function should not:
-
Consume more than 1,000 usage points (same as Mass Update script)
-
Run for more than 5 minutes
-
Exceed the instruction count limit for User Event scripts
A single execution of the reduce function should not:
-
Consume more than 5,000 usage points
-
Run for more than 15 minutes
-
Exceed the instruction count limit for User Event scripts
A single execution of the getInputData or summarize function should not:
-
Consume more than 10,000 usage points
-
Run for more than 60 minutes
-
Exceed the instruction count limit for Scheduled scripts
Note that if you are using the map/reduce script type as intended, your script should not approach these limits, particularly for map and reduce function invocations. In general, each invocation of a map or reduce function should do a relatively small portion of work. For more details, see the Map/Reduce Script Best Practices section in the SuiteScript Developer Guide.
Soft Limits on Long-Running Map and Reduce Jobs
In addition to the limits described in Hard Limits on Function Invocations, the system includes a soft limit of 10,000 usage units on each map and reduce job.
To understand how this limit works, first be aware that all map/reduce scripts are processed by SuiteCloud Processors. A processor is a virtual unit of processing power that executes a job.
The 10,000-unit soft limit is a mechanism designed to prevent any processor from being monopolized by a long-running map or reduce job. During the map and reduce stages, after each function invocation, the system checks the total number of units that have been used by the job. If the total usage has surpassed 10,000 units, the job gracefully ends its execution and a new job is created to take its place. The new job has the same priority as the old one, but a later timestamp. This behavior is known as yielding.
Yielding is also affected by the script deployment record’s Yield After Minutes field. This time limit works in the same way as the 10,000-unit limit: The system waits until after each function invocation ends to determine whether the time limit has been exceeded. If it has, the job yields, even if the 10,000-unit limit has not been exceeded. By default, Yield After Minutes is set to 60, but you can enter any number from 3 to 60. For more details, see Yield After Minutes. See also Map/Reduce Yielding.
If your map/reduce script triggers other operations, such as other scripts or workflows, the execution of those operations does not affect the initial time and usage limits for the map/reduce script. For example, if your map/reduce script triggers multiple user event scripts, the execution time of the user event scripts will not count toward the initial map/reduce time limit that is set. In these instances, the real execution time may exceed the soft limit of 60 minutes, but it will not count against script execution time.
Per enqueue limits are soft limits that are in queue at any particular stage of the script instance. The script execution of per enqueue limits will not halt when the limit is reached, but will be checked after each script function invocation. When the script exceeds the soft limit, it must yield control of the queue after the current script execution returns.
Yielding is unrelated to the limits that exist for a single invocation of a map function and a single invocation of a reduce function. Those limits are described in Hard Limits on Function Invocations. Exceeding the limits for a single invocation of a map or reduce function causes the system to throw an SSS_USAGE_LIMIT_EXCEEDED error and ends the function invocation, even if it is not complete.
Character Limits on Map and Reduce Jobs
The character limit for keys in map/reduce scripts (specifically, in mapContext or reduceContext objects) is 3,000 characters. In addition, error messages are returned when a key is longer than 3,000 characters or a value is larger than 10 MB. Keys longer than 3,000 characters will return the error KEY_LENGTH_IS_OVER_3000_BYTES. Values larger than 10 MB will return the error VALUE_LENGTH_IS_OVER_10_MB.
If you have map/reduce scripts that use the mapContext.write(options) or reduceContext.write(options) methods, make sure that key strings are shorter than 3,000 characters and value strings are smaller than 10 MB. Make sure that you consider the potential length of any dynamically generated strings, which may exceed these limits. You should also avoid using keys, instead of values, to pass your data.