SuiteScript 2.x Map/Reduce Script Type
The map/reduce script type is designed for scripts that need to handle large amounts of data. It is best suited for situations where the data can be divided into small, independent parts. When the script is executed, a structured framework automatically creates enough jobs to process all of these parts. You do not have to manage this process; NetSuite does it all for you. Another advantage of map/reduce is that these jobs can work in parallel and you can choose the level of parallelism when you deploy the script.
Like a scheduled script, a map/reduce script can be invoked manually or on a predefined schedule. However, map/reduce scripts offer several advantages over scheduled scripts. One advantage is that, if a map/reduce job violates certain aspects of NetSuite governance, the map/reduce framework automatically causes the job to yield and its work to be rescheduled, without disruption to the script. However, be aware that some aspects of map/reduce governance cannot be handled through automatic yielding. For that reason, if you use this script type, you should familiarize yourself with the Map/Reduce Governance guidelines.
In general, you should use a map/reduce script for any scenario where you want to process multiple records, and where your logic can be separated into relatively lightweight segments. In contrast, a map/reduce script is not as well suited to situations where you want to enact a long, complex function for each part of your data set. A complex series of steps might be one that includes the loading and saving of multiple records.
You can use SuiteCloud Development Framework (SDF) to manage map/reduce scripts as part of file-based customization projects. For information about SDF, see SuiteCloud Development Framework. You can use the Copy to Account feature to copy an individual map/reduce script to another of your accounts. Each map/reduce script page has a clickable Copy to Account option in the upper right corner. For information about Copy to Account, see Copy to Account.
You can use SuiteScript Analysis to learn about when the script was installed and how it performed in the past. For more information, see Analyzing Scripts.
For more information about map/reduce scripts, see the following topics:
Also see the Map/Reduce Script Best Practices section in the SuiteScript Developer Guide for a list of best practices to follow when using client scripts.
Map/Reduce Script Use Cases
Map/reduce scripts are ideal for scenarios where you want to apply the same logic repeatedly, one time for each object in a series. For example, you could use a map/reduce script to do any of the following:
-
Identify a list of purchase requisitions and transform each one into a purchase order.
-
Search for invoices that meet certain criteria and apply a discount to each one.
-
Search for customer records that appear to be duplicates, then process each apparent duplicate according to your business rules.
-
Search for outstanding tasks assigned to sales reps, then send each person an email that summarizes their outstanding work.
-
Identify files in the NetSuite File Cabinet, use the content of the files to create new documents, and upload the new documents to an external server.
Map/Reduce Key Concepts
Inspired by the map/reduce paradigm, the general idea behind a map/reduce script is as follows:
-
Your script identifies some data that requires processing.
-
This data is split into key-value pairs.
-
Your script defines a function that the system invokes one time for each key-value pair.
-
Optionally, your script can also use a second round of processing.
Depending on how you deploy the script, the system can create multiple jobs for each round of processing and process the data in parallel.
If you are familiar with other SuiteScript 2.x script types, then you may notice that map/reduce scripts are significantly different from most types. Before you begin writing a map/reduce script, make sure you understand these differences. Consider the following:
Map/reduce scripts are executed in stages
With most script types, each script is executed as a single continuous process. In contrast, a map/reduce script is executed in five discrete stages that occur in a specific sequence.
You can control the script’s behavior in four of the five stages. That is, each of these four stages corresponds to an entry point. Your corresponding function defines the script’s behavior during that stage. For example:
-
For the getInputData stage, you write a function that returns an object that can be transformed into a list of key-value pairs. For example, if your function returns a search of NetSuite records, the system would run the search. The key-value pairs would be the results of the search where each key would be the internal ID of a record and each value would be a JSON representation of the record’s field IDs and values.
-
For the map stage, you can optionally write a function that the system invokes one time for each key-value pair. If appropriate, your map function can write output data, in the form of new key-value pairs. If the script also uses a reduce function, this output data is sent as input to the shuffle and then the reduce stage. Otherwise, the new key-value pairs are sent directly to the summarize stage.
-
You do not write a function for the shuffle stage. In this stage, the system sorts through any key-value pairs that were sent to the reduce stage, if a reduce function is defined. These pairs may have been provided by the map function, if a map function is used. If a map function was not used, the shuffle stage uses data provided by the getInputData stage. The shuffle stage groups this data by key to form a new set of key-value pairs, where each key is unique and each value is an array. For example, suppose there are 100 key-value pairs. Suppose that each key represents an employee and each value represents a record that the employee created. If there were only two unique employees, and one employee created 90 records, while the other employee created 10 records, then the shuffle stage would provide two key-value pairs. The keys would be the IDs of the employees. One value would be an array of 90 elements and the other would be an array of 10 elements.
-
For the reduce stage, you write a function that is invoked one time for each key-value pair that was provided by the shuffle stage. Optionally, this function can write data as key-value pairs that are sent to the summarize stage.
-
In the summarize stage, your function can retrieve and log statistics about the script’s work. It can also take actions with data sent by the reduce stage.
Note that you may omit either the map or reduce function. You can also omit the summarize function. For more details, review Map/Reduce Entry Points.
The system supplements your logic
With most script types, the functionality of the script is determined entirely by the code within your script file. A map/reduce script is handled different. In a map/reduce script, the logic in your script is important, but the system also supplements your logic with standardized logic of its own. For example, the system moves data between the stages. Additionally, the system invokes your map and reduce function multiple times. For this reason, think of the logic of the map and reduce functions as being similar to the logic you would use in a loop. Each of these functions should perform a relatively small amount of work. For details about the system’s behavior during and between the stages, see Map/Reduce Script Stages.
The system provides robust context objects
For each entry point function you write in your map/reduce script, the system provides a context object. The system makes a context object available to most SuiteScript 2.x script type entry points, however, the objects provided to map/reduce entry point functions are especially robust. These objects contain data and properties that are critical to writing an effective map/reduce script. For example, you can use these objects to access data from the previous stage and write output data that is sent to the next stage. Context objects can also contain data about errors, usage units consumed, and other statistics. For details, see SuiteScript 2.x Map/Reduce Script Entry Points and API.
Multiple jobs are used to execute one script
All map/reduce scripts are executed using SuiteCloud Processors, which handle work through a series of jobs. Each job is executed by a processor which is a virtual unit of processing power. SuiteCloud Processors are also used to process scheduled scripts. However, these two script types are handled differently. For example, the system always creates only one job to handle a scheduled script. In contrast, the system creates multiple jobs to process a single map/reduce script. Specifically, the system creates at least one job to execute each stage. Additionally, multiple jobs can be created to handle the work of the map stage, and multiple jobs can be created for the reduce stage. When the system creates multiple map and reduce jobs, these jobs work independently of each other and may work in parallel across multiple processors. For this reason, the map and reduce stages are considered parallel stages.
In contrast, the getInputData and summarize stages are each executed by one job. In each case, that job invokes your function only one time. These stages are serial stages. The shuffle stage is also a serial stage.
Map/reduce scripts permit yielding and other interruptions
Since the map and reduce stages consist of multiple independent map and reduce function invocations, the work of these stages can easily be divided among multiple jobs. The structure is naturally flexible. It allows for parallel processing, and it also permits map and reduce jobs to manage some aspects of their own resource consumption.
If a job monopolizes a processor for too long, the system can naturally finish the job after the current map or reduce function has completed. In this case, the system creates a new job to continue executing remaining key-value pairs. Based on its priority and submission time stamp, the new job either starts right after the original job has finished, or it starts later, to allow higher-priority jobs processing other scripts to execute. For more details, see Map/Reduce Yielding.
Also be aware that the system imposes some usage limits on map/reduce scripts that are not managed through yielding. For details, see Map/Reduce Governance.
Map/Reduce Entry Points
A map/reduce script can go through a total of five stages. One stage, shuffle, does not correspond with an entry point. All other stages do correspond with an entry point. Their entry points are described in the following table.
|
Entry point |
Purpose of corresponding function |
Required? |
|---|---|---|
|
To identify data that needs processing. The system passes this data to the next stage. |
yes |
|
|
To apply processing to each key-value pair provided from the getInputData stage, and optionally pass data to the next stage. |
One of these two entry points is required. You can also use both entry points. |
|
|
To apply processing to each key-value pair provided from the map stage. In this stage, each key is unique, and each value is an array of values. This function can optionally pass data to the summarize stage. |
||
|
To retrieve data about the script’s execution and take any needed actions with the output of the reduce stage. |
no |
For full details on the map/reduce entry points and their corresponding context objects, see SuiteScript 2.x Map/Reduce Script Entry Points and API.
Map/Reduce Script Samples
Two map/reduce script samples are provided:
-
Counting Characters Example – counts the number of times that each letter of the alphabet occurs within the string. This script is provided as a basic script to help you understand how map/reduce scripts function.
-
Processing Invoices Example– processes invoices and contains logic to handle errors.
These script samples use SuiteScript 2.x. A newer version, SuiteScript 2.1, is also available and supports new language features that are included in the ES2019 specification. You can write map/reduce scripts using either SuiteScript 2.0 or SuiteScript 2.1.
-
For help with writing scripts in SuiteScript 2.x, see SuiteScript 2.x Hello World and SuiteScript 2.x Entry Point Script Creation and Deployment.
-
For more information about SuiteScript versions and SuiteScript 2.1, see SuiteScript Versioning Guidelines and SuiteScript 2.1.
Counting Characters Example
The following sample is a basic map/reduce script. This sample does not accomplish a realistic business objective but rather is designed to demonstrate how the script type works.
This script defines a hard-coded string. The script then counts the number of times each letter of the alphabet occurs within the string and creates a file that shows its results. Refer to the comments in the script for details about how the system processes the script.
/**
* @NApiVersion 2.x
* @NScriptType MapReduceScript
*/
define(['N/file'], function(file) {
// Define characters that should not be counted when the script performs its
// analysis of the text.
const PUNCTUATION_REGEXP = /[\u2000-\u206F\u2E00-\u2E7F\\'!"#\$%&\(\)\*\+,\-\.\/:;<=>\?@\[\]\^_`\{\|\}~]/g;
// Use the getInputData function to return two strings (1) the quick brown fox and (2) jumps over the lazy dog.
function getInputData() {
return "the quick brown fox \njumps over the lazy dog.".split('\n');
}
// After the getInputData function is executed, the system creates the following
// key-value pairs:
//
// key: 0, value: 'the quick brown fox'
// key: 1, value: 'jumps over the lazy dog.'
// The map function is invoked one time for each key-value pair. Each time the
// function is invoked, the relevant key-value pair is made available through
// the context.key and context.value properties.
function map(context) {
// Create a loop that examines each character in the string. Exclude spaces
// and punctuation marks.
for (var i = 0; context.value && i < context.value.length; i++) {
if (context.value[i] !== ' ' && !PUNCTUATION_REGEXP.test(context.value[i])) {
// For each character, invoke the context.write() method. This method saves
// a new key-value pair. For the new key, save the character currently being
// examined by the loop. For each value, save the number 1.
context.write({
key: context.value[i],
value: 1
});
}
}
}
// After the map function has been invoked for the last time, the shuffle stage
// begins. In this stage, the system sorts the 35 key-value pairs that were saved
// by the map function during its two invocations. From those pairs, the shuffle
// stage creates a new set of key-value pairs, where the each key is unique. In
// this way, the number of key-value pairs is reduced to 25. For example, the map
// stage saved three instances of {'e','1'}. In place of those pairs, the shuffle
// stage creates one pair: {'e', ['1','1','1']}. These pairs are made available to
// the reduce stage through the context.key and context.values properties.
// The reduce function is invoked one time for each of the 25 key-value pairs
// provided.
function reduce(context) {
// Use the context.write() method to save a new key-value pair, where the new key
// equals the key currently being processed by the function. This key is a letter
// in the alphabet. Make the value equal to the length of the context.values array.
// This number represents the number of times the letter occurred in the original
// string.
context.write({
key: context.key,
value: context.values.length
});
}
// The summarize stage is a serial stage, so this function is invoked only one
// time.
function summarize(context) {
// Log details about the script's execution.
log.audit({
title: 'Usage units consumed',
details: context.usage
});
log.audit({
title: 'Concurrency',
details: context.concurrency
});
log.audit({
title: 'Number of yields',
details: context.yields
});
// Use the context object's output iterator to gather the key-value pairs saved
// at the end of the reduce stage. Also, tabulate the number of key-value pairs
// that were saved. This number represents the total number of unique letters
// used in the original string.
var text = '';
var totalKeysSaved = 0;
context.output.iterator().each(function(key, value) {
text += (key + ' ' + value + '\n');
totalKeysSaved++;
return true;
});
// Log details about the total number of pairs saved.
log.audit({
title: 'Unique number of letters used in string',
details: totalKeysSaved
});
// Use the N/file module to create a file that stores the reduce stage output,
// which you gathered by using the output iterator.
var fileObj = file.create({
name: 'letter_count_result.txt',
fileType: file.Type.PLAINTEXT,
contents: text
});
fileObj.folder = -15;
var fileId = fileObj.save();
log.audit({
title: 'Id of new file record',
details: fileId
});
}
// Link each entry point to the appropriate function.
return {
getInputData: getInputData,
map: map,
reduce: reduce,
summarize: summarize
};
});
The character limit for keys in map/reduce scripts (specifically, in mapContext or reduceContext objects) is reduced to 3,000 characters. In addition, error messages are returned when a key is longer than 3,000 characters or a value is larger than 10 MB. Keys longer than 3,000 characters will return the error KEY_LENGTH_IS_OVER_3000_BYTES. Values larger than 10 MB will return the error VALUE_LENGTH_IS_OVER_10_MB.
If you have map/reduce scripts that use the mapContext.write(options) or reduceContext.write(options) methods, ensure that key strings are shorter than 3,000 characters and value strings are smaller than 10 MB. Consider the potential length of any dynamically generated strings which may exceed these limits and avoid using keys, instead of values, to pass your data.
There is a 200 MB size limit on a single execution for any data handed between stages. Should persistent data exceed this limit, the script will require provisions for a data custom field or record to be used in either context.
Processing Invoices Example
The following example shows a sample script that processes invoices and contains logic to handle errors. This script is designed to do the following:
-
Find the customers associated with all open invoices.
-
Apply a location-based discount to each invoice.
-
Write each invoice to the reduce stage so it is grouped by customer.
-
Initialize a new CustomerPayment for each customer applied only to the invoices specified in the reduce values.
-
Create a custom record capturing the details of the records that were processed.
-
Notify administrators of any exceptions using an email notification.
Prior to running this sample, you need to manually create a custom record type with id "customrecord_summary", and text fields with id "custrecord_time", "custrecord_usage", and "custrecord_yields".
Script Sample Prerequisites
-
From the NetSuite UI, select Customization > List, Records, & Fields > Record Types > New.
-
From the Custom Record Type page, enter a value for name.
-
In the ID field, enter "customrecord_summary".
-
Select Save.
-
From the Fields subtab, do the following:
-
Select New Field. Enter a label and set ID to "custrecord_time". Ensure that the Type field is set to Free-Form Text. Select Save & New.
-
Select New Field. Enter a label and set ID to "custrecord_usage". Ensure that the Type field is set to Free-Form Text. Select Save & New.
-
Select New Field. Enter a label and set ID to "custrecord_yields". Ensure that the Type field is set to Free-Form Text. Select Save.
-
/**
* @NApiVersion 2.x
* @NScriptType MapReduceScript
*/
define(['N/search', 'N/record', 'N/email', 'N/runtime', 'N/error'],
function(search, record, email, runtime, error)
{
function handleErrorAndSendNotification(e, stage)
{
log.error('Stage: ' + stage + ' failed', e);
var author = -5;
var recipients = 'notify@example.com';
var subject = 'Map/Reduce script ' + runtime.getCurrentScript().id + ' failed for stage: ' + stage;
var body = 'An error occurred with the following information:\n' +
'Error code: ' + e.name + '\n' +
'Error msg: ' + e.message;
email.send({
author: author,
recipients: recipients,
subject: subject,
body: body
});
}
function handleErrorIfAny(summary)
{
var inputSummary = summary.inputSummary;
var mapSummary = summary.mapSummary;
var reduceSummary = summary.reduceSummary;
if (inputSummary.error)
{
var e = error.create({
name: 'INPUT_STAGE_FAILED',
message: inputSummary.error
});
handleErrorAndSendNotification(e, 'getInputData');
}
handleErrorInStage('map', mapSummary);
handleErrorInStage('reduce', reduceSummary);
}
function handleErrorInStage(stage, summary)
{
var errorMsg = [];
summary.errors.iterator().each(function(key, value){
var msg = 'Failure to accept payment from customer id: ' + key + '. Error was: ' + JSON.parse(value).message + '\n';
errorMsg.push(msg);
return true;
});
if (errorMsg.length > 0)
{
var e = error.create({
name: 'RECORD_TRANSFORM_FAILED',
message: JSON.stringify(errorMsg)
});
handleErrorAndSendNotification(e, stage);
}
}
function createSummaryRecord(summary)
{
try
{
var seconds = summary.seconds;
var usage = summary.usage;
var yields = summary.yields;
var rec = record.create({
type: 'customrecord_summary',
});
rec.setValue({
fieldId : 'name',
value: 'Summary for M/R script: ' + runtime.getCurrentScript().id
});
rec.setValue({
fieldId: 'custrecord_time',
value: seconds
});
rec.setValue({
fieldId: 'custrecord_usage',
value: usage
});
rec.setValue({
fieldId: 'custrecord_yields',
value: yields
});
rec.save();
}
catch(e)
{
handleErrorAndSendNotification(e, 'summarize');
}
}
function applyLocationDiscountToInvoice(recordId)
{
var invoice = record.load({
type: record.Type.INVOICE,
id: recordId,
isDynamic: true
});
var location = invoice.getText({
fieldId: 'location'
});
var discount;
if (location === 'East Coast')
discount = 'Eight Percent';
else if (location === 'West Coast')
discount = 'Five Percent';
else if (location === 'United Kingdom')
discount = 'Nine Percent';
else
discount = '';
invoice.setText({
fieldId: 'discountitem',
text: discount,
ignoreFieldChange : false
});
log.debug(recordId + ' has been updated with location-based discount.');
invoice.save();
}
function getInputData()
{
return search.create({
type: record.Type.INVOICE,
filters: [['status', search.Operator.IS, 'open']],
columns: ['entity'],
title: 'Open Invoice Search'
});
}
function map(context)
{
var searchResult = JSON.parse(context.value);
var invoiceId = searchResult.id;
var entityId = searchResult.values.entity.value;
applyLocationDiscountToInvoice(invoiceId);
context.write({
key: entityId,
value: invoiceId
});
}
function reduce(context)
{
var customerId = context.key;
var custPayment = record.transform({
fromType: record.Type.CUSTOMER,
fromId: customerId,
toType: record.Type.CUSTOMER_PAYMENT,
isDynamic: true
});
var lineCount = custPayment.getLineCount('apply');
for (var j = 0; j < lineCount; j++)
{
custPayment.selectLine({
sublistId: 'apply',
line: j
});
custPayment.setCurrentSublistValue({
sublistId: 'apply',
fieldId: 'apply',
value: true
});
}
var custPaymentId = custPayment.save();
context.write({
key: custPaymentId
});
}
function summarize(summary)
{
handleErrorIfAny(summary);
createSummaryRecord(summary);
}
return {
getInputData: getInputData,
map: map,
reduce: reduce,
summarize: summarize
};
});
Related Support Article
Related Topics
- SuiteScript Versioning Guidelines
- SuiteScript 2.1
- SuiteScript 2.x Script Types
- SuiteScript 2.x Bundle Installation Script Type
- SuiteScript 2.x Client Script Type
- SuiteScript 2.x Mass Update Script Type
- SuiteScript 2.x Portlet Script Type
- SuiteScript 2.x RESTlet Script Type
- SuiteScript 2.x Scheduled Script Type
- SuiteScript 2.x Suitelet Script Type
- SuiteScript 2.x User Event Script Type
- SuiteScript 2.x Workflow Action Script Type