SuiteScript 2.x Map/Reduce Script Type
The map/reduce script type is designed for scripts that need to handle large amounts of data. It's best for situations where you can divide the data into small, independent parts. When the script is executed, a structured framework automatically creates enough jobs to process all of these parts. You do not have to manage this process; NetSuite does it all for you. Another advantage of map/reduce is that these jobs can work in parallel and you can choose the level of parallelism when you deploy the script.
A map/reduce script can be run manually or on a schedule, like a scheduled script. Map/reduce scripts have several advantages over scheduled scripts. One advantage is that, if a map/reduce job violates certain aspects of NetSuite governance, the map/reduce framework automatically causes the job to yield and its work to be rescheduled, without disruption to the script. However, be aware that some aspects of map/reduce governance can't be handled through automatic yielding. For that reason, if you use this script type, you should familiarize yourself with the Map/Reduce Governance guidelines.
In general, use a map/reduce script when you need to process multiple records and can break your logic into smaller pieces. In contrast, a map/reduce script is not as well suited to situations where you want to enact a long, complex function for each part of your data set. A complex series of steps might include loading and saving multiple records.
You can use SuiteCloud Development Framework (SDF) to manage map/reduce scripts as part of file-based customization projects. For information about SDF, see SuiteCloud Development Framework. You can use the Copy to Account feature to copy an individual map/reduce script to another of your accounts. Each map/reduce script page has a clickable Copy to Account option in the upper right corner. For information about Copy to Account, see Copy to Account.
You can use SuiteScript Analysis to learn about when the script was installed and how it performed in the past. For more information, see Analyzing Scripts.
For more information about map/reduce scripts, see the following topics:
Also see the Map/Reduce Script Best Practices section in the SuiteScript Developer Guide for a list of best practices to follow when using client scripts.
Map/Reduce Script Use Cases
Map/reduce scripts are perfect for applying the same logic to multiple objects, one at a time. For instance, you can use map/reduce scripts for tasks like:
-
Identify a list of purchase requisitions and transform each one into a purchase order.
-
Search for invoices that meet certain criteria and apply a discount to each one.
-
Search for customer records that appear to be duplicates, then process each apparent duplicate according to your business rules.
-
Search for outstanding tasks assigned to sales reps, then send each person an email that summarizes their outstanding work.
-
Identify files in the NetSuite File Cabinet, use the content of the files to create new documents, and upload the new documents to an external server.
Map/Reduce Key Concepts
Inspired by the map/reduce paradigm, the general idea behind a map/reduce script is as follows:
-
Your script identifies some data that requires processing.
-
This data is split into key-value pairs.
-
Your script defines a function that the system invokes one time for each key-value pair.
-
Optionally, your script can also use a second round of processing.
The system can create multiple jobs for each round of processing and process the data in parallel, depending on how you deploy the script.
If you've worked with other SuiteScript 2.x scripts, you'll find that map/reduce scripts work differently. Before you start writing a map/reduce script, make sure you understand the differences. Keep the following in mind:
Map/reduce scripts are executed in stages
Most script types run as a single continuous process. In contrast, map/reduce scripts run in five stages, in a specific order.
You can control the script’s behavior in four of the five stages. That is, each of these four stages corresponds to an entry point. Your corresponding function defines the script’s behavior during that stage. For example:
-
For the getInputData stage, you write a function that returns an object that can be transformed into a list of key-value pairs. For example, if your function returns a search of NetSuite records, the system runs the search. The key-value pairs would be the results of the search where each key would be the internal ID of a record and each value would be a JSON representation of the record’s field IDs and values.
-
For the map stage, you can optionally write a function that the system invokes one time for each key-value pair. If needed, your map function can generate new key-value pairs as output. If the script also uses a reduce function, this output data is sent as input to the shuffle and then the reduce stage. Otherwise, the new key-value pairs are sent directly to the summarize stage.
-
You do not write a function for the shuffle stage. In this stage, the system organizes key-value pairs for the reduce stage, if you've defined one. These pairs may have been provided by the map function, if you're using one. If a map function was not used, the shuffle stage uses data provided by the getInputData stage. The shuffle stage groups this data by key to form a new set of key-value pairs, where each key is unique and each value is an array.
For example, suppose there are 100 key-value pairs. Let's say each key is an employee, and each value is a record they created. If there were only two unique employees, and one employee created 90 records, while the other created 10, then the shuffle stage would provide two key-value pairs. The keys are the employee IDs. One value is an array with 90 elements, and the other has 10.
-
For the reduce stage, you write a function that runs one time for each key-value pair from the shuffle stage. This function can also generate key-value pairs to send to the summarize stage.
-
In the summarize stage, your function can get and log stats about the script's work. It can also act on data from the reduce stage.
Note that you can omit either the map or reduce function. You can also skip the summarize function. For more details, review Map/Reduce Entry Points.
The system supplements your logic
Most script types rely solely on the code in your script file. Map/reduce scripts work differently. In a map/reduce script, your logic is key, but the system also adds its own standardized logic. For instance, the system handles data transfer between stages. The system also calls your map and reduce functions multiple times. For this reason, think of the logic of the map and reduce functions as being similar to the logic you'd use in a loop. Each of these functions should perform a relatively small amount of work. For details about the system’s behavior during and between the stages, see Map/Reduce Script Stages.
The system provides robust context objects
For each entry point function you write in your map/reduce script, the system provides a context object. The system provides context objects to most SuiteScript 2.x entry points, but map/reduce entry point functions are especially robust. These objects contain data and properties that are critical to writing an effective map/reduce script. For instance, you can use these objects to get data from the previous stage and send output to the next one. These objects can also hold error data, usage stats, and other metrics. For details, see SuiteScript 2.x Map/Reduce Script Entry Points and API.
Multiple jobs are used to execute one script
Map/reduce scripts run using SuiteCloud Processors, which handle work through a series of jobs. Each job is run by a processor, a virtual unit of processing power. SuiteCloud Processors also handle scheduled scripts. However, these two script types are handled differently. For instance, the system uses only one job for a scheduled script. In contrast, the system creates multiple jobs for a single map/reduce script. The system creates at least one job per stage. The system can also create multiple jobs for the map and reduce stages. When there are multiple map and reduce jobs, they work independently and can run in parallel across processors. For this reason, the map and reduce stages are considered parallel stages.
In contrast, the getInputData and summarize stages each use one job. In each case, that job calls your function only one time. These stages are serial stages. The shuffle stage is also a serial stage.
Map/reduce scripts permit yielding and other interruptions
The map and reduce stages can be easily split into multiple jobs because they consist of independent function invocations. The structure is naturally flexible. It enables parallel processing and lets map and reduce jobs manage their own resource usage to some extent.
If a job takes up a processor for too long, the system can finish it after the current function is done. In this case, the system creates a new job to continue executing remaining key-value pairs. The new job starts either right after the original job finishes or later, depending on its priority and when it was submitted, to let higher-priority jobs run. For more details, see Map/Reduce Yielding.
Note that the system has usage limits for map/reduce scripts that aren't managed through yielding. For details, see Map/Reduce Governance.
Map/Reduce Entry Points
A map/reduce script can go through a total of five stages. For more information about map reduce stages, see Map/Reduce Script Stages. The shuffle stage doesn't have a corresponding entry point. Their entry points are described in the following table.
|
Entry point |
Purpose of corresponding function |
Required? |
|---|---|---|
|
Identify the data that needs processing. The system sends this data to the next stage. |
Yes |
|
|
Process each key-value pair from the getInputData stage and optionally send data to the next stage. |
One of these two entry points is required. You can use both if needed. |
|
|
Process each key-value pair from the map stage. In this stage, each key is unique, and each value is an array of values. This function can also send data to the summarize stage. |
||
|
Get data about the script's execution and take any necessary actions with the reduce stage's output. |
No |
For full details on the map/reduce entry points and their corresponding context objects, see SuiteScript 2.x Map/Reduce Script Entry Points and API.
Map/Reduce Script Samples
Two map/reduce script samples are provided:
-
Counting Characters Example – counts how often each alphabet letter appears in a string. This basic script helps you understand how map/reduce scripts work.
-
Processing Invoices Example– handles invoices and includes error-handling logic.
These script samples use SuiteScript 2.x. A newer version, SuiteScript 2.1, is also available and supports new language features that are included in the ES2019 specification. You can write map/reduce scripts using either SuiteScript 2.0 or SuiteScript 2.1.
-
For help with writing scripts in SuiteScript 2.x, see SuiteScript 2.x Hello World and SuiteScript 2.x Entry Point Script Creation and Deployment.
-
For more information about SuiteScript versions and SuiteScript 2.1, see SuiteScript Versioning Guidelines and SuiteScript 2.1.
Counting Characters Example
The following sample is a basic map/reduce script. This sample isn't meant for real-world business use, but rather to show how map/reduce scripts work.
This script defines a hard-coded string. The script counts each alphabet letter's occurrences in the string and creates a results file. Check the script comments for details on how the system processes it.
/**
* @NApiVersion 2.x
* @NScriptType MapReduceScript
*/
define(['N/file'], function(file) {
// Define characters that should not be counted when the script performs its
// analysis of the text.
const PUNCTUATION_REGEXP = /[\u2000-\u206F\u2E00-\u2E7F\\'!"#\$%&\(\)\*\+,\-\.\/:;<=>\?@\[\]\^_`\{\|\}~]/g;
// Use the getInputData function to return two strings (1) the quick brown fox and (2) jumps over the lazy dog.
function getInputData() {
return "the quick brown fox \njumps over the lazy dog.".split('\n');
}
// After the getInputData function is executed, the system creates the following
// key-value pairs:
//
// key: 0, value: 'the quick brown fox'
// key: 1, value: 'jumps over the lazy dog.'
// The map function is invoked one time for each key-value pair. Each time the
// function is invoked, the relevant key-value pair is made available through
// the context.key and context.value properties.
function map(context) {
// Create a loop that examines each character in the string. Exclude spaces
// and punctuation marks.
for (var i = 0; context.value && i < context.value.length; i++) {
if (context.value[i] !== ' ' && !PUNCTUATION_REGEXP.test(context.value[i])) {
// For each character, invoke the context.write() method. This method saves
// a new key-value pair. For the new key, save the character currently being
// examined by the loop. For each value, save the number 1.
context.write({
key: context.value[i],
value: 1
});
}
}
}
// After the map function has been invoked for the last time, the shuffle stage
// begins. In this stage, the system sorts the 35 key-value pairs that were saved
// by the map function during its two invocations. From those pairs, the shuffle
// stage creates a new set of key-value pairs, where the each key is unique. In
// this way, the number of key-value pairs is reduced to 25. For example, the map
// stage saved three instances of {'e','1'}. In place of those pairs, the shuffle
// stage creates one pair: {'e', ['1','1','1']}. These pairs are made available to
// the reduce stage through the context.key and context.values properties.
// The reduce function is invoked one time for each of the 25 key-value pairs
// provided.
function reduce(context) {
// Use the context.write() method to save a new key-value pair, where the new key
// equals the key currently being processed by the function. This key is a letter
// in the alphabet. Make the value equal to the length of the context.values array.
// This number represents the number of times the letter occurred in the original
// string.
context.write({
key: context.key,
value: context.values.length
});
}
// The summarize stage is a serial stage, so this function is invoked only one
// time.
function summarize(context) {
// Log details about the script's execution.
log.audit({
title: 'Usage units consumed',
details: context.usage
});
log.audit({
title: 'Concurrency',
details: context.concurrency
});
log.audit({
title: 'Number of yields',
details: context.yields
});
// Use the context object's output iterator to gather the key-value pairs saved
// at the end of the reduce stage. Also, tabulate the number of key-value pairs
// that were saved. This number represents the total number of unique letters
// used in the original string.
var text = '';
var totalKeysSaved = 0;
context.output.iterator().each(function(key, value) {
text += (key + ' ' + value + '\n');
totalKeysSaved++;
return true;
});
// Log details about the total number of pairs saved.
log.audit({
title: 'Unique number of letters used in string',
details: totalKeysSaved
});
// Use the N/file module to create a file that stores the reduce stage output,
// which you gathered by using the output iterator.
var fileObj = file.create({
name: 'letter_count_result.txt',
fileType: file.Type.PLAINTEXT,
contents: text
});
fileObj.folder = -15;
var fileId = fileObj.save();
log.audit({
title: 'Id of new file record',
details: fileId
});
}
// Link each entry point to the appropriate function.
return {
getInputData: getInputData,
map: map,
reduce: reduce,
summarize: summarize
};
});
The character limit for keys in map/reduce scripts (specifically, in mapContext or reduceContext objects) is reduced to 3,000 characters. You'll also get error messages if your key is too long or your value is over 10 MB. Keys over 3,000 characters will return a KEY_LENGTH_IS_OVER_3000_BYTES error. Values over 10 MB will return a VALUE_LENGTH_IS_OVER_10_MB error. For more information about map/reduce limits see Map/Reduce Governance.
If you have map/reduce scripts that use the mapContext.write(options) or reduceContext.write(options) methods, keep your key strings under 3,000 characters and value strings under 10 MB. Consider the potential length of any dynamically generated strings which may exceed these limits and avoid using keys, instead of values, to pass your data.
There's a 200 MB limit on data passed between stages in a single execution. Should persistent data exceed this limit, the script will require provisions for a data custom field or record to be used in either context.
Processing Invoices Example
The following example shows a sample script that processes invoices and contains logic to handle errors. This script does the following:
-
Find the customers associated with all open invoices.
-
Apply a location-based discount to each invoice.
-
Write each invoice to the reduce stage so it is grouped by customer.
-
Initialize a new CustomerPayment for each customer applied only to the invoices specified in the reduce values.
-
Create a custom record capturing the details of the records that were processed.
-
Send an email to administrators about any exceptions.
Before running this sample, create a custom record type with ID "customrecord_summary", and text fields with id "custrecord_time", "custrecord_usage", and "custrecord_yields".
Script Sample Prerequisites
-
In NetSuite, go to Customization > List, Records, & Fields > Record Types > New.
-
On the Custom Record Type page, enter a name.
-
In the ID field, enter "customrecord_summary".
-
Select Save.
-
From the Fields subtab, do the following:
-
Select New Field. Enter a label and set ID to "custrecord_time". Ensure that the Type field is set to Free-Form Text. Select Save & New.
-
Select New Field. Enter a label and set ID to "custrecord_usage". Ensure that the Type field is set to Free-Form Text. Select Save & New.
-
Select New Field. Enter a label and set ID to "custrecord_yields". Ensure that the Type field is set to Free-Form Text. Select Save.
-
/**
* @NApiVersion 2.x
* @NScriptType MapReduceScript
*/
define(['N/search', 'N/record', 'N/email', 'N/runtime', 'N/error'],
function(search, record, email, runtime, error)
{
function handleErrorAndSendNotification(e, stage)
{
log.error('Stage: ' + stage + ' failed', e);
var author = -5;
var recipients = 'notify@example.com';
var subject = 'Map/Reduce script ' + runtime.getCurrentScript().id + ' failed for stage: ' + stage;
var body = 'An error occurred with the following information:\n' +
'Error code: ' + e.name + '\n' +
'Error msg: ' + e.message;
email.send({
author: author,
recipients: recipients,
subject: subject,
body: body
});
}
function handleErrorIfAny(summary)
{
var inputSummary = summary.inputSummary;
var mapSummary = summary.mapSummary;
var reduceSummary = summary.reduceSummary;
if (inputSummary.error)
{
var e = error.create({
name: 'INPUT_STAGE_FAILED',
message: inputSummary.error
});
handleErrorAndSendNotification(e, 'getInputData');
}
handleErrorInStage('map', mapSummary);
handleErrorInStage('reduce', reduceSummary);
}
function handleErrorInStage(stage, summary)
{
var errorMsg = [];
summary.errors.iterator().each(function(key, value){
var msg = 'Failure to accept payment from customer id: ' + key + '. Error was: ' + JSON.parse(value).message + '\n';
errorMsg.push(msg);
return true;
});
if (errorMsg.length > 0)
{
var e = error.create({
name: 'RECORD_TRANSFORM_FAILED',
message: JSON.stringify(errorMsg)
});
handleErrorAndSendNotification(e, stage);
}
}
function createSummaryRecord(summary)
{
try
{
var seconds = summary.seconds;
var usage = summary.usage;
var yields = summary.yields;
var rec = record.create({
type: 'customrecord_summary',
});
rec.setValue({
fieldId : 'name',
value: 'Summary for M/R script: ' + runtime.getCurrentScript().id
});
rec.setValue({
fieldId: 'custrecord_time',
value: seconds
});
rec.setValue({
fieldId: 'custrecord_usage',
value: usage
});
rec.setValue({
fieldId: 'custrecord_yields',
value: yields
});
rec.save();
}
catch(e)
{
handleErrorAndSendNotification(e, 'summarize');
}
}
function applyLocationDiscountToInvoice(recordId)
{
var invoice = record.load({
type: record.Type.INVOICE,
id: recordId,
isDynamic: true
});
var location = invoice.getText({
fieldId: 'location'
});
var discount;
if (location === 'East Coast')
discount = 'Eight Percent';
else if (location === 'West Coast')
discount = 'Five Percent';
else if (location === 'United Kingdom')
discount = 'Nine Percent';
else
discount = '';
invoice.setText({
fieldId: 'discountitem',
text: discount,
ignoreFieldChange : false
});
log.debug(recordId + ' has been updated with location-based discount.');
invoice.save();
}
function getInputData()
{
return search.create({
type: record.Type.INVOICE,
filters: [['status', search.Operator.IS, 'open']],
columns: ['entity'],
title: 'Open Invoice Search'
});
}
function map(context)
{
var searchResult = JSON.parse(context.value);
var invoiceId = searchResult.id;
var entityId = searchResult.values.entity.value;
applyLocationDiscountToInvoice(invoiceId);
context.write({
key: entityId,
value: invoiceId
});
}
function reduce(context)
{
var customerId = context.key;
var custPayment = record.transform({
fromType: record.Type.CUSTOMER,
fromId: customerId,
toType: record.Type.CUSTOMER_PAYMENT,
isDynamic: true
});
var lineCount = custPayment.getLineCount('apply');
for (var j = 0; j < lineCount; j++)
{
custPayment.selectLine({
sublistId: 'apply',
line: j
});
custPayment.setCurrentSublistValue({
sublistId: 'apply',
fieldId: 'apply',
value: true
});
}
var custPaymentId = custPayment.save();
context.write({
key: custPaymentId
});
}
function summarize(summary)
{
handleErrorIfAny(summary);
createSummaryRecord(summary);
}
return {
getInputData: getInputData,
map: map,
reduce: reduce,
summarize: summarize
};
});
Related Support Article
Related Topics
- SuiteScript Versioning Guidelines
- SuiteScript 2.1
- SuiteScript 2.x Script Types
- SuiteScript 2.x Bundle Installation Script Type
- SuiteScript 2.x Client Script Type
- SuiteScript 2.x Mass Update Script Type
- SuiteScript 2.x Portlet Script Type
- SuiteScript 2.x RESTlet Script Type
- SuiteScript 2.x Scheduled Script Type
- SuiteScript 2.x Suitelet Script Type
- SuiteScript 2.x User Event Script Type
- SuiteScript 2.x Workflow Action Script Type