Compresses one or more in-memory R objects that the mappers or reducers must write as the values in key-value pairs.
One or more variables, data frames, or other in-memory objects in a comma-separated list.
You should use this function when passing nonscalar or complex R objects, such as data frames and R classes, between the mapper and reducer functions. You do not need to use it on scalar or other simple objects. You can use orch.pack
to vary the data formats, data sets, and variable names for each output value.
You should also use orch.pack
when storing the resultant data set in HDFS. The compressed data set is not corrupted by being stored in an HDFS file.
The orch.pack
function must always be followed by the orch.unpack
function to restore the data to a usable format.