[/map {"- map/map "}) [/map/topicref {"- map/topicref "}) [/map/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicmeta/navtitle {"- topic/navtitle "}) Overview (navtitle][/map/topicref/topicmeta/linktext {"- map/linktext "}) Overview (linktext][/map/topicref/topicmeta/shortdesc {"- map/shortdesc "}) This section describes transformations and the custom transform functions available in Big Data Discovery. (shortdesc] (topicmeta][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) About transformations and transformation scripts (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) About transformations and transformation scripts (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Transformations are changes you can make to your project data set, after the source data has been processed and loaded into Studio. Transformations can be thought of as a substitute for an ETL process of cleaning your data. Transformations can overwrite an existing attribute, modify attributes, or create new attributes. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) About Groovy (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) About Groovy (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Groovy is a dynamically-typed scripting language. Code written in the Java language is valid in Groovy, so users not familiar with Groovy may resort to the Java syntax. All custom transform functions available for you in Big Data Discovery are written in Groovy. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) About transform functions (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) About transform functions (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Transform functions are customized Groovy functions available in Big Data Discovery that you can include in your transformation scripts. Each transform function performs a specific operation on your data, from simple ones, such as converting an attribute to a different data type, to more complex ones, such as determining the overall sentiment of a document or a string of text. (shortdesc] (topicmeta] (topicref] (topicref] [/map/topicref {"- map/topicref "}) [/map/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicmeta/navtitle {"- topic/navtitle "}) Working with Transformation Scripts (navtitle][/map/topicref/topicmeta/linktext {"- map/linktext "}) Working with Transformation Scripts (linktext][/map/topicref/topicmeta/shortdesc {"- map/shortdesc "}) These topics describe the process of creating and applying transformation scripts, using the custom transform functions. (shortdesc] (topicmeta][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Transformation script workflow (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Transformation script workflow (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) At a high level, writing a transformation script and applying it to your data involves the following steps: (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Writing transformations (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Writing transformations (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) You can write transformations in the Transform area of Studio, using the Transformation Editor. Transformations can contain attributes and records from your project data set as variables, and can create new attributes to hold the transformed values. (shortdesc] (topicmeta][/map/topicref/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) The Transformation Editor (navtitle][/map/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) The Transformation Editor (linktext][/map/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) You create transformations in the Transformation Editor, this is the built-in Groovy editor within the Transform area in Studio. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Formats for variables (navtitle][/map/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Formats for variables (linktext][/map/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) You can use attributes from your project data set as variables in your transformation scripts. This allows you to pass attributes to transform functions as parameters and perform other operations on them. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Setting transformation outputs (navtitle][/map/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Setting transformation outputs (linktext][/map/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) You can set your transformation to output to either the selected attribute or a new attribute (this is useful if a transformation is creating a new column). (shortdesc] (topicmeta] (topicref][/map/topicref/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Functional and dot notation and function chaining (navtitle][/map/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Functional and dot notation and function chaining (linktext][/map/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) You must use proper syntax when adding transform functions to your script, or your script won't run properly. You can reference all transform functions using functional notation, as described in this topic. (shortdesc] (topicmeta] (topicref] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Exception handling and debugging (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Exception handling and debugging (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) These topics describe exception handling in Transform and show how to debug individual transformations. (shortdesc] (topicmeta][/map/topicref/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Script evaluation (navtitle][/map/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Script evaluation (linktext][/map/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Transformation scripts are evaluated top-down on each input row. This means that each transformation in the script is applied in order to the first input row, then again to the second row, and so on. This is illustrated by the following pseudo code: (shortdesc] (topicmeta] (topicref][/map/topicref/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Dynamic typing vs. static typing (navtitle][/map/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Dynamic typing vs. static typing (linktext][/map/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) This topic is provided for reverence only as it explains the differences between dynamic and static typing. Understanding the differences between dynamic and static typing is key to understanding the way in which transformation script errors are handled, and how it is different from the way Groovy handles errors. This will also help you interpret errors created by your transformation script. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Exception handling and troubleshooting your scripts (navtitle][/map/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Exception handling and troubleshooting your scripts (linktext][/map/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Transform uses a static parser to override some of Groovy's dynamic typing behavior and detect parsing errors, such as undefined variables, when you preview or save your transformations. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Transform logging (navtitle][/map/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Transform logging (linktext][/map/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) If your transformation script fails to commit, you can learn more about the cause of the failure by looking through the Data Processing logs. (shortdesc] (topicmeta] (topicref] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Preview mode (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Preview mode (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) You can preview a transformation at any time by clicking Preview, to see the effect it will have on your project data set. Preview mode is also a useful debugging tool, as it detects any runtime errors or corner case exceptions your transformation contains. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Editing, deleting and rearranging your transformations (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Editing, deleting and rearranging your transformations (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) You can edit individual transformations after you have added them to your transformation script. You can also edit the transformation script itself by rearranging and deleting transformations. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Applying transformation scripts to project data sets (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Applying transformation scripts to project data sets (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) You can apply your transformation script at any point to make changes in your project data set. When the script finishes running, users working with your project can view, search, use guided navigation and interact with the transformed data in Transform, Explore, and Discover areas of Studio. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Transform locking (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Transform locking (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) The Transform area in Studio provides a locking mechanism to ensure that multiple users working with the same data set within a project can't transform the data set at the same time. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Creating a new Hive table with the transformation script (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Creating a new Hive table with the transformation script (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) When you use Create a Data Set in the Transformation Editor, your transformation script is applied to the source Hive table your project data set was created from. This operation creates a new Hive table in the Dgraph index and adds a new data set to the Catalog. (shortdesc] (topicmeta] (topicref] (topicref][/map/topicref {"- map/topicref "}) [/map/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicmeta/navtitle {"- topic/navtitle "}) Transform Function Reference and Examples (navtitle][/map/topicref/topicmeta/linktext {"- map/linktext "}) Transform Function Reference and Examples (linktext][/map/topicref/topicmeta/shortdesc {"- map/shortdesc "}) This section lists data types, discusses data type conversions that take place when transformation scripts are applied, provides a list of reserved words and unsupported features of Groovy, and contains examples of custom transform function usage. It also includes the reference documentation for the custom transform functions in Big Data Discovery. (shortdesc] (topicmeta][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Data types (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Data types (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) In transformation scripts, the attribute's data type is represented as a Groovy data type. This topic discusses how the Dgraph data types match the Groovy data types. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Data type conversions (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Data type conversions (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) When you apply your transformation script to the project data set or to the source Hive table (when you create a new data set from within Transform), the data processing in Big Data Discovery converts most of the Hive data types to its corresponding Dgraph data types. However, this can result in some of the original data types being changed or omitted. This topic discusses these data type conversions in detail. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Unsupported Groovy language features and Reserved Keywords (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Unsupported Groovy language features and Reserved Keywords (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) This topic lists reserved keywords and those Groovy language features that are not supported in Big Data Discovery. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Examples (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Examples (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) This section contains examples of different types of transformations you can create using transform functions and Groovy. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) List of transform functions (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) List of transform functions (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) This section lists and describes custom transform functions in Big Data Discovery. (shortdesc] (topicmeta][/map/topicref/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Conversion functions (navtitle][/map/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Conversion functions (linktext][/map/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Conversion functions change a value from one data type to another. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Date functions (navtitle][/map/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Date functions (linktext][/map/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Date functions perform actions on Date objects, such as obtaining the month information from a specific date or adding time to a date. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Enrichment functions (navtitle][/map/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Enrichment functions (linktext][/map/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Enrichment functions are based on Data Enrichment modules used as part of data processing in Big Data Discovery. You can use these functions to extract meaningful information from your data and modify attributes to make them more useful for analysis. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Geocode functions (navtitle][/map/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Geocode functions (linktext][/map/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Geocode functions perform different actions on Geocode objects, such as calculating the distance between two Geocode values or obtaining a Geocode's latitude coordinate. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Math functions (navtitle][/map/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Math functions (linktext][/map/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Math functions perform mathematical operations on your data. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Set functions (navtitle][/map/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Set functions (linktext][/map/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Set functions perform various functions on values for multi-assign attributes, such as obtaining the size of the set, checking whether a set is empty, or converting an attribute from a multi-value to a single-value attribute. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) String functions (navtitle][/map/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) String functions (linktext][/map/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) String functions perform different actions on Strings, such as converting an entire String to uppercase or removing whitespace from a String. (shortdesc] (topicmeta] (topicref] (topicref] (topicref] (map]