[/map {"- map/map "}) [/map/topicref {"- map/topicref "}) [/map/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicmeta/navtitle {"- topic/navtitle "}) Working with Transformation Scripts (navtitle][/map/topicref/topicmeta/linktext {"- map/linktext "}) Working with Transformation Scripts (linktext][/map/topicref/topicmeta/shortdesc {"- map/shortdesc "}) These topics describe the process of creating and applying transformation scripts, using the custom transform functions. (shortdesc] (topicmeta][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Transformation script workflow (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Transformation script workflow (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) At a high level, writing a transformation script and applying it to your data involves the following steps: (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Writing transformations (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Writing transformations (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) You can write transformations in the Transform area of Studio, using the Transformation Editor. Transformations can contain attributes and records from your project data set as variables, and can create new attributes to hold the transformed values. (shortdesc] (topicmeta][/map/topicref/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) The Transformation Editor (navtitle][/map/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) The Transformation Editor (linktext][/map/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) You create transformations in the Transformation Editor, this is the built-in Groovy editor within the Transform area in Studio. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Formats for variables (navtitle][/map/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Formats for variables (linktext][/map/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) You can use attributes from your project data set as variables in your transformation scripts. This allows you to pass attributes to transform functions as parameters and perform other operations on them. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Setting transformation outputs (navtitle][/map/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Setting transformation outputs (linktext][/map/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) You can set your transformation to output to either the selected attribute or a new attribute (this is useful if a transformation is creating a new column). (shortdesc] (topicmeta] (topicref][/map/topicref/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Functional and dot notation and function chaining (navtitle][/map/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Functional and dot notation and function chaining (linktext][/map/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) You must use proper syntax when adding transform functions to your script, or your script won't run properly. You can reference all transform functions using functional notation, as described in this topic. (shortdesc] (topicmeta] (topicref] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Exception handling and debugging (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Exception handling and debugging (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) These topics describe exception handling in Transform and show how to debug individual transformations. (shortdesc] (topicmeta][/map/topicref/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Script evaluation (navtitle][/map/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Script evaluation (linktext][/map/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Transformation scripts are evaluated top-down on each input row. This means that each transformation in the script is applied in order to the first input row, then again to the second row, and so on. This is illustrated by the following pseudo code: (shortdesc] (topicmeta] (topicref][/map/topicref/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Dynamic typing vs. static typing (navtitle][/map/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Dynamic typing vs. static typing (linktext][/map/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) This topic is provided for reverence only as it explains the differences between dynamic and static typing. Understanding the differences between dynamic and static typing is key to understanding the way in which transformation script errors are handled, and how it is different from the way Groovy handles errors. This will also help you interpret errors created by your transformation script. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Exception handling and troubleshooting your scripts (navtitle][/map/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Exception handling and troubleshooting your scripts (linktext][/map/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Transform uses a static parser to override some of Groovy's dynamic typing behavior and detect parsing errors, such as undefined variables, when you preview or save your transformations. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Transform logging (navtitle][/map/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Transform logging (linktext][/map/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) If your transformation script fails to commit, you can learn more about the cause of the failure by looking through the Data Processing logs. (shortdesc] (topicmeta] (topicref] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Preview mode (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Preview mode (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) You can preview a transformation at any time by clicking Preview, to see the effect it will have on your project data set. Preview mode is also a useful debugging tool, as it detects any runtime errors or corner case exceptions your transformation contains. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Editing, deleting and rearranging your transformations (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Editing, deleting and rearranging your transformations (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) You can edit individual transformations after you have added them to your transformation script. You can also edit the transformation script itself by rearranging and deleting transformations. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Applying transformation scripts to project data sets (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Applying transformation scripts to project data sets (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) You can apply your transformation script at any point to make changes in your project data set. When the script finishes running, users working with your project can view, search, use guided navigation and interact with the transformed data in Transform, Explore, and Discover areas of Studio. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Transform locking (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Transform locking (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) The Transform area in Studio provides a locking mechanism to ensure that multiple users working with the same data set within a project can't transform the data set at the same time. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Creating a new Hive table with the transformation script (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Creating a new Hive table with the transformation script (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) When you use Create a Data Set in the Transformation Editor, your transformation script is applied to the source Hive table your project data set was created from. This operation creates a new Hive table in the Dgraph index and adds a new data set to the Catalog. (shortdesc] (topicmeta] (topicref] (topicref] (map]