[/map {"- map/map "}) [/map/topicref {"- map/topicref "}) [/map/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicmeta/navtitle {"- topic/navtitle "}) About this guide (navtitle][/map/topicref/topicmeta/linktext {"- map/linktext "}) About this guide (linktext][/map/topicref/topicmeta/shortdesc {"- map/shortdesc "}) This guide describes available extension and customization options for Oracle Big Data Discovery. It describes how to install and use the Studio's Component SDK that lets you create custom Security Managers, develop custom Studio components, and work with QueryFunction Java classes. Additionally, this guide has a section on how to use the custom transform functions (also known as the Transform API for Big Data Discovery). (shortdesc] (topicmeta] (topicref] [/map/topichead {"+ map/topicref mapgroup-d/topichead "}) [/map/topichead/topicmeta {"- map/topicmeta "}) [/map/topichead/topicmeta/navtitle {"- topic/navtitle "}) Using the Component SDK (navtitle] (topicmeta][/map/topichead/topicref {"- map/topicref "}) [/map/topichead/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicmeta/navtitle {"- topic/navtitle "}) Installing and Configuring the Component SDK (navtitle][/map/topichead/topicref/topicmeta/linktext {"- map/linktext "}) Installing and Configuring the Component SDK (linktext][/map/topichead/topicref/topicmeta/shortdesc {"- map/shortdesc "}) The Component SDK supports custom development for components and data security. (shortdesc] (topicmeta][/map/topichead/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) About the Component SDK (navtitle][/map/topichead/topicref/topicref/topicmeta/linktext {"- map/linktext "}) About the Component SDK (linktext][/map/topichead/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) The Component SDK allows developers to extend Studio by creating and deploying custom Security Managers and components. (shortdesc] (topicmeta] (topicref][/map/topichead/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Requirements for using the Component SDK (navtitle][/map/topichead/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Requirements for using the Component SDK (linktext][/map/topichead/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Before using the Component SDK, make sure that you meet the system and skill set requirements. (shortdesc] (topicmeta] (topicref][/map/topichead/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Installing the Component SDK (navtitle][/map/topichead/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Installing the Component SDK (linktext][/map/topichead/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) The Component SDK is contained in a .zip file in the Big Data Discovery Media Pack. (shortdesc] (topicmeta] (topicref][/map/topichead/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Preparing your system for Component SDK development (navtitle][/map/topichead/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Preparing your system for Component SDK development (linktext][/map/topichead/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) After installing the Component SDK, before you can start development, you must complete some initial preparation on your system. (shortdesc] (topicmeta] (topicref] (topicref][/map/topichead/topicref {"- map/topicref "}) [/map/topichead/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicmeta/navtitle {"- topic/navtitle "}) Developing a Custom Security Manager (navtitle][/map/topichead/topicref/topicmeta/linktext {"- map/linktext "}) Developing a Custom Security Manager (linktext][/map/topichead/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Using the Component SDK, you can create a custom Security Manager to customize how Big Data Discovery filters data that is displayed to users. (shortdesc] (topicmeta][/map/topichead/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Creating and implementing a new Security Manager (navtitle][/map/topichead/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Creating and implementing a new Security Manager (linktext][/map/topichead/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) The Component SDK includes a batch script for creating a new Security Manager. (shortdesc] (topicmeta] (topicref][/map/topichead/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Security Manager interface (navtitle][/map/topichead/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Security Manager interface (linktext][/map/topichead/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) The com.endeca.portal.data.security.BddSecurityManager interface represents a Security Manager capable of applying record-level security filters for BDD. (shortdesc] (topicmeta] (topicref][/map/topichead/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Building and deploying a new Security Manager (navtitle][/map/topichead/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Building and deploying a new Security Manager (linktext][/map/topichead/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Before you can use your custom Security Manager, you must deploy it to Studio. To do this, you generate a .jar file for it, then add the .jar file to the Studio .ear file. (shortdesc] (topicmeta] (topicref][/map/topichead/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Configuring Studio to use a different Security Manager (navtitle][/map/topichead/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Configuring Studio to use a different Security Manager (linktext][/map/topichead/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) In order to for Studio to use your Security Manager, you must configure Studio to pick up and use the new class. (shortdesc] (topicmeta] (topicref] (topicref][/map/topichead/topicref {"- map/topicref "}) [/map/topichead/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicmeta/navtitle {"- topic/navtitle "}) Developing Custom Components (navtitle][/map/topichead/topicref/topicmeta/linktext {"- map/linktext "}) Developing Custom Components (linktext][/map/topichead/topicref/topicmeta/shortdesc {"- map/shortdesc "}) The most common use of the Component SDK is to create and deploy custom components. (shortdesc] (topicmeta][/map/topichead/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Generating the Eclipse project for the component (navtitle][/map/topichead/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Generating the Eclipse project for the component (linktext][/map/topichead/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) The Component SDK includes a script to generate an Eclipse project for a new component. (shortdesc] (topicmeta] (topicref][/map/topichead/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Obtaining query results for components (navtitle][/map/topichead/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Obtaining query results for components (linktext][/map/topichead/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) When developing a component, use the QueryState and QueryResults classes to request and receive data from data sets. (shortdesc] (topicmeta] (topicref][/map/topichead/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Building a component (navtitle][/map/topichead/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Building a component (linktext][/map/topichead/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) After completing the component development, you set the build properties, then build the component in Eclipse. (shortdesc] (topicmeta] (topicref][/map/topichead/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Deploying and removing custom components (navtitle][/map/topichead/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Deploying and removing custom components (linktext][/map/topichead/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Once you have built the component .war file, you can add the component to a Big Data Discovery instance. You can also remove a component. (shortdesc] (topicmeta] (topicref] (topicref][/map/topichead/topicref {"- map/topicref "}) [/map/topichead/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicmeta/navtitle {"- topic/navtitle "}) Working with QueryFunction Classes (navtitle][/map/topichead/topicref/topicmeta/linktext {"- map/linktext "}) Working with QueryFunction Classes (linktext][/map/topichead/topicref/topicmeta/shortdesc {"- map/shortdesc "}) When developing custom components, you can use the provided QueryFunction classes to filter and query data. You can also create and implement your own QueryFunction classes. (shortdesc] (topicmeta][/map/topichead/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Provided QueryFunction filter classes (navtitle][/map/topichead/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Provided QueryFunction filter classes (linktext][/map/topichead/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Big Data Discovery provides the following QueryFunction filter classes. Filters are used to change the current query state. (shortdesc] (topicmeta] (topicref][/map/topichead/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Provided QueryConfig functions (navtitle][/map/topichead/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Provided QueryConfig functions (linktext][/map/topichead/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Studio provides the following QueryConfig functions, used to manage the results returned by a query. These are more advanced functions for component development. (shortdesc] (topicmeta] (topicref][/map/topichead/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Creating and deploying a custom QueryFunction class (navtitle][/map/topichead/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Creating and deploying a custom QueryFunction class (linktext][/map/topichead/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) The Component SDK allows you to create custom QueryFunction classes. (shortdesc] (topicmeta][/map/topichead/topicref/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Generating the Eclipse project for the QueryFunction class (navtitle][/map/topichead/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Generating the Eclipse project for the QueryFunction class (linktext][/map/topichead/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) The Component SDK includes a script to generate the Eclipse project for the QueryFunction class. (shortdesc] (topicmeta] (topicref][/map/topichead/topicref/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Implementing a custom QueryFunction class (navtitle][/map/topichead/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Implementing a custom QueryFunction class (linktext][/map/topichead/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) After you create your new QueryFunction class, you then implement it. (shortdesc] (topicmeta] (topicref][/map/topichead/topicref/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Building and deploying a custom QueryFunction class (navtitle][/map/topichead/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Building and deploying a custom QueryFunction class (linktext][/map/topichead/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) When you have finished development on your custom QueryFunction class, you build it, then add the resulting .jar file to the .ear file. (shortdesc] (topicmeta] (topicref][/map/topichead/topicref/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Adding a custom QueryFunction to a custom component project (navtitle][/map/topichead/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Adding a custom QueryFunction to a custom component project (linktext][/map/topichead/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) If you just want to use a custom QueryFunction in a specific custom component, you add its .jar file to the component's Eclipse build path. (shortdesc] (topicmeta] (topicref] (topicref] (topicref] (topichead] [/map/topichead {"+ map/topicref mapgroup-d/topichead "}) [/map/topichead/topicmeta {"- map/topicmeta "}) [/map/topichead/topicmeta/navtitle {"- topic/navtitle "}) Using the Transform API (navtitle] (topicmeta][/map/topichead/topicref {"- map/topicref "}) [/map/topichead/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicmeta/navtitle {"- topic/navtitle "}) Overview (navtitle][/map/topichead/topicref/topicmeta/linktext {"- map/linktext "}) Overview (linktext][/map/topichead/topicref/topicmeta/shortdesc {"- map/shortdesc "}) This section describes transformations and the custom transform functions available in Big Data Discovery. (shortdesc] (topicmeta][/map/topichead/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) About transformations and transformation scripts (navtitle][/map/topichead/topicref/topicref/topicmeta/linktext {"- map/linktext "}) About transformations and transformation scripts (linktext][/map/topichead/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Transformations are changes you can make to your project data set, after the source data has been processed and loaded into Studio. Transformations can be thought of as a substitute for an ETL process of cleaning your data. Transformations can overwrite an existing attribute, modify attributes, or create new attributes. (shortdesc] (topicmeta] (topicref][/map/topichead/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) About Groovy (navtitle][/map/topichead/topicref/topicref/topicmeta/linktext {"- map/linktext "}) About Groovy (linktext][/map/topichead/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Groovy is a dynamically-typed scripting language. Code written in the Java language is valid in Groovy, so users not familiar with Groovy may resort to the Java syntax. All custom transform functions available for you in Big Data Discovery are written in Groovy. (shortdesc] (topicmeta] (topicref][/map/topichead/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) About transform functions (navtitle][/map/topichead/topicref/topicref/topicmeta/linktext {"- map/linktext "}) About transform functions (linktext][/map/topichead/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Transform functions are customized Groovy functions available in Big Data Discovery that you can include in your transformation scripts. Each transform function performs a specific operation on your data, from simple ones, such as converting an attribute to a different data type, to more complex ones, such as determining the overall sentiment of a document or a string of text. (shortdesc] (topicmeta] (topicref] (topicref][/map/topichead/topicref {"- map/topicref "}) [/map/topichead/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicmeta/navtitle {"- topic/navtitle "}) Working with Transformation Scripts (navtitle][/map/topichead/topicref/topicmeta/linktext {"- map/linktext "}) Working with Transformation Scripts (linktext][/map/topichead/topicref/topicmeta/shortdesc {"- map/shortdesc "}) These topics describe the process of creating and applying transformation scripts, using the custom transform functions. (shortdesc] (topicmeta][/map/topichead/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Transformation script workflow (navtitle][/map/topichead/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Transformation script workflow (linktext][/map/topichead/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) At a high level, writing a transformation script and applying it to your data involves the following steps: (shortdesc] (topicmeta] (topicref][/map/topichead/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Writing transformations (navtitle][/map/topichead/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Writing transformations (linktext][/map/topichead/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) You can write transformations in the Transform area of Studio, using the Transformation Editor. Transformations can contain attributes and records from your project data set as variables, and can create new attributes to hold the transformed values. (shortdesc] (topicmeta][/map/topichead/topicref/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) The Transformation Editor (navtitle][/map/topichead/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) The Transformation Editor (linktext][/map/topichead/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) You create transformations in the Transformation Editor, this is the built-in Groovy editor within the Transform area in Studio. (shortdesc] (topicmeta] (topicref][/map/topichead/topicref/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Formats for variables (navtitle][/map/topichead/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Formats for variables (linktext][/map/topichead/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) You can use attributes from your project data set as variables in your transformation scripts. This allows you to pass attributes to transform functions as parameters and perform other operations on them. (shortdesc] (topicmeta] (topicref][/map/topichead/topicref/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Setting transformation outputs (navtitle][/map/topichead/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Setting transformation outputs (linktext][/map/topichead/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) You can set your transformation to output to either the selected attribute or a new attribute (this is useful if a transformation is creating a new column). (shortdesc] (topicmeta] (topicref][/map/topichead/topicref/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Functional and dot notation and function chaining (navtitle][/map/topichead/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Functional and dot notation and function chaining (linktext][/map/topichead/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) You must use proper syntax when adding transform functions to your script, or your script won't run properly. You can reference all transform functions using functional notation, as described in this topic. (shortdesc] (topicmeta] (topicref] (topicref][/map/topichead/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Exception handling and debugging (navtitle][/map/topichead/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Exception handling and debugging (linktext][/map/topichead/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) These topics describe exception handling in Transform and show how to debug individual transformations. (shortdesc] (topicmeta][/map/topichead/topicref/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Script evaluation (navtitle][/map/topichead/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Script evaluation (linktext][/map/topichead/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Transformation scripts are evaluated top-down on each input row. This means that each transformation in the script is applied in order to the first input row, then again to the second row, and so on. This is illustrated by the following pseudo code: (shortdesc] (topicmeta] (topicref][/map/topichead/topicref/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Dynamic typing vs. static typing (navtitle][/map/topichead/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Dynamic typing vs. static typing (linktext][/map/topichead/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) This topic is provided for reverence only as it explains the differences between dynamic and static typing. Understanding the differences between dynamic and static typing is key to understanding the way in which transformation script errors are handled, and how it is different from the way Groovy handles errors. This will also help you interpret errors created by your transformation script. (shortdesc] (topicmeta] (topicref][/map/topichead/topicref/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Exception handling and troubleshooting your scripts (navtitle][/map/topichead/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Exception handling and troubleshooting your scripts (linktext][/map/topichead/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Transform uses a static parser to override some of Groovy's dynamic typing behavior and detect parsing errors, such as undefined variables, when you preview or save your transformations. (shortdesc] (topicmeta] (topicref][/map/topichead/topicref/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Transform logging (navtitle][/map/topichead/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Transform logging (linktext][/map/topichead/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) If your transformation script fails to commit, you can learn more about the cause of the failure by looking through the Data Processing logs. (shortdesc] (topicmeta] (topicref] (topicref][/map/topichead/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Preview mode (navtitle][/map/topichead/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Preview mode (linktext][/map/topichead/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) You can preview a transformation at any time by clicking Preview, to see the effect it will have on your project data set. Preview mode is also a useful debugging tool, as it detects any runtime errors or corner case exceptions your transformation contains. (shortdesc] (topicmeta] (topicref][/map/topichead/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Editing, deleting and rearranging your transformations (navtitle][/map/topichead/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Editing, deleting and rearranging your transformations (linktext][/map/topichead/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) You can edit individual transformations after you have added them to your transformation script. You can also edit the transformation script itself by rearranging and deleting transformations. (shortdesc] (topicmeta] (topicref][/map/topichead/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Applying transformation scripts to project data sets (navtitle][/map/topichead/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Applying transformation scripts to project data sets (linktext][/map/topichead/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) You can apply your transformation script at any point to make changes in your project data set. When the script finishes running, users working with your project can view, search, use guided navigation and interact with the transformed data in Transform, Explore, and Discover areas of Studio. (shortdesc] (topicmeta] (topicref][/map/topichead/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Transform locking (navtitle][/map/topichead/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Transform locking (linktext][/map/topichead/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) The Transform area in Studio provides a locking mechanism to ensure that multiple users working with the same data set within a project can't transform the data set at the same time. (shortdesc] (topicmeta] (topicref][/map/topichead/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Creating a new Hive table with the transformation script (navtitle][/map/topichead/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Creating a new Hive table with the transformation script (linktext][/map/topichead/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) When you use Create a Data Set in the Transformation Editor, your transformation script is applied to the source Hive table your project data set was created from. This operation creates a new Hive table in the Dgraph index and adds a new data set to the Catalog. (shortdesc] (topicmeta] (topicref] (topicref][/map/topichead/topicref {"- map/topicref "}) [/map/topichead/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicmeta/navtitle {"- topic/navtitle "}) Transform Function Reference and Examples (navtitle][/map/topichead/topicref/topicmeta/linktext {"- map/linktext "}) Transform Function Reference and Examples (linktext][/map/topichead/topicref/topicmeta/shortdesc {"- map/shortdesc "}) This section lists data types, discusses data type conversions that take place when transformation scripts are applied, provides a list of reserved words and unsupported features of Groovy, and contains examples of custom transform function usage. It also includes the reference documentation for the custom transform functions in Big Data Discovery. (shortdesc] (topicmeta][/map/topichead/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Data types (navtitle][/map/topichead/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Data types (linktext][/map/topichead/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) In transformation scripts, the attribute's data type is represented as a Groovy data type. This topic discusses how the Dgraph data types match the Groovy data types. (shortdesc] (topicmeta] (topicref][/map/topichead/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Data type conversions (navtitle][/map/topichead/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Data type conversions (linktext][/map/topichead/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) When you apply your transformation script to the project data set or to the source Hive table (when you create a new data set from within Transform), the data processing in Big Data Discovery converts most of the Hive data types to its corresponding Dgraph data types. However, this can result in some of the original data types being changed or omitted. This topic discusses these data type conversions in detail. (shortdesc] (topicmeta] (topicref][/map/topichead/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Unsupported Groovy language features and Reserved Keywords (navtitle][/map/topichead/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Unsupported Groovy language features and Reserved Keywords (linktext][/map/topichead/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) This topic lists reserved keywords and those Groovy language features that are not supported in Big Data Discovery. (shortdesc] (topicmeta] (topicref][/map/topichead/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Examples (navtitle][/map/topichead/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Examples (linktext][/map/topichead/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) This section contains examples of different types of transformations you can create using transform functions and Groovy. (shortdesc] (topicmeta] (topicref][/map/topichead/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) List of transform functions (navtitle][/map/topichead/topicref/topicref/topicmeta/linktext {"- map/linktext "}) List of transform functions (linktext][/map/topichead/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) This section lists and describes custom transform functions in Big Data Discovery. (shortdesc] (topicmeta][/map/topichead/topicref/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Conversion functions (navtitle][/map/topichead/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Conversion functions (linktext][/map/topichead/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Conversion functions change a value from one data type to another. (shortdesc] (topicmeta] (topicref][/map/topichead/topicref/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Date functions (navtitle][/map/topichead/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Date functions (linktext][/map/topichead/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Date functions perform actions on Date objects, such as obtaining the month information from a specific date or adding time to a date. (shortdesc] (topicmeta] (topicref][/map/topichead/topicref/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Enrichment functions (navtitle][/map/topichead/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Enrichment functions (linktext][/map/topichead/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Enrichment functions are based on Data Enrichment modules used as part of data processing in Big Data Discovery. You can use these functions to extract meaningful information from your data and modify attributes to make them more useful for analysis. (shortdesc] (topicmeta] (topicref][/map/topichead/topicref/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Geocode functions (navtitle][/map/topichead/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Geocode functions (linktext][/map/topichead/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Geocode functions perform different actions on Geocode objects, such as calculating the distance between two Geocode values or obtaining a Geocode's latitude coordinate. (shortdesc] (topicmeta] (topicref][/map/topichead/topicref/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Math functions (navtitle][/map/topichead/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Math functions (linktext][/map/topichead/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Math functions perform mathematical operations on your data. (shortdesc] (topicmeta] (topicref][/map/topichead/topicref/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Set functions (navtitle][/map/topichead/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Set functions (linktext][/map/topichead/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Set functions perform various functions on values for multi-assign attributes, such as obtaining the size of the set, checking whether a set is empty, or converting an attribute from a multi-value to a single-value attribute. (shortdesc] (topicmeta] (topicref][/map/topichead/topicref/topicref/topicref {"- map/topicref "}) [/map/topichead/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topichead/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) String functions (navtitle][/map/topichead/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) String functions (linktext][/map/topichead/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) String functions perform different actions on Strings, such as converting an entire String to uppercase or removing whitespace from a String. (shortdesc] (topicmeta] (topicref] (topicref] (topicref] (topichead] [/map/topicref {"- map/topicref "}) [/map/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicmeta/navtitle {"- topic/navtitle "}) Conventions used in this document (navtitle][/map/topicref/topicmeta/linktext {"- map/linktext "}) Conventions used in this document (linktext][/map/topicref/topicmeta/shortdesc {"- map/shortdesc "}) The following conventions are used in this document. (shortdesc] (topicmeta] (topicref] [/map/topicref {"- map/topicref "}) [/map/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicmeta/navtitle {"- topic/navtitle "}) Copyright and disclaimer (navtitle][/map/topicref/topicmeta/linktext {"- map/linktext "}) Copyright and disclaimer (linktext] (topicmeta] (topicref] (map]