[/map {"- map/map "}) [/map/topicref {"- map/topicref "}) [/map/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicmeta/navtitle {"- topic/navtitle "}) Introduction (navtitle][/map/topicref/topicmeta/linktext {"- map/linktext "}) Introduction (linktext][/map/topicref/topicmeta/shortdesc {"- map/shortdesc "}) This topic provides a high-level introduction to the Data Processing component of Big Data Discovery. (shortdesc] (topicmeta][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) About Integration with Hadoop (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) About Integration with Hadoop (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) This topic discusses how BDD fits into the Hadoop environment. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Data Processing within Big Data Discovery (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Data Processing within Big Data Discovery (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Data Processing collectively refers to a set of processes and jobs, all launched by Big Data Discovery once it is deployed. Many of these processes run in Hadoop and perform discovery, sampling, profiling, and enrichment of source data. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Preparing your data for ingest (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Preparing your data for ingest (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Although not required, it is recommended that you clean your source data so that it is in a state that makes Data Processing workflows run smoother and prevents ingest errors. (shortdesc] (topicmeta] (topicref] (topicref][/map/topicref {"- map/topicref "}) [/map/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicmeta/navtitle {"- topic/navtitle "}) Data Processing Workflows (navtitle][/map/topicref/topicmeta/linktext {"- map/linktext "}) Data Processing Workflows (linktext][/map/topicref/topicmeta/shortdesc {"- map/shortdesc "}) This section describes how Data Processing discovers data in Hive tables and prepares it for ingest into the Dgraph. (shortdesc] (topicmeta][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) About workflows (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) About workflows (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) This topic provides an overview of Data Processing workflows. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Working with Hive tables (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Working with Hive tables (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Hive tables contain the data for the Data Processing workflows. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Sampling and attribute handling (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Sampling and attribute handling (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Data Processing samples (processes) a number of records from a Hive table. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Data type discovery (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Data type discovery (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) When Data Processing retrieves data from a Hive table, the Hive data types are mapped to Dgraph data types when the data is ingested into the Dgraph. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Studio creation of Hive tables (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Studio creation of Hive tables (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Hive tables can be created from Studio. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Creation of a search interface (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Creation of a search interface (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Data Processing creates a search interface for each data set. (shortdesc] (topicmeta] (topicref] (topicref][/map/topicref {"- map/topicref "}) [/map/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicmeta/navtitle {"- topic/navtitle "}) Data Processing Configuration (navtitle][/map/topicref/topicmeta/linktext {"- map/linktext "}) Data Processing Configuration (linktext][/map/topicref/topicmeta/shortdesc {"- map/shortdesc "}) This section describes configuration for date formats and configuration for Spark. It also discusses how to add a SerDe JAR to the Data Processing workflows. (shortdesc] (topicmeta][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Date format configuration (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Date format configuration (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) The dateFormats.txt file provides a list of date formats supported by Data Processing workflows. This topic lists the defaults used in this file. You can add or remove a date format from this file if you use the formats supported by it. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Spark configuration (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Spark configuration (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Data Processing uses a Spark configuration file, sparkContext.properties. This topic describes how Data Processing obtains the settings for this file and includes a sample of the file. It also describes options you can adjust in this file to tweak the amount of memory required to successfully complete a Data Processing workflow. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Adding a SerDe JAR to DP workflows (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Adding a SerDe JAR to DP workflows (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) This topic describes the process of adding a custom Serializer-Deserializer (SerDe) to the Data Processing (DP) classpath, instead of the SerDe class that is shipped in the Data Processing package. (shortdesc] (topicmeta] (topicref] (topicref][/map/topicref {"- map/topicref "}) [/map/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicmeta/navtitle {"- topic/navtitle "}) DP Command Line Interface Utility (navtitle][/map/topicref/topicmeta/linktext {"- map/linktext "}) DP Command Line Interface Utility (linktext][/map/topicref/topicmeta/shortdesc {"- map/shortdesc "}) This section provides information on configuring and using the Data Processing Command Line Interface utility. (shortdesc] (topicmeta][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) About the DP CLI (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) About the DP CLI (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) The DP CLI (Command Line Interface) shell utility is used to launch Data Processing workflows. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) DP CLI Configuration (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) DP CLI Configuration (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) The DP CLI configuration properties are contained in the data_processing_CLI script. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) DP CLI flags (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) DP CLI flags (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) The DP CLI has a number of runtime flags that control its behavior. You can list these flags if you use the --help flag. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Using whitelists and blacklists (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Using whitelists and blacklists (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) A whitelist specifies which Hive tables should be processed in Big Data Discovery, while a blacklist specifies which Hive tables should be ignored during data processing. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) DP CLI cron job (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) DP CLI cron job (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) You can specify that the BDD installer creates a cron job to run the DP CLI. (shortdesc] (topicmeta] (topicref] (topicref][/map/topicref {"- map/topicref "}) [/map/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicmeta/navtitle {"- topic/navtitle "}) Data Processing Logging (navtitle][/map/topicref/topicmeta/linktext {"- map/linktext "}) Data Processing Logging (linktext][/map/topicref/topicmeta/shortdesc {"- map/shortdesc "}) This section describes logging for the Data Processing component of Big Data Discovery. (shortdesc] (topicmeta][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Logging configuration (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Logging configuration (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Data Processing has a default configuration file, logging.properties, that sets the logging properties. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Data Processing logging (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Data Processing logging (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) This topic provides an overview of the Data Processing logging files. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Useful CDH logs (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Useful CDH logs (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) There are some CDH log files that may contain valuable information for debugging issues with the Data Processing component of Big Data Discovery. (shortdesc] (topicmeta] (topicref] (topicref] [/map/topicref {"- map/topicref "}) [/map/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicmeta/navtitle {"- topic/navtitle "}) Data Enrichment Modules (navtitle][/map/topicref/topicmeta/linktext {"- map/linktext "}) Data Enrichment Modules (linktext][/map/topicref/topicmeta/shortdesc {"- map/shortdesc "}) This section describes the Data Enrichment modules. (shortdesc] (topicmeta][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) About the Data Enrichment modules (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) About the Data Enrichment modules (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) The Data Enrichment modules increase the usability of your data by discovering value in its content. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Entity extractor (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Entity extractor (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) The Entity extractor module extracts the names of people, companies and places from the input text inside records in source data. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Noun Group extractor (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Noun Group extractor (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) This plugin extracts noun groups from the input text. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) TF.IDF Term extractor (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) TF.IDF Term extractor (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) This module extracts key words from the input text. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Sentiment Analysis (document level) (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Sentiment Analysis (document level) (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) The document-level Sentiment Analysis module analyzes a piece of text and determines whether the text has a positive or negative sentiment. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Sentiment Analysis (sub-document level) (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Sentiment Analysis (sub-document level) (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) The sub-document-level Sentiment Analysis module returns a list of sentiment-bearing phrases which fall into one of the two categories: positive or negative. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Address GeoTagger (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Address GeoTagger (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) The Address GeoTagger returns geographical information for a valid global address. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) IP Address GeoTagger (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) IP Address GeoTagger (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) The IP Address GeoTagger returns geographical information for a valid IP address. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Reverse GeoTagger (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Reverse GeoTagger (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) The Reverse GeoTagger returns geographical information for a valid geocode latitude/longitude coordinates that resolve to a metropolitan area. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Tag Stripper (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Tag Stripper (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) The Tag Stripper module removes any HTML, XML and XHTML markup from the input text. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Phonetic Hash (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Phonetic Hash (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) The Phonetic Hash module returns a string attribute that contains the hash value of an input string. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Language Detection (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Language Detection (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) The Language Detection module can detect the language of input text. (shortdesc] (topicmeta] (topicref] (topicref] [/map/topicref {"- map/topicref "}) [/map/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicmeta/navtitle {"- topic/navtitle "}) Data Model in Big Data Discovery (navtitle][/map/topicref/topicmeta/linktext {"- map/linktext "}) Data Model in Big Data Discovery (linktext][/map/topicref/topicmeta/shortdesc {"- map/shortdesc "}) This section introduces basic concepts associated with the schema of records in the Dgraph, and describes how data is structured and configured in the Dgraph data model. When a Data Processing workflow runs, a resulting data set is created in the Dgraph. The records in this data set, as well as their attributes, are discussed in this section. (shortdesc] (topicmeta][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) About the data model (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) About the data model (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) The data model in the Dgraph consists of data sets, records, and attributes. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Data records (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Data records (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Records are the fundamental units of data in the Dgraph. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Attributes (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Attributes (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) An attribute is the basic unit of a record schema. Assignments from attributes (also known as key-value pairs) describe records in the Dgraph. (shortdesc] (topicmeta][/map/topicref/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Assignments on attributes (navtitle][/map/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Assignments on attributes (linktext][/map/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) Records are assigned values from attributes. An assignment indicates that a record has a value from an attribute. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Primary keys (navtitle][/map/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Primary keys (linktext][/map/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) In the Dgraph data model, a primary-key attribute is used, for identifying records and collections (data sets). This topic provides a summary of each of these primary keys. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Attribute data types (navtitle][/map/topicref/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Attribute data types (linktext][/map/topicref/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) The attribute type identifies the type of data allowed for the Dgraph attribute value (key-value pair). (shortdesc] (topicmeta] (topicref] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Supported languages (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Supported languages (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) The Dgraph uses a language code to identify a language for a specific attribute. (shortdesc] (topicmeta] (topicref] (topicref][/map/topicref {"- map/topicref "}) [/map/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicmeta/navtitle {"- topic/navtitle "}) Dgraph HDFS Agent (navtitle][/map/topicref/topicmeta/linktext {"- map/linktext "}) Dgraph HDFS Agent (linktext][/map/topicref/topicmeta/shortdesc {"- map/shortdesc "}) This section describes the role of the Dgraph HDFS Agent in the exporting and ingesting of data. (shortdesc] (topicmeta][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) About the Dgraph HDFS Agent (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) About the Dgraph HDFS Agent (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) The Dgraph HDFS Agent acts as a data transport layer between the Dgraph and an HDFS environment. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Importing records from HDFS for loading into BDD (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Importing records from HDFS for loading into BDD (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) The Dgraph HDFS Agent plays a major part in the loading of data from a Data Processing workflow into the Dgraph. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Exporting data from Studio into HDFS (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Exporting data from Studio into HDFS (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) The Dgraph HDFS Agent is the conduit for exporting data from a Studio project. (shortdesc] (topicmeta] (topicref][/map/topicref/topicref {"- map/topicref "}) [/map/topicref/topicref/topicmeta {"- map/topicmeta "}) [/map/topicref/topicref/topicmeta/navtitle {"- topic/navtitle "}) Dgraph HDFS Agent logging (navtitle][/map/topicref/topicref/topicmeta/linktext {"- map/linktext "}) Dgraph HDFS Agent logging (linktext][/map/topicref/topicref/topicmeta/shortdesc {"- map/shortdesc "}) The Dgraph HDFS Agent writes its stdout/stderr output to a log file. (shortdesc] (topicmeta] (topicref] (topicref] (map]