Administering Stream Analytics is essential to get the required results.
Typical Workflow for Administering Stream Analytics
The typical workflow lists the artifacts required to create a pipeline in Stream Analytics.
The prerequisites for a pipeline are:
A reference is used to create a stream.
A target is required to create a connection.
A connection is required to create a stream.
A stream is required to create a pipeline.
Configure Stream Analytics System Settings
Only users with the Administrator role can set the console settings in Stream Analytics.
- Click the user name in the top right corner of the screen.
- Click System Settings.The System Settings page opens.
- Click Environment.
- Specify the server names and URLs where the Kafka Zookeeper, Yarn Resource Manager, and Spark Standalone are deployed and running.
Kafka Zookeeper Connection — the URL where the Zookeeper server or servers are configured, separated by comma. This value is required to push events to the stream.
Runtime Server — the runtime server you want your Stream Analytics instance to run on
YARN Resource Manager URL — the URL where the YARN Resource Manager is configured, if the runtime server is Yarn
Spark REST URL — the URL where the Spark REST Proxy is configured, if the runtime server is Spark
Storage — the type of storage for all your artifacts within Stream Analytics
Path — the path where the storage exists
HA Namenodes — the namenodes of HA cluster
Hadoop Authentication — the type of Hadoop authentication you would like
- Click Pipelines. Specify the various settings for the pipeline within Stream Analytics.
Batch Duration — the default duration of the batch for each pipeline
Executor Count — the default number of executors per pipeline
Cores per Executor — the default number of cores. A minimum value of 2 is required.
Executor Memory — the default allocated memory for each executor instance in megabytes
Cores per Driver — the default number of cores
Driver Memory — the default allocated memory per driver instance in megabytes
High Availability — toggle the default HA value as on/off for each pipeline
- Click Analytics. Enable Analytics and specify the Druid Zookeeper Connection details. This is required to work with cubes.
- Click Proxy. If you set proper proxy, the back-end system will use these settings to test the REST target.
- Click Save.
Configure User Preferences
Based on the preferences that users set in this page, the characteristics of Stream Analytics vary.
To set/update user preferences:
Click the user name in the top right corner of the screen.
Click Preferences. The Preferences page opens.
Provides a set of general preferences that you can view and set according to your requirements.
Select if you want the Home page, the Catalog page, or the Patterns page to appear as the Start Page.
Provides a set of view mode preferences that you can view and set according to your requirements.
Select this option to view the pipeline in a browser mode, just like any other web pipeline appears in a browser.
Presentation mode (projector)
Select this option to view the pipeline in a presentation mode, as if the pipeline is being viewed on a projector as a presentation. When you select the Presentation mode, a different color skin is applied to the pipeline.
Provides a set of notifications preferences that you can view and set according to your requirements.
Show Information Notifications
Select this option if you want the information notifications to appear in the pipeline. This option is selected by default.
Information Notification duration (in seconds)
Choose the number of seconds for which the notifications appear. The default value is 5.
Provides a set of catalog preferences that you can view and set according to your requirements.
Default Sorting Column
Select the column by which you want the columns to be sorted. This value will be used as the default for all columns until you change the value again.
Default Sorting Order
Select the order by which you want the columns to be sorted. This value will be used as the default value for all columns until you change the value again.
Default Page Size
Select the value to be used as the default page size. Based on the value selected, the number of records that appear on a page vary. This value will be used as the default for all pages until you change the value again.
Provides a set of pipeline preferences that you can view and set according to your requirements.
Select Yes if you want to display the User Assistance text for the pipelines in the Pipeline Editor.
Live Output Stream
Provides a set of pipeline live output stream preferences that you can view and set according to your requirements.
Select a value that you want to be applied as the default table size for the data in Live Output Stream of a pipeline.
Provides a set of pipeline timestamp preferences that you can view and set according to your requirements.