Building Your Stream Analytics Pipeline

Before You Begin

This 20-minute tutorial shows you how to build a pipeline that monitors public transportation in the Atlanta area. This is the second tutorial in Monitoring Public Transportation Using Stream Analytics. Read the tutorials sequentially.

• Creating Artifacts for Your Stream Analytics Pipeline
• Building a Stream Analytics Pipeline
• Creating Business Rules in Your Stream Analytics Pipeline
• Configuring Graphical Views in Your Stream Analytics Pipeline
• Defining and Configuring a Target in Your Stream Analytics Pipeline
• Publishing Your Stream Analytics Pipeline

Background

Stream Analytics is a graphical tool with an intuitive web-based interface that enables you to explore, analyze, and manipulate streaming data sources in real time.

This traffic management solution uses GPS fleet data. This data is low cost, accurate, and it's created in real time. Its value for government sector customers is that this fleet data reduces congestion on roadways and enhances the traveling experience. The general features represented in this solution are real-time traffic analytics, speed violation tracking, and congestion detection. These features are combined with GPS streaming sensor feeds and historical trend data using map-based visualizations. This solution uses enterprise-grade Spark Streaming, Kafka open-source messaging, a highly scalable, extensible platform built with Stream Analytics.

Some of the key benefits with this solution are low-cost rollout with zero-road, network disruption, real-time operational intelligence, which is essential for meaningful congestion reduction, and an enhanced traveler experience, with a leading streaming big data technology.

What Do You Need?

Perform the tasks described in Creating Artifacts for Your Stream Analytics Pipeline.

Create a Pipeline

1. Log in to Stream Analytics.
   
2. On the Navigation pane on the left side, click Catalog.
3. In the Create New Item menu, select Pipeline.
   

   

4. In the Create New Pipeline dialog box, enter or select the following values and click Save:
   
   • Name: Tutorial
   • Description: Tutorial: Vehicle Monitoring Pipeline
   • Tags: transportation, tutorial
   • Stream: Tutorial
   The Tutorial pipeline is created and displayed. Listening for events begins, and then streaming events start to flow.

Create a Stage to Identify Vehicles Only in Atlanta

1. In the Add a Stage menu, select Pattern.
   The Select a Pattern to Create dialog box is displayed.
2. Under Spatial, click Geo Filter.

   

3. In the Create Pattern Stage dialog box, enter the following values and then click Save:
   • Name: PatternGeoFenceInside
   • Description: Spatial analytics in the city of Atlanta

     

4. On the PatternGeoFenceInside page, on the Parameters tab, enter or select the following values:
   
   • Geo Fence: Tutorial
   • Latitude: BUSlat
   • Longitude: BUSlong
   • Object Key: BUStripIFirst, select d
   • Coordinate System: 8307

   

   The data populates in the Live Output Stream table.

   

   As you complete the parameters, notice that the Map section indicates only the vehicles within the Atlanta Geo Fence  – created previously.

   

Create a Query Stage to Ignore Stationary Vehicles

1. In the Add a Stage menu, select Query.
2. In the Create Query dialog box, enter the following values and click Save:
   • Name: BusSpeed0Plus
   • Description: Analytics only on moving vehicles
3. In the BusSpeed0Plus query stage page, do the following:
   • On the Filters tab, click Add a Filter.
   • From the first drop-down list, select Bus_Speed, from the second drop-down list select greater than, and then accept the default value 0.
     
4. On the Sources tab, enter or select the following values:
   • For PatternGeoFenceInside, click Timer in the second row, below Sources.
   • First, select Seconds and then enter a range of 10 seconds and an evaluation frequency of 10 seconds. PatternGeoFenceInside 10s/10s is displayed.

   

5. On the Summaries tab, enter or select the following values:
   • On the right side, click Add a GroupBy.
   • Click the Retain All Columns toggle button.
   • Click Add a Field, and from the drop-down list, select BUStripId.

     

   • Click Add a Summary.
   • From the first drop-down list, select AVG, and from the next drop-down list, select Bus_Speed.

     

6. Let’s do some event shape manipulation by changing the column names associated with the streaming data. Right-click the column name, select Rename, and change BusType_ID  to BusID and AVG_of_Bus_Speed to AvgSpeed.
7. Click the Expression Editor icon on the top-right of the table.
8. Using the intuitive integrated expression builder function, we can add a new live output stream column. This adds a new event type shape to the stream that provides a way to identify the speeding violation levels. In the Expression Builder field, enter NO_VIOLATION.

   

9. Click the check mark to apply the changes. The calc column appears on the left side of the table.

   

10. In the table, rename the calc column to Violation. You can view the Live Output Stream table to see how the data is populating in real time.

Next Tutorial

Creating Business Rules in Your Stream Analytics Pipeline