Oracle by Example brandingCreating Artifacts for Your Stream Analytics Pipeline

section 0Before You Begin

This 20-minute tutorial shows you how to create supporting artifacts for your Stream Analytics pipeline that monitors public transportation in the Atlanta area.

In this series, you learn how to use a Stream Analytics pipeline for real-time traffic analysis, speed violation tracking, and congestion detection. This is the first tutorial in Monitoring Public Transportation Using Stream Analytics. Read the tutorials sequentially.

Access to Stream Analytics

To complete the tutorials in this learning path, you must have access to an installation of Oracle Stream Analytics on-premise or in the cloud as part of Oracle Data Integration Platform Cloud (DIPC), which has Stream Analytics included in it.

Scenario

The scenario is analyzing and monitoring a public transportation system that allows you to identify bottlenecks.

Background

Stream Analytics is a graphical tool with an intuitive web-based interface that enables you to explore, analyze, and manipulate streaming data sources in real time.

This traffic management solution uses GPS fleet data. This data is low cost, accurate, and it's created in real time. Its value for government sector customers is that this fleet data reduces congestion on roadways and enhances the traveling experience. The general features represented in this solution are real-time traffic analytics, speed violation tracking, and congestion detection. These features are combined with GPS streaming sensor feeds and historical trend data using map-based visualizations. This solution uses enterprise-grade Spark Streaming, Kafka open-source messaging, a highly scalable, extensible platform built with Stream Analytics.

Some of the key benefits with this solution are low-cost rollout with zero-road, network disruption, real-time operational intelligence, which is essential for meaningful congestion reduction, and an enhanced traveler experience, with a leading streaming big data technology.

What Do You Need?

  • Java Runtime Environment and Java Development Kit (8 or later)
  • Access to an instance of Oracle Stream Analytics (on-premise or cloud)
  • resourcesfortutorials-4015265.zip
  • Simulated event stream pushed to Kafka
  • Important: Keep the utility running while you work on this tutorial.

section 1Push Simulated Events Stream to Kafka

  1. Extract the resourcesfortutorials-4015265.zip file to a folder on your machine. Notice that the extracted contents include the BusEvents.json and OsacsEventFeeder.jar files.
  2. Open a command prompt.
  3. Go to the location where you extracted the resourcesfortutorials-4015265.zip file.
  4. Push events: 
    java -jar OsacsEventFeeder.jar -z hostname:2181 -t tutorial -j BusEvents.json

    5. Note: The topic name used in this tutorial is tutorial.

    The following response to the command confirms that data is seeding the tutorial topic. This response is mandatory for you to proceed with the tutorial. 

    RESPONSE:{"offsets" : [{"partition" :0, "offset" :28, 
    "error_code" :null, "error" :null}], "key_schema_id" :null, 
    "value_schema_id" :null}

section 2Create a Kafka Connection

  1. In the navigation menu on the left, click Catalog.
  2. In the Create New Item menu, select Connection.
  3. In the Create Connection dialog box, on the Type Properties tab, enter or select the following values and click Next:
  4. On the Connection Details tab, enter the following values:
    • In the Zookeepers field, enter:
      • the Zookeepers URL you saved during provisioning for Oracle Data Integration Platform Cloud
      • the Zookeeper hostname and port (for example: myhost:2181) for Oracle Stream Analytics on-premise
    • When a message appears indicating that the connection is successful, click Save.

section 3Create a Stream

  1. On the navigation pane on the left, click Catalog.
  2. In the Create New Item menu, select Stream.
  3. In the Create Stream dialog box, on the Type Properties tab, enter or select the following values and click Next:
    • Name: Tutorial
    • Tags: tutorial, transportation
    • Stream Type: Kafka
  4. On the Source Details tab, select the following values and click Next:
    • Connection: Tutorial
    • Topic name: Tutorial
  5. On the Shape tab, select the following values and click Next:

section 4Create a Geo Fence

  1. In the Create New Item menu, select Geo Fence.
  2. In the Create Geo Fence dialog box, enter or select the following and click Next:
  3. Click Save. A page with the world map is displayed. This is where you will mark the area around Atlanta.
  4. On top left of the page, click the Polygon tool on the toolbar and do the following:
    • Start marking the area around Atlanta on the map. When you're done, click the Polygon tool again.
    • Name: Atlanta
    • Description: Monitor public buses transport in Atlanta
    • On the left side, click the Save icon to save your changes. The Atlanta Geo Fence looks like this.
      Atlanta region marked with the polygon tool
      Description of the illustration t0_4.png
  5. Click Return to Catalog.

next stepNext Tutorial

Building a Stream Analytics Pipeline