Create a file stream pipeline

Learn to create a simple pipeline in Stream Analytics using a file, such as CSV or JSON, for quick prototyping, testing, or proofs of concept use cases.

Task 1: Create the Stream Analytics deployment

To create a Stream Analytics deployment:
  1. In the Console navigation menu, click Oracle Database, and then select GoldenGate.
  2. On the Deployments page, click Create deployment.
  3. In the Create deployment panel, enter a name and optionally, a description.
  4. From the Compartment dropdown, select a compartment in which to create the deployment.

    Note:

    A deployment can only access the connections within the same tenancy on the same VCN. You must also have explicit access to the connection.

  5. For OCPU Count enter the number of Oracle Compute units (OCPUs) to use.

    Note:

    One OCPU is equivalent to 16GB of memory. 3 to 4 OCPUs is sufficient for one Stream analytics pipeline. For more information, see OCPU management and billing.
  6. (Optional) Select Auto Scaling.

    Note:

    Auto scaling enables OCI GoldenGate to scale up to three times the number of OCPUs you specify for OCPU Count, up to 24 OCPUs. For example, if you specify your OCPU Count as 2 and enable Auto Scaling, then your deployment can scale up to 6 OCPUs. If you specify your OCPU Count as 20 and enable Auto Scaling, OCI GoldenGate can only scale up to 24 OCPUs.
  7. From the Subnet in <Compartment> dropdown, select the subnet to which a private endpoint is created from the OCI GoldenGate service tenancy. This ensures that the deployment is always available over this subnet, as long as the policies for this subnet allow access. The private endpoint is only used to access the deployment console, and doesn't provide access to other resources in the subnet.

    To select a subnet in a different compartment, click Change compartment.

  8. Select a license type.
  9. (Optional) Click Show advanced options for network options and to add tags.
    1. In the Network tab,
      1. Select Enable GoldenGate console public access if you want to include a public endpoint in addition to a private endpoint, and allow public access to the deployment console for users.
      2. Select Customize endpoint to provide a private fully qualified domain name (FQDN) prefix that you'll use to access the private service console URL. You can also optionally upload an SSL/TLS certificate (.pem) and its corresponding private key, however, password protected certificates are not supported.It is your responsibility to ensure the FQDN is resolvable in the subnet you have previously selected.

        A self-signed certificate is generated for you, if you don't provide one.

        Note:

        It is your responsibility to ensure the FQDN is resolvable in the subnet you have previously selected.
    2. In the Maintenance tab:
      1. Select Customize maintenance window to define the start of the maintenance window to upgrade the deployment.
      2. (Optional) For Major release auto-upgrade period in days, enter the number of days, between 0 and 365.
      3. (Optional) For Bundle release auto-upgrade period in days, enter the number of days, between 0 and 180 days.
      4. (Optional) For Security patch auto-upgrade period in days, enter the number of days, between 0 and 14 days.
      5. Select Enable interim release auto-upgrade, and, optionally, enter the number of days.

      Note:

      Learn more about scheduling upgrades.
    3. In the Tags tab, add tags to help track the resources within your tenancy. Click + Additional tag to add more tags. Learn more about tagging.
  10. Click Next.
  11. Select Stream analytics for deployment type.
  12. The Stream analytics technology type is automatically selected for you.
  13. For GoldenGate instance name, enter a name for the stream analytics instance.
  14. For Administrator username, enter osaadmin.
  15. For Administrator password, enter a password, and then confirm that password.
  16. Click Create.

Task 2: Create and publish the pipeline

  1. Launch the Stream Analytics pipeline.
    1. From the Stream Analytics deployment details page, click Launch console.
    2. Log in to the Stream Analytics deployment console using the Administrator username and password specified when you created the deployment in Task 1, steps 14 and 15.
  2. In the Stream Analytics deployment console, click Catalog.
  3. Create a File Stream.
  4. Create a pipeline using the File stream created in Step 3.
    Learn more about the Pipeline Editor. Here are some actions you can perform on your File stream pipeline:
  5. Publish the pipeline.