Create the Automation

Create and deploy the application and automation function.

Create the Oracle Functions Application

An application is logical grouping of functions that allows you to allocate and configure resources for the entire group and to isolate the functions during runtime.

Before you can create the application, you must set up your tenancy for function development and you must set up your function development environment.

When you define an application in Oracle Functions, you specify the subnets in which to run the functions in the application. You also specify whether to enable logging for the functions in the application.

Log in to the Console as a functions developer.
In the Console, open the navigation menu. Under Solutions and Platform, go to Developer Services and click Functions.
Select the region you are using with Oracle Functions. Oracle recommends that you use the same region as the Docker registry that's specified in the Fn Project CLI context.
Select the compartment specified in the Fn Project CLI context. The Applications page shows the applications already defined in the compartment.
Click Create Application and specify:
- Name: A name for the new application. Avoid entering confidential information.
- VCN and subnet: The VCN and subnet in which to run functions.
Click Create.

The new application appears in the list of applications.

Create and Deploy the Automation Function

Create, deploy, and configure the function loadziptojson in the application ocijsonpub.

Initialize the function:
```
 fn init --runtime python loadziptojson
```
A directory is created with the function name you specified, containing func.py, func.yaml, and requirements.txt files. The func.yaml file contains the minimum amount of information required to build and run the function.

Open func.py and replace the contents with the following code blocks:

Import the necessary Python modules:

import io
import json
import oci
import csv
import requests
import traceback
from io import StringIO
from io import BytesIO
 
from zipfile import ZipFile, is_zipfile
from fdk import responsecode

Define a function handler to unzip the file and insert into JSON database when the file is uploaded to object storage:

def handler(ctx, data: io.BytesIO=None):
    signer = oci.auth.signers.get_resource_principals_signer()
 
    object_name = bucket_name = namespace = ordsbaseurl = schema = dbuser = dbpwd = ""
    try:
        cfg = ctx.Config()
        input_bucket = cfg["inputbucket"]
        processed_bucket = cfg["processedbucket"]
        ordsbaseurl = cfg["ordsbaseurl"]
        schema = cfg["dbschema"]
        dbuser = cfg["dbuser"]
        dbpwd = cfg["dbpwd"]
    except Exception as e:
        print('Missing function parameters: bucket_name, ordsbaseurl, schema, dbuser, dbpwd', flush=True)
        raise
    try:
        body = json.loads(data.getvalue())
        object_name = body["data"]["resourceName"]
        if body["data"]["additionalDetails"]["bucketName"] != input_bucket:
            raise ValueError("Event Bucket name error")
        namespace = body["data"]["additionalDetails"]["namespace"]
    except Exception as e:
        print('ERROR: bad Event!', flush=True)
        raise
    try:   
        insert_status = load_data(signer, namespace, input_bucket, object_name, ordsbaseurl, schema, dbuser, dbpwd)
        #move_object(signer, namespace, input_bucket, processed_bucket, object_name)
 
        return response.Response(
           ctx,
           response_data=json.dumps(insert_status),
           headers={"Content-Type": "application/json"}
        )
    except Exception as e:
        return response.Response(
            ctx, response_data=json.dumps([{'Error': traceback.format_exc()}
                                          , ]),
            headers={"Content-Type": "application/json"}
        )

Define a data loading function which the handler function calls to unzip the file, read the csv file that is in the zip file, and loops through line by line to call the soda_insert function.

def load_data(signer, namespace, bucket_name, object_name, ordsbaseurl, schema, dbuser, dbpwd):
    client = oci.object_storage.ObjectStorageClient(config={}, signer=signer)
    zipfile = client.get_object(namespace, bucket_name, object_name)
    insert_status = {}
 
    if zipfile.status == 200:
       print("INFO - Object {0} is read".format(object_name), flush=True)
       #object_storage_client.get_object
 
       with ZipFile(BytesIO(zipfile.data.content)) as zippy:
          for item in zippy.infolist():
             print(item.filename, flush=True)
             try:
                 if not 'MACOS' in item.filename and 'csv' in item.filename:
                    csvdata = csv.DictReader(zippy.read(item.filename).decode('utf-8').split('\n'),delimiter=',')
                    for row in csvdata:
                       insert_status = soda_insert(ordsbaseurl, schema, dbuser, dbpwd, dict(row))
             except Exception:
                 print('trouble decoding the file ', flush=True)
                 continue
  
        
    else:
         raise SystemExit("cannot retrieve the object" + str(object_name))
    return insert_status

Define an insert function that creates the collection and inserts data into the collection.

def soda_insert(ordsbaseurl, schema, dbuser, dbpwd, document):
   
    auth=(dbuser, dbpwd)
    sodaurl = ordsbaseurl + schema + '/soda/latest/'
    collectionurl = sodaurl + "censusdata"
 
    headers = {'Content-Type': 'application/json'}
    print("INFO - calling SODA with requests:", flush=True)
    print("INFO - before calling SODA :dbuser : "+dbuser, flush=True)
    print("INFO - before calling SODA :dbpwd : "+dbpwd, flush=True)
    reqjson={}
    r = requests.put(collectionurl, auth=auth, headers=headers, data=reqjson)
    print("INFO - before calling SODA :document : "+json.dumps(document), flush=True)
    i = requests.post(collectionurl, auth=auth, headers=headers, data=json.dumps(document))
 
    r_json = {}
    r_json = json.dumps(r.text)
    return r_json

Open func.yaml and replace the contents with the following:

schema_version: 20180708
name: loadziptojson
version: 0.0.31
runtime: python
entrypoint: /python/bin/fdk /function/func.py handler
memory: 256

Open requirements.txt and replace the contents with the following:
```
fdk
oci
requests
```
Enter the following single Fn Project command to build the function and its dependencies as a Docker image, push the image to the specified Docker registry, and deploy the function to Oracle Functions in the specified application:
```
fn -v deploy --app ocijsonpub
```

Configure function parameters using the following format:

$ fn config function <application_name> <function_name> --<property> <value>

For example, configure the function ocijsonpub using the following statements, substituting the required values where indicated:

#Autonomous Database schema
fn config function ocijsonpub loadziptojson dbschema <db_schema>
#Autonomous Database Username
fn config function ocijsonpub loadziptojson dbuser <db_user_name>
#Autonomous Database password
fn config function ocijsonpub loadziptojson dbpwd <db_password>
# Object storage bucket for input file
fn config function ocijsonpub loadziptojson inputbucket <input_files_bucket>
# Object storage bucket to archive the zip file after processing
fn config function ocijsonpub loadziptojson processedbucket <processed_files_bucket>