A APIs for Model Pipelines
All the listed APIs/Functions can be utilized by users in Model Pipelines.
Table A-1 Python Paragraph Scripting
| API Type | Functionality | Prerequisite | Notebook Execution Script | Notebook Script Input | Minimum Supported Version |
|---|---|---|---|---|---|
| Set User | Set the user for a session | - |
%python from mmg.constants import * user = "SAUSER" set_user(user) |
user - username to be set | 8.1.1 |
| Get User | Get the user that has been set | Set User |
%python from mmg.constants import * get_user() |
- | 8.1.1 |
| Attach Workspace | Attach the workspace for a session | - |
%python from mmg.workspace import attach_workspace workspace = "CS" attach_workspace(workspace) |
workspace- workspace to be attached | 8.1.1 |
| Get Workspace | Get the workspace that has been attached | Attach Workspace |
%python from mmg.workspace import get_workspace get_workspace() |
8.1.1 | |
| List Workspaces | Lists all workspaces available to a user | Set User |
%python from mmg.workspace import list_workspaces list_workspaces() |
|
8.1.1 |
| Check AIF | Check if AIF is set or not | Set User |
%python from mmg.workspace import check_aif check_aif() |
|
8.1.1 |
| List All Data Sources | List all data sources available in a workspace for a set user. The default workspace chosen is the one that is attached. | Set User, Attach Workspace |
%python from mmg.workspace import list_datasources list_datasources(order=None) |
order - None ensures that all the data sources will be listed for the attached workspace | 8.1.1 |
| List All Data Sources of Workspace | List all Data Sources of a specific Workspace that is not the attached workspace for a set user. | Set User |
%python from mmg.workspace import list_datasources workspace="HELLO NEW" list_datasources(workspace,order=None) |
workspace - Name of workspace for which data sources should be listed order - None ensures that ALL data sources will be listed for that workspace |
8.1.1 |
| Fetch First Order Data Source | This method will fetch the data source related to attached workspace with default order 1. | Set User, Attach Workspace |
%python from mmg.workspace import list_datasources list_datasources() |
order - 1 by default, if not specified |
8.1.1 |
| Fetch First Order Data Source of Workspace | This method will fetch the data source with default order 1 related to specified workspace. | Set User |
%python from mmg.workspace import list_datasources workspace="CS" list_datasources(workspace) |
workspace - Name of workspace for which data source should be listed order- 1 by default if not specified |
8.1.1 |
| Fetch Nth Order Data Source | This method will fetch the data source related to attached workspace with given order | Set User, Attach Workspace |
%python from mmg.workspace import list_datasources list_datasources(order=2) |
order - Order of data source to be fetched | 8.1.1 |
| Fetch Nth Order Data Source of Workspace | This method will fetch the data source with given order related to specified workspace. | Set User |
%python from mmg.workspace import list_datasources workspace="CS" list_datasources(workspace,order=2) |
workspace - Name of workspace for which data source should be listed order = Order of data source to be fetched |
8.1.1 |
| Get Connection Object | Returns Oracle/ SQL Alchemy Connection Object to Data Source of order 1 in attached workspace. |
Set User, Attach Workspace
|
%python import cx_Oracle from mmg.workspace import get_connection conn=get_connection() if not isinstance(conn,cx_Oracle.connect): print("Not Connection Object") print(conn) |
8.1.2.0 | |
| Get Connection Object To Specific Data Source | Returns Oracle/ SQL Alchemy Connection Object to specified Data Source | Set User |
%python import cx_Oracle from mmg.workspace import get_connection datasource="DS001" conn=get_connection(datasource) if not isinstance(conn,cx_Oracle.connect): print("Not Conn Object") print(conn) |
datasource - Name of datasource for which the connection object has to be returned | 8.1.2.0 |
| Get Connection Object To Data Source Using Order | Returns Oracle/ SQL Alchemy Connection Object to Data Source of specified order in attached workspace. | Set User, Attach Workspace |
%python import cx_Oracle from mmg.workspace import get_connection datasource=2 conn=get_connection (datasource) if not isinstance(conn,cx_Oracle.connect): print("Not Conn Object") print(conn) |
datasource - Order number of datasource in attached workspace for which the connection object has to be returned | 8.1.2.0 |
| Test Connection Object | This script can be used to test the connection object to data source |
Get Connection Object OR Get Connection Object To Specific Data Source OR Get Connection Object To Data Source Using Order |
%python query="SELECT table_name FROM user_tables" cur = conn.cursor() cur.execute(query) print(cur.fetchall()) if cur: cur.close() |
8.1.2.0 |
Note:
The property RESTRICT_UNMAPPED_DATASTORES_ACCESS has been introduced in the NEXTGENEMF_CONFIG table to control datastore access within notebook sessions. By default, this property is set to false, which means access to all datastores is allowed. If a user wants to restrict access so that only the datastores attached to the current workspace are accessible, the value of this property must be set to true. When enabled, the get_conn() OR get_connection() method will only allow connections to those datastores that are mapped to the same workspace as the notebook session. This restriction is effective only when the session-mode in the application.yml file is configured as NOTEBOOK_USER. The property accepts only two values: true or false.Table A-2 PYTHON LINEPARSER APIs (for MMG version 8.1.0)
| API Type | Functionality | Notebook Execution Script | Note |
|---|---|---|---|
| Attach Workspace | Attach the workspace for a session |
%python mmg.attachWorkspace(workspace) #eg- mmg.attachWorkspace("CS") |
workspace - workspace to be attached (String) Sets 'mmg' python variable which contains the information about the attached workspace. Output- Print a boolean in one line on the status of workspace attachment and another line of boolean on status of AIF enabled and attached. |
| List Workspaces | Lists all workspaces available to a user |
%python mmg.listWorkspaces() |
Sets 'ofs_wsList' python variable which is the list of all available workspaces. Output- Print the list of workspaces. |
| Check AIF | Check if AIF is set or not |
%python print(aif__isAttached) |
Output- Boolean on the status of AIF. |
| Get Connection Object | Returns cx_Oracle Connection Object |
%python mmg.getConnection(schema_name) #eg- mmg.getConnection("OFSAA") |
schema_name = name of the schema for which connection is required (String). Sets 'conn' python variable which is the cx_Oracle connection object to be used for firing queries. |
| Publish | Fetch and sets the Notebook JSON dump |
%python mmg.publish() |
Sets 'emf_para' python variable which is the JSON dump for the notebook component. Output- Print the set 'emf_para' value. |
| Register | Register the Notebook |
%python mmg.register(register) #eg- mmg.register({"modelname":"model1",\ # "modeldescription":"Model description"}) #eg- mmg.register(emf_para) {After doing mmg.publish()} |
register - JSON Stringify object for RegisterNotebookBean Object. Sets 'publishedNotebookId' python variable which is the studio notebook id for the published version. Output- Prints the published notebook id. |
| Load Flat File | - |
%python mmg.loadFlatFile() #eg- mmg.loadFlatFile() Incomplete |
|
| Profile Data | - |
%python mmg.profileData(code,title) #eg- mmg.profileData(ABC,XYZ) |
Table A-3 MMG Library Python APIs
| Index | API Type | Functionality | Prerequisite | Notebook Execution Script |
|---|---|---|---|---|
| 1 | Set User | Set the user for a session | %python from mmg.constants import * user =
"SAUSER" set_user(user) |
|
| 2 | Get User | Get the user that has been set |
Set User |
%python from mmg.constants import *
get_user() |
| 3 | Attach Workspace | Attach the workspace for a session | %python from mmg.workspace import
attach_workspace workspace = "CS"
attach_workspace(workspace)
|
|
| 4 | Get Workspace | Get the workspace that has been attached |
Attach Workspace |
%python from mmg.workspace import
get_workspace get_workspace()
|
| 5 | Get Base URL | Get MMG Studio Service Base URL | %python from mmg.constants import *
get_mmg_studio_service_url()
|
|
| 6 | List Workspaces | Lists all workspaces available to a user |
Set User |
%python from mmg.workspace import
list_workspaces list_workspaces() |
| 7 | Check AIF | Check if AIF is set or not |
Set User |
%python from mmg.workspace import check_aif
check_aif() |
| 8 | List All Data Sources | List all data sources available in a workspace for a set user. The default workspace chosen is the one that is attached. |
Set User, Attach Workspace |
%python from mmg.workspace import
list_datasources list_datasources(order=None) |
| 9 | List All Data Sources of Workspace | List All Data Sources of a specific Workspace that is not the attached workspace for a set user |
Set User |
%python from mmg.workspace import
list_datasources workspace="HELLO NEW"
list_datasources(workspace,order=None) |
| 10 | Fetch First Order Data Source | This method will fetch the data source related to attached workspace with default order 1 | Set User, Attach Workspace | %python from mmg.workspace import
list_datasources list_datasources()
|
| 11 | Fetch First Order Data Source of Workspace | This method will fetch the data source with default order 1 related to specified workspace | Set User | %python from mmg.workspace import
list_datasources workspace="CS"
list_datasources(workspace) |
| 12 | Fetch Nth Order Data Source | This method will fetch the data source related to attached workspace with given order | Set User, Attach Workspace | %python from mmg.workspace import
list_datasources list_datasources(order=2) |
| 13 | Fetch Nth Order Data Source of Workspace | This method will fetch the data source with given order related to specified workspace | Set User | %python from mmg.workspace import
list_datasources workspace="CS"
list_datasources(workspace,order=2)
|
| 14 | Get Connection Object | Returns Oracle/ SQL Alchemy Connection Object to Data Source of order 1 in attached workspace | Set User, Attach Workspace | %python import cx_Oracle from mmg.workspace
import get_connection conn=get_connection() if not
isinstance(conn,cx_Oracle.connect): print("Not Conn Object")
print(conn) |
| 15 | Get Connection Object To Specific Data Source | Returns Oracle/ SQL Alchemy Connection Object to specified Data Source | Set User | %python import cx_Oracle from mmg.workspace
import get_connection datasource="DS"
conn=get_connection(datasource, conn_type=None) if not
isinstance(conn,cx_Oracle.connect): print("Not Conn Object")
print(conn) |
| 16 | Get Connection Object To Data Source Using Order | Returns Oracle/ SQL Alchemy Connection Object to Data Source of specified order in attached workspace | Set User, Attach Workspace | %python import cx_Oracle from mmg.workspace
import get_connection datasource=2
conn=get_connection(datasource, conn_type=None) if not
isinstance(conn,cx_Oracle.connect): print("Not Conn Object")
print(conn) |
| 17 | Get Connection Object | Returns Oracle/ SQL Alchemy Connection Object to Data Source of order 1 in attached workspace | Set User, Attach Workspace | %python import oracledb from mmg.workspace import
get_conn conn=get_conn() if not
isinstance(conn,oracledb.Connection): print("Not Conn Object")
print(conn) |
| 18 | Get Connection Object To Specific Data Source | Returns Oracle/ SQL Alchemy Connection Object to specified Data Source | Set User | %python import oracledb from mmg.workspace import
get_conn datasource="DS" conn=get_conn(datasource,
conn_type=None) if not isinstance(conn,oracledb.Connection):
print("Not Conn Object") print(conn) |
| 19 | Get Connection Object To Data Source Using Order | Returns Oracle/ SQL Alchemy Connection Object to Data Source of specified order in attached workspace | Set User, Attach Workspace | %python import oracledb from mmg.workspace import
get_conn datasource=2 conn=get_conn(datasource, conn_type=None)
if not isinstance(conn,oracledb.Connection): print("Not Conn
Object") print(conn) |
| 20 | Get Objective Hierarchy | Returns the complete objective hierarchy including pipeline name | %python print(objectiveId) |
|
| 21 | Fetch FTPSHARE Path | Get complete ftpshare path on the server | %python from mmg.datasets import fetch_ftpshare
fetch_ftpshare() |
|
| 22 | Get Pipeline Name | Get the Pipeline Name to which the current notebook is attached | Attach Workspace | %python from mmg.constants import get_pipeline_name
get_pipeline_name() |
| 23 | Get Pipeline ID | Get ID of Pipeline to which the current notebook is attached | Attach Workspace | %python from mmg.constants import get_pipeline_id
get_pipeline_id() |
| 24 | Get Pipeline Version | Get version of Pipeline to which the current notebook is attached | Attach Workspace | %python from mmg.constants import get_pipeline_version
get_pipeline_version() |
| 25 | Get MISDATE | Return MISDATE from pipeline execution if set otherwise takes default value provided | Attach Workspace | %python
misdate=ds.date_picker(name='$FICMISDATE$',format='yyyy-MM-dd',
default_value='2000-01-01', label='FICMISDATE')
print(misdate) |
| 26 | Get BATCHRUNID | Returns BATCHRUNID if set from pipeline, otherwise takes default value | %python
batchrunid=ds.textbox(name='$BATCHRUNID$',default_value='Batch_auto',label='BATCHRUNID')
print(batchrunid) |
|
| 27 | Get TASKID | Returns TASKID if set from pipeline, otherwise takes default value | %python
taskid=ds.textbox(name='$TASKID$',default_value='task1',label='TASKID')
print(taskid) |
Miscellaneous Helper Python Scripts
Table A-4 Miscellaneous Helper Python Scripts
| Index | Name | Description | Prerequisite | Script |
|---|---|---|---|---|
| 1 | Create Connection Object Using Wallet Alias | This script can be used to test the connection to any Oracle database using wallet alias. Make sure to close connection object after use. | %python wallet_alias = "WALLET_ALIAS_NAME" import
oracledb oracledb.init_oracle_client() conn =
oracledb.connect(dsn=wallet_alias |
|
| 2 | Test Connection Object | This script can be used to test the connection object to data source | Get Connection Object OR Get Connection Object To Specific Data Source OR Get Connection Object To Data Source Using Order | %python query="SELECT table_name FROM user_tables" cur =
conn.cursor() cur.execute(query) print(cur.fetchall()) if cur:
cur.close() |
| 3 | Close Connection Object | Connection Object should be created using any of the above methods | %python query="SELECT table_name FROM user_tables" cur =
conn.cursor() cur.execute(query) print(cur.fetchall()) if cur:
cur.close() |
|
| 4 | Close Cursor | Cursor Object should be created using any of the above methods | %python if cur: cur.close() |
|
| 5 | Check environment variables |
For example: Check wallet alias value of MMG_LIB_WALLET_ALIAS |
%python import os
print(os.environ['MMG_LIB_WALLET_ALIAS']) |
|
| 6 | Save Dataframe to a File | Save dataframe to a file stored on server | %python from mmg.datasets import fetch_ftpshare,
save_dataframe from mmg.workspace import get_workspace from
mmg.constants import get_pipeline_version
misdate=ds.date_picker(name='$FICMISDATE$', format='yyyy-MM-dd',
default_value='2000-01-01', label='FICMISDATE')
batchrunid=ds.textbox(name='$BATCHRUNID$',
default_value='Batch_auto', label='BATCHRUNID')
taskid=ds.textbox(name='$TASKID$', default_value='task1',
label='TASKID') filename="df.csv" filepath=fetch_ftpshare() +
"/mmg/" + get_workspace() + "/pipelines/" + objectiveId + "/" +
get_pipeline_version() + "/output/" + misdate + "/" + batchrunid
+ "/" + taskid + "/" + filename res=save_dataframe(df,filepath)
message=res["messages"] if res["status"]=="ERROR": raise
Exception(message) else: print(message) |
Python Linepraser APIs (for MMG version 8.1.0)
Table A-5 Python Linepraser APIs (for MMG version 8.1.0)
| Index | API Type | Functionality | Notebook Execution Script | Note |
|---|---|---|---|---|
| 1 | Attach Workspace | Attach the workspace for a session | %python mmg.attachWorkspace(workspace) #eg-
mmg.attachWorkspace("CS") |
workspace = workspace to be attached (String) Sets 'mmg' python variable which contains the information about the attached workspace Output- Print a boolean in one line on the status of workspace attachment and another line of boolean on status of AIFenabled and attached |
| 2 | List Workspaces | Lists all workspaces available to a user | %python mmg.listWorkspaces() |
Sets 'ofs_wsList' python variable which is the list of all available workspaces Output- Print the list of workspaces |
| 3 | Check AIF | Check if AIF is set or not | %python print(aif__isAttached) |
Output- Boolean on the status of AIF |
| 4 | Get Connection Object | Returns cx_Oracle Connection Object | %python mmg.getConnection(schema_name) #eg-
mmg.getConnection("OFSAA") |
schema_name = Name of the schema for which connection is required (String) Sets 'conn' python variable which is the cx_Oracle connection object to be used for firing queries |
| 5 | Publish | Fetch and sets the Notebook JSON dump | %python mmg.publish() |
Sets 'emf_para' python variable which is the JSON dump for the notebook component Output- Print the set 'emf_para' value |
| 6 | Register | Register the Notebook | %python mmg.register(register) #eg-
mmg.register({"modelname":"model1",\ # "modeldescription":"Model
description"}) #eg- mmg.register(emf_para) {After doing
mmg.publish()} |
register = JSON Stringify object for RegisterNotebookBean Object Sets 'publishedNotebookId' python variable which is the studio notebook id for the published version. Output- Prints the published notebook ID |
| 7 | Load Flat File | %python mmg.loadFlatFile() #eg- mmg.loadFlatFile()
Incomplete |
||
| 8 | Profile Data | %python mmg.profileData(code,title) #eg-
mmg.profileData(ABC,XYZ) |
code = title = |