18.6 Create Data Entities

Data entities represent the metadata of the source or target data, such as column/field name, data type, scale, etc. The data entitities can be created programmatically through the APIs or using the utility provided to discover the data entites and generate the Python code for the same.

This example shows a script that creates data entities in Data Transforms (less recommended):

from datatransforms.workbench import DataTransformsWorkbench,WorkbenchConfig
from datatransforms.dataentity import DataEntity
 
##Load active workbench from configuration
pswd="<your deployment pswd from secret store>"
connect_params = WorkbenchConfig.get_workbench_config(pswd)
workbench = DataTransformsWorkbench()
workbench.connect_workbench(connect_params)
 
channels_entity = DataEntity().from_connection("Demo Source Data","SH").entity_name("CHANNELS")
channels_entity.add_column(name="CHANNEL_ID",position=1,dataType="NUMBER",dataTypeCode="NUMBER",length=0,scale=-127)
channels_entity.add_column(name="CHANNEL_DESC",position=2,dataType="VARCHAR2",dataTypeCode="VARCHAR2",length=20,scale=0)
channels_entity.add_column(name="CHANNEL_CLASS",position=3,dataType="VARCHAR2",dataTypeCode="VARCHAR2",length=20,scale=0)
channels_entity.add_column(name="CHANNEL_CLASS_ID",position=4,dataType="NUMBER",dataTypeCode="NUMBER",length=0,scale=-127)
channels_entity.add_column(name="CHANNEL_TOTAL",position=5,dataType="VARCHAR2",dataTypeCode="VARCHAR2",length=13,scale=0)
channels_entity.add_column(name="CHANNEL_TOTAL_ID",position=6,dataType="NUMBER",dataTypeCode="NUMBER",length=0,scale=-127)
workbench.save_data_entity(channels_entity)

Using the utility to create the Python File (Preferred)

% python -m datatransforms.cli.generate_data_entities.py --connection "Demo Source Data" --schema SH --live true
 
Oracle DataTransforms
 
Generating data entities using options ...
Namespace(log_level=None, connection='Demo Source Data', schema='SH', live=True, matching=None)
 
 
INFO:root:Loading active work bench
INFO:root:Loading active deployment configuration
Data Transforms URL http://xxx.xxx.xxx.xx:9999/odi-rest
INFO:root:Data Entities will be generated in script file demosourcedata_sh_entities.py
INFO:root:Fetching data entities using live connection
INFO:root:***No matching filter applied, 9 tables will be used to generate data entities***
DataEntity Generation Complete
%

Note:

When –live is true the utility will go to the database to fetch the data entities. If it is false, it will fetch them from the Data Transforms Repository.

Notice that in the work area directory there is a new file called demosourcedata_sh_entities.py. View this file to see the Create Data Entity commands for all tables in the SH schema. Move this file to your project folder.

% mv demosourcedata_sh_entities.py MyProject
%

From your Work Folder, you can run this Python file to create the Data Entities.

% cd /Users/someuser/DT4D_Projects
% python3 MyProject/demosourcedata_sh_entities.py
Demo Source Data.SH.CHANNELS
Demo Source Data.SH.COSTS
Demo Source Data.SH.COUNTRIES
Demo Source Data.SH.CUSTOMERS
Demo Source Data.SH.PRODUCTS
Demo Source Data.SH.PROMOTIONS
Demo Source Data.SH.SALES
Demo Source Data.SH.SUPPLEMENTARY_DEMOGRAPHICS
Demo Source Data.SH.TIMES
%

If you like, you can connect to Data Transforms, navigate to the Data Entities tab, select "Demo Source Data"
in the  connect filter to see the Data Entities