Manage Tasks

Tasks are procedures that perform analytics or system administration. For example, analytic tasks might include aggregating data and system administrative tasks might include importing and exporting data. Tasks can be processed manually (ad hoc) or included within a run’s flow.

The Manage Tasks page gives you the ability to view information associated to current and historic tasks. In addition, you can create and run new tasks. As an example, this page could be used as a starting point when you know the tasks names, while you are building a run, and if you want to see all the tasks of a specific type that are currently active. It is also common that you may reference this page through the Activity Dashboard and the Manage Runs page.

On this page:

Fields and Buttons

The top section of the Manage Tasks page allows you to search for tasks that match criteria in the fields.

  • Task ID: Task system-assigned, unique identifier
  • Task Type Code: Search for tasks that are a specific task type code.
  • Task Name: Name that was given to the task when it was added.
  • Task Lookup: Secondary task description assigned when the task was added.
  • Status: Search for tasks that are a specific status.
  • Get Tasks: Search for tasks matching the criteria set in the top section’s fields.
  • Add Task: Opens the Create Tasks dialog box where you may build a new task.

Back to Top

Data Table

The Data Table columns include:

  • ID: Task system-assigned, unique identifier.
  • Task Type Code: Task type assigned to the run when it was created.
  • Task Name: Task name as designated when the task was added.
  • Task Lookup: Secondary task description that was assigned when the task was added.
  • Status: Task status (Active or Inactive).
  • Create Time: Date and time when the task was created.
  • Update Time: Date and time when the task was updated.
  • File Log: Navigates to the Manage File Logs page and displays the current context to provide further information.

    Note: This link only appears when a log exists.

  • File Log: Navigates to the Manage File Logs page and displays the current context to provide further information.

    Note: This link only appears when a log exists.

  • XML Editor: Displays the Tasks XML Editor. This link only appears for ETL tasks since they have XML embedded in the YAML.
  • View: Opens the View Task dialog box, which displays the same data as provided in the data table row as well as task description and locked flag information.
  • Edit: Opens the Manage Tasks dialog box, which provides the same fields as the View Tasks dialog box and allows you to edit the Locked Flag and Status.
  • YAML: Opens the Task Definition dialog box, which allows you to view and edit the associated YAML.
  • Task Log: Navigates to the Task Log page and displays the current context to provide further information.

Back to Top

Adding Tasks

To add a task:

  1. Click Add Task on the Manage Tasks page.
  2. Enter the appropriate type in the Task Type Code field of the Create Tasks dialog box:
    • Dataset Export (DSEXP)
    • ETL on core schema (ETL)
    • Fact audit (FACT_AUDIT)
    • Geocode data (GEO)
    • Point Aggregation (PAGG)
    • Relation aggregation or derivation (RAGG)
    • RAN maintenance (RANMAINT)
    • Scenario to run calc (SCENARIO)
    • Time Aggregation (TAGG)
  3. Enter a descriptive name in the Task Name field.
  4. Enter a value for task lookup in the Task Lookup field. The task lookup value can be any unique alphanumeric value without spaces. A common practice is to use the task name in uppercase without spaces.
  5. Enter a short description of the task function in the Task Desc field.
  6. Select whether the task is Active or Inactive from the Status drop-down list. Only active tasks may be processed.
  7. Enter the template YAML for the task type in the Task Definition field.

Back to Top

Viewing Task Information

The View Tasks dialog box is accessed from the View link, which is found in the Manage Tasks page’s data table. This feature offers additional information to what is displayed in the Manage Tasks data table.

Fields

  • ID: Task system assigned, unique identifier.
  • Task Type Code: Type code type that was assigned to the task when it was added.
  • Task Name: Name that was assigned to the task when it was added.
  • Task Lookup: Secondary description that was assigned when the task was added.
  • Task Desc: Additional information that describes the task’s function.
  • Locked Flag: Task lock flag value (Locked or Unlocked).
  • Status: Task status (Active or Inactive).
  • Create Time: Date and time when the task was created.
  • Update Time: Date and time when the task was updated.

Links and Buttons

  • Task Definition: Contains the YAML link, which opens the Task Definition dialog box.
  • Edit: Opens the Manage Tasks dialog box.
  • Cancel: Closes the dialog box.
  • Execute Now: Opens the Execute Tasks Details dialog box.

Back to Top

Managing Task Information

The Manage Tasks dialog box opens when you click the Edit link on the task data table row or the Edit button on the View Tasks dialog box.

  • ID: Task system assigned, unique identifier.
  • Task Type Code: Type code type that was assigned to the task when it was added.
  • Task Name: Name that was assigned to the task when it was added.
  • Task Lookup: Secondary task description assigned when the task was added.
  • Task Desc: Additional information that describes the task’s function.
  • Locked Flag: Set the task's lock flag (Locked or Unlocked).
  • Status: Set the task's status (Active or Inactive).
  • Create Time: Date and time when the task was created.

Back to Top

Viewing Task Definition YAML

The Task Definition dialog box displays the YAML code that defines the task. From this dialog box, you may view the parameters defined in the YAML or edit the content by clicking Edit.

Back to Top

Editing Task YAML

The YAML that defines a task is edited from the Task Definition dialog box.

To make task definition YAML editable:

  1. Click the YAML link in either the Task Data Table or the View Tasks dialog box.
  2. Click Edit in the Task Definition dialog box .

Note: Each task type has unique requirements that are specified in its YAML. You define your task by updating generic YAML with task parameters. All tasks begin with:

Copy 
---
task:
# <name or descriptor> task definition
# -------------------------------------------------------
name: task_<task name>
module_name: dr.task.<task-code> #.py is always assumed by python
task_method_name: startTask
post_task_method_name: postTask

Back to Top

YAML Task Templates

With the exception of the ETL task, all tasks require that you add and modify template YAML to the Task Definition. The templates provide comments (beginning with #) that provide usage information.

For example, if the requirement is to add a file format, the usage instructions would be:

Copy 
# CSV, JSON, XML

Back to Top

DSEXP: Dataset Export

Exports dataset values in the format specified.

Copy 
---
    task:
    # Dataset export task definition
    # -------------------------------------------------------       
    name: task_dsexp
    module_name: dr.task.dsExp #.py is always assumed by python
    task_method_name: startTask
    post_task_method_name: postTask
    
    debug_flag: False
    header_flag: False
    delimiter: "|"
    quoting:  none
    footer_flag: False
    export_format: CSV # Formats: CSV, JSON, XML
    point_format: name # Formats: ID, lookup, name, verbose
    target_name: XXX
    target_date_format: ".%Y%m%d."
    target_extension: txt
    target_directory: /XXX/XXX
    target: XXX
    #blacklistedFields: ['point', 'startDate']

Back to Top

ETL

The ETL task contains YAML with embedded XML. Some XML data may only be added in the Task Description, while others may be added and modified with the Tasks XML Editor.

Copy 
---
    task:
    # etl task definition
    # -------------------------------------------------------
    name: etl
    module_name: dr.task.etl #.py is always assumed by python
    task_method_name: startTask
    post_task_method_name: postTask
    
    XML: 
    <etl xmlns="http://dataraker.net/etl.xsd"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://dataraker.net/etl.xsd">
    <source desc="XXX" name="mr" xsi:type="source_type_file"
    file_archive_flag="true" file_compress_flag="false">
    <source_name></source_name>
    <conn_name></conn_name>
    <file_path>/XXX/XXX/</file_path>
    <archive_path>/XXX/XXX/archive</archive_path>
    <file_mask>XXX.XXX.[%Y.%m.%d].*.XXX</file_mask> 
    </source>
    <formats>
    <format xsi:type="format_type_fixed_width"
    name="XXXX_fixed_width" desc="XXX" line_terminator="\n"
    min_row_length="154" max_row_length="1000">
    <format_name></format_name>
    <format_options></format_options>
    <fields xsi:type="field_type" length="XXX" start_pos="1"
    name="PTC" data_type="string" date_format=""></fields>
    <fields xsi:type="field_type" length="XXX" start_pos="1"
    name="PARENT_PTC" data_type="string" date_format=""></fields>
    <fields xsi:type="field_type" length="XXX" start_pos="1"
    name="VAL" data_type="string" date_format=""></fields>
    </format>
    </formats>
    <etl_options>
    <option name="read_batch_size" value="10000"/>
    <option name="point_batch_size" value="1000"/>
    <option name="target_batch_size" value="10000"/>
    <option name="log_all_errors" value="true"/>
    <option name="debug_table" value="false"/>
    <option name="debug_file" value="false"/>
    <option name="auto_trim" value="true"/>
    <option name="max_error_count" value="0"/>
    </etl_options>
    <global_variables></global_variables>
    <variables>  
    <variable is_active="true" name="point_lookup">
    <value>XXXX</value>
    </variable>
    </variables>
    <points>
    <point duplicates="last" is_active="true" name="PTC"
    xsi:type="point_lookup_type_ins">
    <point_type_code></point_type_code>
    <point_lookup></point_lookup>
    <point_name></point_name>
    <point_tag></point_tag>
    <point_utc_offset></point_utc_offset>
    <point_desc></point_desc>
    </point>
    <point duplicates="last" is_active="true" name="PARENT_PTC"
    xsi:type="point_lookup_type_ins">
    <point_type_code></point_type_code>
    <point_lookup></point_lookup>
    <point_name></point_name>
    <point_tag></point_tag>
    <point_utc_offset></point_utc_offset>
    <point_desc></point_desc>
    </point>
    </points>
    <variables_pt>
    </variables_pt>
    <targets>
    <target xsi:type="target_type_metric" name="XXX"
    duplicates="max" is_active="true">
    <fact_attr>
    <attr name="fact_lookup" value="XXX"/>
    <attr name="fact_name" value="XXX"/>
    <attr name="fact_category" value="XXX"/>
    <attr name="fact_sequence" value="0"/>
    <attr name="time_basis" value="XXX"/>
    <attr name="agg_method" value="n/a"/>
    <attr name="infer_delete_flag" value="false"/>          
    </fact_attr>
    <point>P_PTC</point>
    <start_time>XXX</start_time>
    <filters>
    <filter>P_PTC</filter>            
    </filters>
    <value>F_VAL</value>
    </target>
    <target xsi:type="target_type_relation" name="XXX"
    duplicates="last" is_active="true">
    <fact_attr>
    <attr name="fact_lookup" value="XXX"/>
    <attr name="fact_name" value="XXX"/>
    <attr name="fact_category" value="XXX"/>
    <attr name="fact_sequence" value="0"/>
    <attr name="time_basis" value="n/a"/>
    <attr name="agg_method" value="n/a"/>
    <attr name="infer_delete_flag" value="false"/>
    </fact_attr>
    <point>P_PTC</point>
    <start_time>XXX</start_time>
    <filters>
    <filter>P_PTC</filter>          
    </filters>
    <parent_point>P_PARENT_PTC</parent_point>        
    </target> 
    </targets>
    </etl>     

Once the data has been entered in the XML elements, you may use the Tasks XML Editor to modify your settings.

Back to Top

FACT_AUDIT: Fact Audit

The fact audit task allows you to perform a fact audit of generated or loaded data (METRIC, INT_METRIC, and/or RELATION fact types). The fact audit looks for trends in data.

Copy 
---
   task:
   # fact audit task definition
   # -------------------------------------------------------
   name: task_factaudit
   module_name: dr.task.factAudit #.py is always assumed by python
   task_method_name: startTask
   post_task_method_name: postTask
   
   targets: [
   # [ 'METRIC', '*'],
   # [ 'INT_METRIC', '*'],
   # [ 'RELATION', '*'],
   
   ]
  
   # subscriber_list: [] # Default behavior. Will process all subscribers as set in FactAudit 
                         # worker code
   subscriber_list: ['FASDBAudit'] # Optionally override the subscribers you want to run for. 
                                   # If defined subscriber is invalid its run is skipped
   
   # The targets criteria may be limited by substituting a point type code for the '*' wildcard, 
   # which will audit facts for all points types. For example, if you want metric facts 
   # for an electric meter, the target would be:
   targets: [
   [ 'METRIC', 'EM'],
   ]
  

Back to Top

FOPS: File Operations

File operations rename files.

Copy 
---
   task:
   # ETL trigger for CSS Demand Reads
   # -------------------------------------------------------
   name: task_file_ops
   module_name: dr.task.fileOps #.py is always assumed by python
   task_method_name: startTask
   file_ops:
   - fops_rename:
   is_active: True
   conn_type: SFTP
   conn_name: ssh_sftp_int
   content_keyword_inspect: 
   file_path: /XXX/XXX
   source_file_pattern: XXX.XXX.[%Y.%m.%d].*.txt
   filename_keyword_exclude: xxx
   target_file_pattern: XXX.XXX.[%Y.%m.%d].*.txt
   target_task_env: XXX
   target_task_lookup: YYYY
   target_task_type_code: ETL

Back to Top

GEO: Geospatial

Copy 
---
  task:
  # GeoSpatial task definition
  # -------------------------------------------------------
  name: task_geospatial
  module_name: dr.task.geoSpatial #.py is always assumed by python
  task_method_name: startTask
  post_task_method_name: postTask
  
  debug_flag: False
  filter_limit: 1000
  proxy: http://geocoding.dataraker.net/geocoder
  #relations: [['PTC','PARENT_PTC']]  

Back to Top

PAGG: Point Aggregation

The point aggregation task aggregates data from child to parent types such as from a meter to a line transformer. Inputs include:

  • PTC: Point type code for child.
  • Mode: Fact type (INT, INT_METRIC, RELATION). See Fact Types.
  • factLookup List: Facts within the mode that you wish to aggregate.
  • targetPTC List: Point type that you wish to aggregate to.
  • targetMetric List: Metrics that you want returned.

Copy 
---
   task:
   # pagg task definition
   # -------------------------------------------------------
   name: task_pagg
   module_name: dr.task.pagg #.py is always assumed by python
   task_method_name: startTask
   post_task_method_name: postTask
   min_batch_count: 30
   
   # List of definitions in the following format:
   # [[ PointTypeCode, Mode, [factLookupList], [targetPTCList], [targetMetricList] ], ...]
   # Example:
   #    targets: [
   #       ['EM', 'INT_METRIC', ['HOURLY_KWH'], ['PRMS'], ['SUM']],
   #       ['EM', 'COUNT', ['DAILY_POWER_OUTAGE_FLAG'], ['PRMS'], []],
   #       ['EM', 'METRIC', ['DAILY_KWH_CSM'], [], ['AVG','SUM','P10','P90']]
   #    ]
   targets: [
   #['PTC', 'METRIC', ['FACT_LOOKUP'] ,['PARENT_PTC'], ['CNT','SUM']]
   ]  

If the target metric list is empty, all metrics will be returned.

Back to Top

RAGG: Relation Aggregation

The relation aggregation connects all parent point types to their child point types. There is no configuration needed.

Copy 
---
   task:
   # ragg task definition
   # -------------------------------------------------------
   name: task_ragg
   module_name: dr.task.ragg #.py is always assumed by python
   task_method_name: startTask
   post_task_method_name: postTask
   
   output_to_file_flag: False # sample output only for testing purpose
   source_ptc_list: ['*'] # this results in standard behavior

Back to Top

RANMAINT: Relation and Numeric Attribute Maintenance

The relation and numeric attribute maintenance task caches attribute data to enhance system performance. This task is performed after daily core processing. The cache is valid for 24 hours.

Copy 
---
   task:
   # RAN Maint task definition
   # -------------------------------------------------------
   name: task_ranmaint
   module_name: dr.task.cacheheatup #.py is always assumed by python
   task_method_name: startTask
   post_task_method_name: postTask
   source_ptc_list: ['*']

Back to Top

SCENARIO: Scenario

The scenario task runs a scenario.

Copy 
---
   task:
   # Scenario task definition
   # -------------------------------------------------------
   name: task_scenario
   module_name: dr.task.scenario #.py is always assumed by python
   task_method_name: startTask
   post_task_method_name: postTask
   debug_flag: False
   variables:         
   DUMMY: true

Back to Top

TAGG: Time Aggregation

The time aggregation task allows you to aggregate data over time.

Inputs

  • Point Type Code: Point type to use in the time aggregation.
  • Mode: Data fact type from:
    • METRIC (daily)
    • INT_METRIC (sub-daily)
    • EVENT (head-end system events)
  • Time Aggregation Basis: What time frame to aggregate to.
  • Day Types: Type of days to use in the aggregation:
    • WEEKDAY: Only weekdays (Monday through Friday)
    • WEEKEND: Only weekend days (Saturday through Sunday)
    • ALL_DAYS: All days (Sunday through Saturday)
  • Fact Types: Which facts to include in the aggregation.
  • Target Metrics: What metrics (e.g., average, count, etc.) to return.

Copy 
---
   task:
   # tagg task definition
   # -------------------------------------------------------
   name: task_tagg
   module_name: dr.task.tagg #.py is always assumed by python
   task_method_name: startTask
   post_task_method_name: postTask
   # List of definitions in the following format:
   # [[ (PointTypeCode, Mode, TAggBasis, [DayTypes], [factLookupList], [targetMetricList])] ..]
   # If DayTypes is not specified it defaults to all the day types supported: 
   # [ 'WEEKDAY' , 'WEEKEND', 'ALL_DAYS' ]
   # If targetMetricList is not specified it defaults to all the target 
   # metrics supported: [ 'CNT','SUM','AVG','SSQ','MAX','MXD']
   # Example:
   #    targets: [
   #       [ ACCNT, METRIC, MONTHLY, [], [SUM_DAILY_KWH_CSM], [] ],
   #       [ EM, INT_METRIC, DAILY, [], [HOURLY_KWH], [] ],
   #       [ EM, EVENT, MONTHLY, [], [DAILY_MAGNETIC_FLAG], [] ]
   #    ]
   #targets: [ [ ACCNT, METRIC, MONTHLY, [], [SUM_DAILY_KWH_CSM], [] ] ]
   
   targets: [
   #[ PTC, INT_METRIC, DAILY, ['ALL_DAYS'], ['FACT_LOOKUP'], ['SUM','CNT']]

Back to Top

Editing ETL Task XML with the Tasks XML Editor

ETL XML may be edited with the Tasks XML Editor. All elements shown in the template YAML's XML element correspond with information found in the XML Editor; however, it is best to fill in as much detail as possible with the Task Definition because some fields may be edited, but not added with the XML Editor.

Back to Top

Processing Tasks

Tasks can be processed manually or scheduled to run periodically by creating a run. Managing Run Information The Execute Tasks Details dialog box can be accessed from the Execute Now link found in the Manage Tasks’ View Tasks or Edit Tasks dialog boxes.

To process tasks:

  1. Click either the View or Edit link on Manage Tasks.
  2. Click the Execute Now link in the dialog box.
  3. In the Execute Tasks Details dialog box, do the following:
    1. Verify the name field (display only). This is the name assigned to the task when it was added.
    2. Enter the date that the task should be processed in the Start Date field,
    3. Enter the date that the task should be stopped if it is still running in the End Date field.
    4. Select the task’s priority on the node from the Priority drop-down. Base values include: Highest, Default, and Lowest.
    5. (Optional) Enter an email address to send a message to if the task ends unexpectedly in the Failure Email field.
    6. Verify the lookup field (display only). This is the task’s lookup name as designated when the task was added.
    7. Select the MANUAL processing method in the Type drop-down,.
    8. Select the type of task from the Run Mode drop-down (STD or Dynamic).

      Note: STD is standard. Dynamic is not in use.

    9. (Optional) Enter an email address to send a message to if the task completes successfully in the Success Email field.
  4. Click OK.

Back to Top