DataRaker Integration

The NMS RTAdapter can be configured to support integration with the Oracle DataRaker application. DataRaker captures large quantities of periodic Advanced Metering Infrastructure (AMI) data – typically once a day. AMI data typically includes usage information like meter load (kWH) and voltage for each interval – where intervals are typically hourly or every 15 minutes. By analyzing months (or more) of AMI usage data, DataRaker can detect a broad range of usage anomalies. A few examples include transformer overloads (by aggregating loads from all meters below the load transformer), voltage sags, voltage swells, and abnormal usage patterns.

If DataRaker analysis is routinely executed (for example, daily to pick up yesterday's AMI data), it can be beneficial to make at least a subset of the DataRaker discovered anomalies clearly visible to NMS operators. RTAdapter can be configured to "import" a set of DataRaker discovered anomalies as NMS conditions (symbols that show in the NMS Viewer or simply records in the NMS Condition Summary tool). These DataRaker anomalies are captured as NMS conditions in the data_raker RDBMS table via the RTAdapter. Once captured, these NMS conditions allow navigation back into DataRaker from NMS.

DataRaker provides raw data files (in DataRaker export file format) that must be mapped/translated to the RTAdapter formats noted below. Typically DataRaker would generate raw DataRaker export files containing the desired data on a relatively routine (daily or weekly) basis. The raw DataRaker files must be transformed (via project specific adapters) into the specified RTAdapter format files for subsequent consumption. A project specific adapter (likely a perl or python script) must be used to translate the DataRaker export format file into the RTAdapter import format data file. This allows a project to determine what DataRaker data they want to include/exclude and how they want to categorize the input data for their NMS operators.

The following three use cases describe how the DataRaker to NMS data integration can be managed by a project using the RTAdapter. The first use case is expected to be the most common – the others are options for NMS administrators to consider as necessary.

Use Cases

Use Case 1 – DataRaker Master (Minimal NMS Involvement)

For this use case, DataRaker is the master of all DataRaker discovered anomalies. This means the anomalies generated by a new DataRaker analysis execution completely replace whatever DataRaker anomalies (NMS conditions) that were previously reported to (or captured by) NMS. This has the advantage of not requiring any periodic updates from NMS operators. NMS operators can view the DataRaker conditions if they choose, act on them if they have time, etc., but they can also ignore the DataRaker conditions and be confident they will not accumulate in NMS over time.

To accomplish this, RTAdapter must be configured to scan files (not the RDBMS) for this instance of the adapter. Below is an example RTAdapter record from the system.dat file that could be applicable for this case:

program DataRaker RTAdapter -scada dataraker -interval 60 -dir ~/dataraker

This means this RTAdapter instance will be known as DataRaker internal to NMS (on the message bus) and will look for configuration records that match scada_ids.id where scada_ids.scada_name='dataraker' in the SCADA_IDS RDBMS table. This id will then need to be matched by appropriate configuration in the SCADA_SYNONYMS and SCADA_STATES tables for this DataRaker instance of the RTAdapter. Note there is no required SCADA_STATES table configuration so it will not be further discussed in this section.

Example configuration:

• Records in the RTAdapter SCADA_SYNONYMS configuration table for DataRaker configuration use SCADA_IDS.ID=100 where SCADA_IDS.SCADA_NAME='dataraker' (to match example system.dat configuration noted above).

• The SCADA_SYNONYMS table configuration records noted below will allow the RTAdapter to process add, rem(ove) and syn(chronize) directive records from an RTAdapter input file for a single type of DataRaker condition (where NMS data_raker condition status=10). See the example OPAL_scada.sql file for more examples of DataRaker conditions. By default the NMS OPAL model is configured to handle 3 different DataRaker condition class statuses (Info=10, Warn=20, Alarm=30). To utilize OPAL type configuration the DataRaker Info, Warn and Alarm conditions must have the specified condition status values of 10, 20 and 30 respectively. These values are somewhat arbitrary and can be changed if necessary – but will require more project specific configuration. Entries for "DR_E2" are similar – just change status_value='20'. Entries for "DR_E3" are also similar – just change status_value='30'.

INSERT into scada_synonyms (id, scada_id, keyword, value,

process_type, attribute_alias, status_value)

VALUES (tmp_seq.nextval, 100, 'DR_E1', 'add', 'C',

'data_raker', '10');

INSERT into scada_synonyms (id, scada_id, keyword, value,

process_type, attribute_alias, status_value)

VALUES (tmp_seq.nextval, 100, 'DR_E1', 'rem', 'C',

'data_raker', '10');

INSERT into scada_synonyms (id, scada_id, keyword, value,

process_type, attribute_alias, status_value)

VALUES (tmp_seq.nextval, 100, 'DR_E1', 'syn', 'C',

'data_raker', '10');

Technically only the "syn" entry above is required for the first use case option (use RTAdapter to periodically synchronize DataRaker conditions). The "add" and "rem" options are configured just in case a project wants RTAdapter to also process individual add/rem directives in the RTAdapter input file. They are not required, but are used for Use Case 2 and Use Case 3.

Below is an example (translated) DataRaker export file suitable for RTAdapter consumption. You can give the file any valid file name, but it must ultimately be placed in the directory specified via "RTAdapter –dir <directory>" before it will be processed. The RTAdapter data files in the specified directory are processed in a first in, first out basis. The "OBJECT" keyword specifies an NMS object that must match an RTAdapter configured rti_alias (from scada_measurements_st.rti_alias) or an alias entry in the alias_mapping table where db_type='OPS' (default alias).

Any data_raker records in NMS that are NOT found in a RTAdapter synchronization input file will be deleted (based on the value of the external id provided by the EXT= keyword). NMS should only be left with conditions that are specified in the RTAdapter synchronization file when processing is completed. The pipe symbol "|" is a delimiter and can be changed via the RTAdapter –delimter <n> command line option. Replace "|" with your project specified delimiter in the examples below, if necessary:

SYNCHRONIZE|TRUE

TYPE|data_raker

DATA

OBJECT|xfmr_1

OBJECT|xmfr_2

END_DATA

Detailed field descriptions of RTAdapter keywords:

• SYNCHRONIZE|TRUE is a keyword sequence that says this entire file is a synchronization file. Always specify exactly as noted for a synchronization file request. Must start in column 1.

• TYPE is a keyword that specifies what condition class the synchronization will focus on for all subsequent entries (data_raker in the above example). All entries in a given RTAdapter synchronization file must be of the same class. Must start in column 1.

• OBJECT is a keyword that precedes the common object id between NMS and DataRaker – as this is the link between the two systems. Normally this is a transformer alias.

• Project configured keywords – defined in scada_synonyms table

• DR_E1 indicates we are dealing with a DataRaker "Info condition".

• DR_E2 indicates we are dealing with a DataRaker "Warn condition".

• DR_E3 indicates we are dealing with a DataRaker "Alarm condition".

• syn is a directive keyword that indicates we want to make sure this condition exists in NMS. It will update an existing condition or (if there is no current matching condition based on the EXT id) it will insert a new one.

• WHO= is a keyword that indicates what external system to indicate as the source of the condition directive. Normally this would be "dataraker" (or similar) for DataRaker integration.

• TIM= is a keyword that specifies a timestamp to go with the directive. Generally from the external system and must be specified in ISO format (the format you see above YYYY-MM-DDTHH:MM:SS – where hours are 00->24 and local time zone is assumed).

• TXT= is a keyword that specifies a brief summation of the condition detected. Can be up to 512 characters, but less is generally more. Suggest this be just enough to convey the type of issue detected – at least on first 30 characters of the text or so.

• EXT= is a keyword that specifies the unique external id for this condition. This should be the primary key for this condition. It could be generated by DataRaker or it could be generated by the NMS translation process. Ultimately it should be unique for at least every transformer and data_raker condition status combination in play (project specific).

Notes on the NMS unique key (EXT=) field. If DataRaker reports an anomaly for NMS transformer "xfmr_1" that maps to an NMS data_raker alarm (status=10) today – we could set the EXT= value to "xfmr_1-10" – indicating there is a data_raker alarm (status=10) on xfmr_1. This way if tomorrow DataRaker reports the same (or essentially the same) condition we would again translate to "xfmr_1-10" – and nothing would change on the NMS side for this condition. This scheme minimizes processing in NMS and allows "old" NMS data_raker conditions to "age" within NMS. This should provide some indication of how long DataRaker has been reporting similar issues for this transformer, which may be useful to NMS operators. Otherwise the conditions will be reported as new every day.

Use Case 2 – DataRaker Only Inputs New Conditions – NMS Deletes

Use Case 2 can use the same configuration as Use Case 1 other than the format of the input file to RTAdapter. Here DataRaker only inputs new conditions into NMS and NMS operators have the option to view/add/delete DataRaker conditions as necessary. If/when subsequent DataRaker input is processed, it will be captured "in addition to" whatever DataRaker conditions were already present within NMS. Ideally, for this use case, NMS operators would delete all (old) DataRaker conditions before processing a new batch of DataRaker conditions. If the operators do not delete the old DataRaker conditions they will tend to accumulate.

This option may be useful if it becomes necessary to leave some DataRaker conditions on the system for a substantial period of time before they are acted on and you do not want to automatically remove previously imported DataRaker conditions during every import (like the "SYNCHRONIZE" option does in Use Case 1). Maybe one execution of DataRaker finds conditions of one type and the next iteration finds conditions of another and you do not want to delete the records from the first pass before applying the second (or similar). The format for the input file would be similar to what follows. The primary differences are that there is no SYNCHRONIZE or TYPE keywords and (instead of the directive "syn") we use the "add" action directive.

DATA OBJECT|xfmr_1

OBJECT|xmfr_2

END_DATA

Use Case 3 – Bulk Delete for NMS Conditions

Use Case 3 can use the same configuration as Use Case 1 other than the format of the input file processed by RTAdapter. In this use case, the RTAdapter input file specifies what conditions to delete from NMS. The creation of this form of input file would need to be done by the NMS project implementers. This option may be useful if an NMS administrator wants to automatically delete some subset of previously applied DataRaker conditions (for whatever reason). The format of the input file is similar to Use Case 2 except that no "TXT=" data is processed. All other fields are processed and can be captured in the NMS data_raker condition table. Note that "add" and "rem" records can also be combined in the same file, but must be specified in proper order ("del" must follow an "add" for same condition, for example).

DATA OBJECT|xfmr_1

DR_E1|rem|WHO=dataraker|TIM=2009-02-27T16:22:17|EXT=DR_E1-1

DR_E1|rem|WHO=dataraker|TIM=2009-02-27T17:22:17|EXT=DR_E1-2

DR_E2|rem|WHO=dataraker|TIM=2009-02-27T18:22:17|EXT=DR_E2-1

OBJECT|xmfr_2

DR_E3|rem|WHO=dataraker|TIM=2009-02-27T16:22:17|EXT=DR_E3-1

DR_E3|rem|WHO=dataraker|TIM=2009-02-27T19:22:17|EXT=DR_E3-2

END_DATA