Understanding Extract Synch for Data Futures

Once an extract is completed and reported to HESA via a successful XML file, the data set can be copied to the extract records. You shouldn't run extract synch until you resolve the errors in the extract data and the extract is submitted to HESA.

The first step of the process is to copy extract records. This creates a full set of new effective-dated/sequenced extract records by copying from the most recent effective-dated rows for all the extract records for the Data Futures (DF) return type. The synch process can run for multiple extracts on the same day so effective sequence is used to track the history of changes.

The Institution extract record is used to find the most recent effective-dated records that need to be copied. To copy extract records:

1. Select the most recent effective date and effective sequence from Institution extract for the Institution where Return Type is DFSTDNT.

2. Find the new effective date as the current date, and the new effective sequence as the highest existing effective sequence for the current date + 1 (or as 1, if the most recent record is for an earlier date).

3. Copy the existing Institution extract record using the new date/sequence from step 2.

4. For each extract entity and field record with return type = DFSTDNT:

   • Rows for the existing date/sequence selected in step 1 where the Entity or Field Status is not Delete are copied to new rows using the new effective date/sequence in step 2.

   • The Entity Status and Field Status values for all the new records is initially set to Unchanged.

   Note: Any rows for earlier effective dates that have been deleted before the most recent record and any records that were marked as Delete in the most recent record aren't copied to the new record.

If there's no existing Institution extract record for the DFSTDNT return type, the copy extract step is skipped. This will happen the first time the extract synch process is run.

Next is to copy the extract records.

If the extract Entity Status isn't one of the values in Entity Status Inclusion, the entity extract record, any child field records, and any child entities are skipped. These records aren't included in the XML and no updates are required to the corresponding extract records.

If the extract Entity Status is Identify, Delete, Unchanged, or Unused the Entity Status of the corresponding new extract record is updated to the new value. Any child field extract records are skipped. If there's no corresponding submitted data record, the record is skipped.

If the extract Entity Status is any of the values in Entity Status Inclusion (like Amended, Error, or New) and there's a corresponding submitted data record:

• The Entity Status of the corresponding new extract record is updated to the extract entity value.

• For each field record where the Field Status isn't one of the values in Field Status Inclusion, the field is skipped.

• If the field record already exists in submitted data, the Field Status of the submitted data record is updated to the extract field value and the other values (Derived Value, Derived Step, Reported Value, Override Flag) are also updated.

• If the field record doesn't already exist in submitted data, a new submitted data record is created using the new date/sequence and the Field Status is set to the extract field value.

If the extract Entity Status is any of the values in Entity Status Inclusion and there's no corresponding submitted data record, a new entity record is created in submitted data using the new date/sequence with Entity Status of the extract entity value. For each child field record where the Field Status is not one of the values in Field Status Inclusion, the field is skipped. A new submitted data field record is created with the same date/sequence as the parent entity and Field Status is set to the extract field value.

Active entity extract records are deleted along with related child field extract records if:

• The extract Entity Status is one of the values in Entity Status Inclusion and where the Synch checkbox is selected.

• All the field extract records have a Field Status that's either blank or is one of the values in Field Status Inclusion and where the Synch checkbox is selected.

• Any child entities have an Entity Status that's one of the values in Entity Status Inclusion and where the Synch checkbox is selected.

• Any child entity field extract records have a Field Status that's either blank or is one of the values in Field Status Inclusion and where the Synch checkbox selected