Understanding Extract Sync for Data Futures

Once an extract is completed and reported to HESA via a successful XML file, the data set can be copied to the extract records. You shouldn't run extract sync until you resolve the errors in the extract data and the extract is submitted to HESA.

The first step of the process is to copy extract records. This creates a full set of new effective-dated/sequenced extract records by copying from the most recent effective-dated rows for all the extract records for the Data Futures (DF) return type. The sync process can run for multiple extracts on the same day so effective sequence is used to track the history of changes.

The Institution extract record is used to determine the most recent effective-dated records that need to be copied. To copy extract records:

1. Select the most recent effective date and effective sequence from Institution extract for the Institution where Return Type is DFSTDNT.

2. Determine the new effective date as the current date, and the new effective sequence as the highest existing effective sequence for the current date + 1 (or as 1, if the most recent record is for an earlier date).

3. Copy the existing Institution extract record using the new date/sequence from step 2.

4. For each extract entity and field record with return type = DFSTDNT:

   • Rows for the existing date/sequence selected in step 1 where the Entity or Field Status is not Delete are copied to new rows using the new effective date/sequence in step 2.

   • The Entity Status and Field Status values for all the new records is initially set to Unchanged.

   Note: Any rows for earlier effective dates that have been deleted prior to the most recent record and any records that were marked as Delete in the most recent record are not copied to the new record.

If there is no existing Institution extract record for the DFSTDNT return type, the copy extract step is skipped. This will happen the first time the extract sync process is run.

Next is to copy the extract records.

If the extract Entity Status is Error, Unchanged, or blank, the entity extract record, any child field records, and any child entities are skipped. These records are not included in the XML and no updates are required to the corresponding extract records.

If the extract Entity Status is Identify or Delete, the Entity Status of the corresponding new extract record is updated to the new value. Any child field extract records are skipped.

If the extract Entity Status is Amended:

• The Entity Status of the corresponding new extract record is updated to Amended.

• For each child field record with Field Status of Unchanged or blank, the field is skipped.

• For each child field record with Field Status of Amended, the Field Status of the corresponding new extract record is updated to Amended and the other values (Derived Value, Derived Step, Reported Value, Override Flag) are updated.

• For each child field record with Field Status of New, a new extract record is created using the new date/sequence with Field Status of New.

If the extract Entity Status is New, a new entity record is created in extract using the new date/sequence with Entity Status of New. For each child field record with Field Status of New, a new extract record is created with the same date/sequence as the parent entity and Field Status of New.

Active entity extract records are deleted along with related child field extract records if the extract Entity Status is not Error and either the entity exists in the latest extract record with Entity Status of New, Amended, Identify, or Delete, or the extract Entity Status is Unchanged and there are no child entities with Entity Status of Error. This deletes extract records with Entity Status of New, Amended, and Identify that have been included in the XML and synced to extract. And also records with Entity Status of Delete and Unchanged that have been skipped by the Create XML process. Parent entities that have child entities with errors are not deleted to allow extracts records to be viewed.