Downloading the Status Document

When you perform the Download Data Content action and provide a content source, GTM logs onto the external system via the URL specified and downloads an XML file known as a status document.

The URL in GTM's external system points to an XML file on the content provider's server. That XML file is called a status document by the content provider. The status document is an XML file that specifies the current status of the data available to GTM, all the available data types, the dates that they were last updated, and a link to download the data files.

Based on the information in the status document and how you have configured GTM (what types to data to download, how often to check the content provider, etc.), GTM will parse the status document and download any new content available.

Note: If you want to download country-specific HS codes, it is recommended that you create all the necessary data loading configurations in only the PUBLIC domain. Also, the download process should be triggered from only the PUBLIC domain.

Note: For the Content Type – RULES OF ORIGIN, the Source Country should be left blank in the Content Source Data Type grid of Content Source UI. It is not considered even if provided. When the content type – RULES OF ORIGIN is selected as supported, GTM downloads all the rules of origin mentioned in the status document.

Sample Status Document

<?xml version="1.0" encoding="us-ascii"?>
<Status_Document xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
 <Version_ID>39</Version_ID>
 <Last_Update_DT>10/14/2010</Last_Update_DT>
 <File_Detail>
  <Files>
         <File_ID>12</File_ID>
        <Update_Indicator>0</Update_Indicator>
        <Data_Type>HSN</Data_Type>
        <Country>mx</Country>
        <Description>Tariff Data, specifically HS Codes and Descriptions. Contains HSMaster and HSParts tables.</Description>
         <URL>https:/example.com/SampleData/mx_**_20080918.zip</URL>
  </Files>
  <Files>
       <File_ID>78</File_ID>
       <Update_Indicator>1</Update_Indicator>
       <Data_Type>DPL</Data_Type>
       <Country>us</Country>
       <Description>Master DPL</Description>
      <URL>https:/example.com/SampleData/dpl_**_20100819.zip</URL>
  </Files>
  <Files>
       <File_ID>97</File_ID>
       <Data_File>DPL</Data_File>
       <Update_Indicator>1</Update_Indicator>
       <Requires_File_ID>96</Requires_File_ID>
       <Data_Type>DPL</Data_Type>
       <Country>us</Country>
       <Publish_DT>10/01/2010</Publish_DT>
       <Description>Delta DPL</Description>
       <URL>https:/example.com/SampleData/dpl_**_20101001_16_20.zip</URL>
  </Files>
    <Files>
      <File_ID>5035</File_ID>
      <Data_File>Eccn-Descriptions-HK-EN</Data_File>
      <Update_Indicator>0</Update_Indicator>
      <Requires_File_ID>0</Requires_File_ID>
      <Data_Type>ECCN</Data_Type>
      <Country>HK</Country>
      <Publish_DT>04/15/2014</Publish_DT>
      <Description>ECCN text broken down by paragraphs</Description>
      <URL>https://example.com/SampleData/ECCN_**_en 20140418_09_04.zip</URL>
      <Language_Cd>EN</Language_Cd>
    </Files> 
    <Files>
      <File_ID>12860</File_ID>
      <Data_File>ScheduleBNomenclature</Data_File>
      <Update_Indicator>0</Update_Indicator>
      <Requires_File_ID>0</Requires_File_ID>
      <Data_Type>SCHEDB</Data_Type>
      <Country>US</Country>
      <Publish_DT>01/24/2017</Publish_DT>
      <Description>Schedule B Nomenclature. Contains HSMaster and HSParts tables.</Description> <URL>https://example.com/SampleData/schedb_en_20170128_12_19.zip</URL>
      <Language_Cd>EN</Language_Cd>
    </Files> 
    <Files>
      <File_ID>14906</File_ID>
      <Data_File>mlNomenclature</Data_File>
      <Update_Indicator>0</Update_Indicator>
      <Requires_File_ID>0</Requires_File_ID>
      <Data_Type>HSN</Data_Type>
      <Country>ML</Country>
      <Publish_DT>03/14/2014</Publish_DT>
      <Description>Munition Lists</Description>
      <URL>https://example.com/SampleData/ml_**_en 20140314_14_30.zip</URL>
      <Language_Cd>EN</Language_Cd>
    </Files>
</File_Detail>
</Status_Document>

Consider the above sample status XML file snippet provided by the third party content provider. There are a couple of key attributes that should be noted:

Version_ID: GTM uses this element to check whether the status document has been previously downloaded or processed.
File_ID: The content provider can push changes to data content in file format. This ID is unique and is tied to a corresponding data .zip file. It is never re-used in any further downloads.
Update_Indicator: This tells you if the .zip file is a master data list or an incremental update list. If the indicator is 0, it is a master data list; if 1, then it is an update/delta list.
Requires_File_ID: This can be used to indicate the dependency between the files. For denied party data, this identifies the data file (master or delta) which should be processed before the current data file (delta). For classification data, this identifies the data content file in which the details are present in the English language. Content providers need to always provide the Required_File ID (the file with English language records) for other languages.
Country: The 2-digit country code for which this .zip file data applies.
URL: URL for the .zip files to download.

The third party content provider should start by publishing a status document with only master data.

For restricted party data, there should be one master file, which is the consolidated list of all the denied parties. When new entries need to be added, they should be created as delta files. All the master data files should have an Update_Indicator of “0”. All the delta data files should have an Update_Indicator of “1” and an appropriate File_ID and a Version_ID in the status document.

There is no delta file concept related to product classification and rules of origin data as GTM downloads the complete list of product classification data and rules of origin data every time. The Update_Indicator should always be 0 in case of product classification code data and rules of origin data.

Processing of the Status Document

GTM processes the status document as follows:

Looks at the Version_ID.
Reads each File_ID and checks if they are processed.
If the File_ID has been processed, there is nothing else for GTM to do.
If the File_ID has not been processed, GTM downloads the .zip file and loads the content.

Data Version

A data version is a set of data to be considered for any GTM service, such as denied party screening, product classification, or rules of origin. A data version is composed of two attributes, the type of data (DPL/HSN/ECCN/ Schedule-B), and the country to which it applies. Therefore, there is one data version for US/DPL data, one version for US/HSN data, one version for MX/HSN data, and so on. Ideally, there is one data version marked as current for type of data and country combination. For restricted parties, there is one data version marked as current for a content source. For product classifications, there is one data version marked as current for type of data and country combination for a content source. For rules of origin, there will be one data version active for any trade agreement and the importing partner for a content source.

GTM loads the content as follows:

If the update indicator is 1, then GTM will look up the current data version for that particular data type and country. A new data version (GTM_DATA_VERSION_GID) is not created and updates are made to the existing data version. These updates are referred to as a delta list and are typically seen when a client downloads restricted party data.
If the update indicator is 0, then this is a master data file. Typically, this is seen when a client first downloads the data or if the data is classification code data or rules of origin data. When the indicator is 0, GTM will create a new data version for this data type and country combination. Therefore, it is possible that there will be multiple data versions for classification data such as HSN, ECCN, ML, and Schedule B. GTM will then set this new data version as current, and set all the other existing data versions that apply for this data type and country to be not current.