About record generations

A set of records that has been committed to a Record Store instance is a record generation.

For example, if you perform a full crawl, all the records produced from the crawl are written to the Record Store and a commit is done. After the commit, the Record Store has one generation of records. A subsequent crawl, either full or incremental, results in a second generation of records.

Each record that is read in contains a unique ID. IAS uses that unique ID as the value of the idPropertyName Record Store configuration property.

If a record already exists with that unique ID during later IAS crawls, then the later version replaces the earlier one. This ensures that when you run an incremental crawl, you always get the latest version of any given record.

A record generation is removed from a Record Store instance by the clean task after the generation becomes stale. A stale generation is a generation that has been in a Record Store instance for a period of time that exceeds the value of the generationRetentionTime Record Store configuration property. A stale generation is retained in several exception cases:

A generation is currently in use. This occurs because either a READ transaction or a READ_WRITE transaction is running.
If there is only one generation in a Record Store instance, it is not removed, even if it is stale.
A generation is marked as a last-read generation for a client.