The 5800 System Query Integrity Model

The result set of any query will only return results that match the query. But will it return ALL the matching results? That is the concept of query completeness, referred to here as query integrity. 100% query integrity for a result set is defined as a state in which the result set contains all the objects in the 5800 system that match that particular query. The 5800 system is not always in a state of 100% query integrity. Various system events can induce a state in which the set of objects that are available for query is smaller than the total set of objects stored in the archive. Each query result set supports an operation (isQueryComplete) whereby the customer application can ask, once all the results from the query result set have been processed, whether that set of results constitutes a complete set.

The format of records as stored in the reliable and scalable object archive is not suitable for fast query. To enable searching, the queryable fields from the metadata are indexed in a query engine that can provide fast and flexible query services. The query engine is basically an SQL database. This is why the 5800 system's query language can borrow so heavily from SQL. At various times, the data as indexed in the query engine can get out of date compared to what is stored in the archive. When this happens, query result sets are not known to be complete until the contents of the query engine can be brought back up to date with the actual contents of the archive again.

The 5800 system concept of query integrity as actually implemented is somewhat looser than that of 100% query integrity. Even if a query result set indicates the result set is complete, the 5800 system allows certain objects, known as store index exceptions, to be missing from the query result set, as long as those exceptions were communicated to the customer application at the time the object was stored.

A store index exception is an object for which the original store of the object into the archive succeeded, but at least some part of the insert into the query engine (database) did not succeed. The object may or may not show up in all of the queries that it matches. A store index exception is communicated to the customer application at the time of store by means of a method SystemRecord.isIndexed. A value of false from isIndexed means that the object is not immediately available for query.

A store index exception is said to be resolved when the object becomes available for query. The checkIndexed method can be used to attempt to resolve a store index exception under program control. The checkIndexed operation checks if the object has been added to the query engine, and attempts to insert it if the object has not been added. If the insert into the query engine succeeds, the object is thereby restored to full queryability.

All store index exceptions will also eventually be resolved automatically by ongoing system healing. Each query result set also exports a method getQueryIntegrityTime that can be used to get detailed status on which store index exceptions might still be unresolved. The query integrity time is a time such that all store index exceptions from before that time have been resolved. There is an “ideal” query integrity time, which is the time of the oldest still-unresolved store index exception: an ideal implementation when asked for the query integrity time would always report this ideal value. In actual implementation, the reported query integrity time might be hours or even days earlier than the ideal query integrity time, depending on how far the ongoing system healing has progressed.