Skip Headers
Oracle® Coherence Developer's Guide
Release 3.6.1

Part Number E15723-03
Go to Documentation Home
Home
Go to Book List
Book List
Go to Table of Contents
Contents
Go to Feedback page
Contact Us

Go to previous page
Previous
Go to next page
Next
View PDF

20 Querying Data In a Cache

Coherence can perform queries and indexes against currently cached data that meets a given set of criteria. Queries and indexes can be simple, employing filters packaged with Coherence, or they can be run against multi-value attributes such as collections and arrays.

Query Overview

Coherence provides the ability to search for cache entries that meet a given set of criteria. The result set may be sorted if desired. Queries are evaluated with Read Committed isolation.

It should be noted that queries apply only to currently cached data (and will not use the CacheLoader interface to retrieve additional data that may satisfy the query). Thus, the data set should be loaded entirely into cache before queries are performed. In cases where the data set is too large to fit into available memory, it may be possible to restrict the cache contents along a specific dimension (for example, "date") and manually switch between cache queries and database queries based on the structure of the query. For maintainability, this is usually best implemented inside a cache-aware data access object (DAO).

Indexing requires the ability to extract attributes on each Partitioned cache node; in the case of dedicated CacheServer instances, this implies (usually) that application classes must be installed in the CacheServer classpath.

For Local and Replicated caches, queries are evaluated locally against unindexed data. For Partitioned caches, queries are performed in parallel across the cluster, using indexes if available. Coherence includes a Cost-Based Optimizer (CBO). Access to unindexed attributes requires object deserialization (though indexing on other attributes can reduce the number of objects that must be evaluated).

Query Concepts

The concept of querying is based on the ValueExtractor interface. A value extractor is used to extract an attribute from a given object for querying (and similarly, indexing). Most developers will need only the ReflectionExtractor implementation of this interface. The ReflectionExtractor uses reflection to extract an attribute from a value object by referring to a method name, typically a "getter" method like getName().

ValueExtractor extractor = new ReflectionExtractor("getName");

Any "void argument" method can be used, including Object methods like toString() (useful for prototype/debugging). Indexes may be either traditional "field indexes" (indexing fields of objects) or "functional indexes" (indexing "virtual" object attributes). For example, if a class has field accessors getFirstName and getLastName, the class may define a function getFullName which concatenates those names, and this function may be indexed. See "Using Query Indexes" for more information on indexes.

To query a cache that contains objects with getName attributes, a Filter must be used. A filter has a single method which determines whether a given object meets a criterion.

Example 20-1 Equality Filter

Filter filter = new EqualsFilter(extractor, "Bob Smith");

Note that the filters also have convenience constructors that accept a method name and internally construct a ReflectionExtractor:

Example 20-2 Filter that Constructs a ReflectionExtractor

Filter filter = new EqualsFilter("getName", "Bob Smith");

Example 20-3 illustrates a routine to select the entries of a cache that satisfy a particular filter:

Example 20-3 Selecting Cache Entries that Satisfy a Filter

for (Iterator iter = cache.entrySet(filter).iterator(); iter.hasNext(); )
    {
    Map.Entry entry = (Map.Entry)iter.next();
    Integer key = (Integer)entry.getKey();
    Person person = (Person)entry.getValue();
    System.out.println("key=" + key + " person=" + person);
    }

Example 20-4 illustrates using a filter to select and sort cache entries:

Example 20-4 Selecting and Sorting Cache Entries that Satisfy a Filter

// entrySet(Filter filter, Comparator comparator) 
Iterator iter = cache.entrySet(filter, null).iterator();

The additional null argument specifies that the result set should be sorted using the "natural ordering" of Comparable objects within the cache. The client may explicitly specify the ordering of the result set by providing an implementation of Comparator. Note that sorting places significant restrictions on the optimizations that Coherence can apply, as sorting requires that the entire result set be available before sorting.

Performing Simple Queries

Example 20-5 demonstrates how to create a simple query and uses the GreaterEqualsFilter filter.

Example 20-5 Querying the Cache with a Filter

Filter filter = new GreaterEqualsFilter("getAge", 18);

for (Iterator iter = cache.entrySet(filter).iterator(); iter.hasNext(); )
    {
    Map.Entry entry = (Map.Entry) iter.next();
    Integer key = (Integer) entry.getKey();
    Person person = (Person) entry.getValue();
    System.out.println("key=" + key + " person=" + person);
    }

Coherence provides a wide range of filters in the com.tangosol.util.filter package.

Note:

  • The Partitioned Cache implements this method using the Parallel Query feature, which is only available in Coherence Enterprise Edition or higher. When working with a Partitioned Cache in Coherence Standard Edition, this method will retrieve the data set to the client for processing.

  • Although queries can be executed through a near cache, the query will not use the front portion of a near cache. If using a near cache with queries, the best approach is to use the following sequence:

    Set setKeys = cache.keySet(filter);
    Map mapResult = cache.getAll(setKeys);
    

Using Query Indexes

Query indexes allow values (or attributes of those values) and corresponding keys to be correlated within a QueryMap to increase query performance. Indexes are a feature of Coherence Enterprise Edition or higher.

The following topics are included in this section:

Creating an Index

The addIndex method of the QueryMap class is used to create indexes. Any attribute able to be queried may be indexed using this method. The method includes three parameters:

addIndex(ValueExtractor extractor, boolean fOrdered, Comparator comparator)

Example 20-6 demonstrates how to create an index:

Example 20-6 Sample Code to Create an Index

NamedCache cache = CacheFactory.getCache("MyCache");
ValueExtractor extractor = new ReflectionExtractor("getAttribute");
cache.addIndex(extractor, true, null);

The fOrdered argument specifies whether the index structure is sorted. Sorted indexes are useful for range queries, such as "select all entries that fall between two dates" or "select all employees whose family name begins with 'S'". For "equality" queries, an unordered index may be used, which may have better efficiency in terms of space and time.

The comparator argument can be used to provide a custom java.util.Comparator for ordering the index.

The addIndex method is only intended as a hint to the cache implementation and, as such, it may be ignored by the cache if indexes are not supported or if the desired index (or a similar index) already exists. It is expected that an application will call this method to suggest an index even if the index may already exist, just so that the application is certain that index has been suggested. For example in a distributed environment, each server will likely suggest the same set of indexes when it starts, and there is no downside to the application blindly requesting those indexes regardless of whether another server has already requested the same indexes.

Note that queries can be combined by Coherence if necessary, and also that Coherence includes a cost-based optimizer (CBO) to prioritize the usage of indexes. To take advantage of an index, queries must use extractors that are equal ((Object.equals()) to the one used in the query.

A list of applied indexes can be retrieved from the StorageManagerMBean by using JMX. For more information, see Chapter 34, "How to Manage Coherence Using JMX".

Creating User-Defined Indexes

Applications can choose to create user-defined indexes to control which entries are added to the index. User-defined indexes are typically used to reduce the memory and processing overhead required to maintain an index. To create a user-defined index, an application must implement the MapIndex interface and the IndexAwareExtractor interfaces. This section also describes the ConditionalIndex and ConditionalExtractor classes which provide an implementation of the interfaces to create a conditional index that uses an associated filter to evaluate whether or not an entry should be indexed.

Implementing the MapIndex Interface

The MapIndex interface is used to correlate values stored in an indexed Map (or attributes of those values) to the corresponding keys in the indexed Map. Applications implement this interface to supply a custom index.

The following example implementation defines an index that only adds entries with non-null values. This would be useful in the case where there is a cache with a large number of entries and only a small subset have meaningful, non-null, values.

public class CustomMapIndex implements MapIndex
   {
   public void insert(Map.Entry entry)
       {
       if (entry.getValue()!= null)
           {
           ...
           }
       }
   ...
   }

In the above example, the value of the entry is checked for null prior to extraction, but it could be done after. If the value of the entry is null then nothing is inserted into the index. A similar check for null would also be required for the MapIndex update method. The rest of the MapIndex methods need to be implemented appropriately as well.

Implementing the IndexAwareExtractor Interface

The IndexAwareExtractor interface is an extension to the ValueExtractor interface that supports the creation and destruction of a MapIndex index. Instances of this interface are intended to be used with the QueryMap API to support the creation of custom indexes. The following example demonstrates how to implement this interface and is for the example CustomMapIndex class that was created above:

public class CustomIndexAwareExtractor
        implements IndexAwareExtractor, ExternalizableLite, PortableObject
    {
    public CustomIndexAwareExtractor(ValueExtractor extractor)
        {
        m_extractor = extractor;
        }
 
    public MapIndex createIndex(boolean fOrdered, Comparator comparator,
            Map mapIndex)
        {
        ValueExtractor extractor = m_extractor;
        MapIndex       index     = (MapIndex) mapIndex.get(extractor);
 
        if (index != null)
            {
            throw new IllegalArgumentException(
                    "Repetitive addIndex call for " + this);
            }
 
        index = new CustomMapIndex(extractor, fOrdered, comparator);
        mapIndex.put(extractor, index);
        return index;
        }
 
    public MapIndex destroyIndex(Map mapIndex)
        {
        return (MapIndex) mapIndex.remove(m_extractor);
        }
    ...
    }

In the above example, an underlying extractor is actually used to create the index and ultimately extracts the values from the cache entries. The IndexAwareExtractor implementation is used to manage the creation and destruction of a custom MapIndex implementation while preserving the existing QueryMap interfaces.

The IndexAwareExtractor is passed into the QueryMap.addIndex and QueryMap.removeIndex calls. Coherence, in turn, calls createIndex and destroyIndex on the IndexAwareExtractor. Also note that it is the responsibility of the IndexAwareExtractor to maintain the Map of extractor-to-index associations that is passed into createIndex and destroyIndex.

Using a Conditional Index

A conditional index is a custom index that implements both the MapIndex and IndexAwareExtractor interfaces as described above and uses an associated filter to evaluate whether or not an entry should be indexed. An entry's extracted value is only added to the index if the filter evaluates to true. The implemented classes are ConditionalIndex and ConditionalExtractor, respectively.

The ConditionalIndex is created by a ConditionalExtractor. The filter and extractor used by the ConditionalIndex are set on the ConditionalExtractor and passed to the ConditionalIndex constructor during the QueryMap.addIndex call.

The ConditionalExtractor is an IndexAwareExtractor implementation that is only used to create a ConditionalIndex. The underlying ValueExtractor is used for value extraction during index creation and is the extractor that is associated with the created ConditionalIndex in the given index map. Using the ConditionalExtractor to extract values in not supported. For example:

ValueExtractor extractor = new ReflectionExtractor("getLastName");
Filter filter = new NotEqualsFilter("getId", null);
ValueExtractor condExtractor = new ConditionalExtractor(filter, extractor, true);
 
// add the conditional index which should only contain the last name values for the
// entries with non-null Ids
cache.addIndex(condExtractor, true, null);

Batching Queries and Memory Usage

In order to preserve memory on the client issuing a query, there are various techniques that can be used to retrieve query results in batches.

Using the keySet form of the queries – combined with getAll() – reduces memory consumption since the entire entry set is not deserialized on the client at once. It also takes advantage of near caching:

Example 20-7 Using a keySet Query Format

// keySet(Filter filter)
Set setKeys = cache.keySet(filter);
Set setPageKeys = new HashSet();
int PAGE_SIZE = 100;
for (Iterator iter = setKeys.iterator(); iter.hasNext();)
    {
    setPageKeys.add(iter.next());
    if (setKeyPage.size() == PAGE_SIZE || !iter.hasNext())
        {
        // get a block of values
        Map mapResult = cache.getAll(setPageKeys);

        // process the block
        // ...

        setPageKeys.clear();
        }
    }

A LimitFilter may be used to limit the amount of data sent to the client, and also to provide paging . Example 20-8 demonstrates using a LimitFilter:

Example 20-8 Using a Limit Filter

int pageSize = 25;
Filter filter = new GreaterEqualsFilter("getAge", 18);
// get entries 1-25
Filter limitFilter = new LimitFilter(filter, pageSize);
Set entries = cache.entrySet(limitFilter);

// get entries 26-50
limitFilter.nextPage();
entries = cache.entrySet(limitFilter);

When using a distributed/partitioned cache, queries can be targeted to partitions and cache servers using a PartitionedFilter. This is the most efficient way of batching query results as each query request will be targeted to a single cache server, thus reducing the number of servers that must respond to a request and making the most efficient use of the network.

Note:

Use of PartitionedFilter is limited to cluster members; it cannot be used by Coherence*Extend clients. Coherence*Extend clients may use one of the two techniques described above, or these queries can be implemented as an Invocable and executed remotely by a Coherence*Extend client.

To execute a query partition by partition:

DistributedCacheService service =
   (DistributedCacheService) cache.getCacheService();
int cPartitions = service.getPartitionCount();
 
PartitionSet parts = new PartitionSet(cPartitions);
for (int iPartition = 0; iPartition < cPartitions; iPartition++)
    {
    parts.add(iPartition);
    Filter filterPart = new PartitionedFilter(filter, parts);
    Set setEntriesPart = cache.entrySet(filterPart);
 
    // process the entries ...
    parts.remove(iPartition);
    }

Queries can also be executed on a server by server basis:

DistributedCacheService service =
   (DistributedCacheService) cache.getCacheService();
int cPartitions = service.getPartitionCount();
 
PartitionSet partsProcessed = new PartitionSet(cPartitions);
for (Iterator iter = service.getStorageEnabledMembers().iterator();
        iter.hasNext();)
    {
    Member member = (Member) iter.next();
    PartitionSet partsMember = service.getOwnedPartitions(member);
 
    // due to a redistribution some partitions may have already been processed
    partsMember.remove(partsProcessed);
    Filter filterPart = new PartitionedFilter(filter, partsMember);
    Set setEntriesPart = cache.entrySet(filterPart);
 
    // process the entries ...
    partsProcessed.add(partsMember);
    }
 
// due to a possible redistribution, some partitions may have been skipped
if (!partsProcessed.isFull())
    {
    partsProcessed.invert();
    Filter filter = new PartitionedFilter(filter, partsProcessed);
 
    // process the remaining entries ...
    }

Queries Involving Multi-Value Attributes

Coherence supports indexing and querying of multi-value attributes including collections and arrays. When an object is indexed, Coherence will verify if it is a multi-value type, and will then index it as a collection rather than a singleton. The ContainsAllFilter, ContainsAnyFilter and ContainsFilter are used to query against these collections.

Example 20-9 Querying on Multi-Value Attributes

Set searchTerms = new HashSet();
searchTerms.add("java");
searchTerms.add("clustering");
searchTerms.add("books");

// The cache contains instances of a class "Document" which has a method
// "getWords" which returns a Collection<String> containing the set of
// words that appear in the document.
Filter filter = new ContainsAllFilter("getWords", searchTerms);

Set entrySet = cache.entrySet(filter);

// iterate through the search results
// ...

ChainedExtractor

The ChainedExtractor implementation allows chained invocation of zero-argument (accessor) methods. In Example 20-10, the extractor will first use reflection to call getName() on each cached Person object, and then use reflection to call length() on the returned String.

Example 20-10 Chaining Invocation Methods

ValueExtractor extractor = new ChainedExtractor("getName.length");

This extractor could be passed into a query, allowing queries (for example) to select all people with names not exceeding 10 letters. Method invocations may be chained indefinitely, for example getName.trim.length.