In-Memory Transaction Example

Some applications use XML documents in a transient manner. That is, they create and store XML documents as a part of their run time, but there is no need for the documents to persist between application restarts. For these class of applications, overall throughput can be improved by abandoning the transactional durability guarantee. To do this, you keep your environment, containers, and logs entirely in-memory so as to avoid the performance impact of unneeded disk I/O.

To do this:

As an example, this section takes the transaction example provided in Transaction Example and it updates that example so that the environment, container, log files, and regions are all kept entirely in-memory.

To do this, we only need to modify our main class, TxnGuide.java. To begin, we can simplify things a bit because we no longer need to know the environment home. We also don't need a container name and because we are not opening anything on disk, we no longer need to import java.io.File. Therefore, we:

The beginning of our program now looks like this:

package dbxml.txn;

import com.sleepycat.dbxml.XmlContainer;
import com.sleepycat.dbxml.XmlContainerConfig;
import com.sleepycat.dbxml.XmlException;
import com.sleepycat.dbxml.XmlManager;
import com.sleepycat.dbxml.XmlManagerConfig;

import com.sleepycat.db.DatabaseException;
import com.sleepycat.db.Environment;
import com.sleepycat.db.EnvironmentConfig;
import com.sleepycat.db.LockDetectMode;

import java.io.FileNotFoundException;

public class TxnGuideInMemory {

    // DBXML handles
    private static Environment myEnv = null;
    private static XmlManager mgr = null;
    private static XmlContainer container = null;

    private static InfoKeeper ik = null;

    private static void usage() {
        String msg =  "\nThis program writes XML documents to a DB XML";
               msg += "container. The documents are written using any\n";
               msg += "number of threads that will perform writes\n";
               msg += "using 50 transactions. Each transaction writes \n";
               msg += "10 documents. You can choose to perform the ";
               msg += "writes using default isolation, or using \n";
               msg += "READ COMMITTED isolation. If READ COMMITTED ";
               msg += "is used, the application will see fewer";
               msg += "deadlocks.\n\n";

               msg += "Note that you can vary the size of the documents ";
               msg += "written to the container by defining the number\n";
               msg += "of nodes in the documents. Up to a point, and";
               msg += "depending on your system's performance,\n";
               msg += "increasing the number of nodes will increase\n";
               msg += "the number of deadlocks that your application\n";
               msg += "will see.\n\n";

               msg += "Command line options are: \n";
               msg += " [-t <number of threads>]\n";
               msg += " [-n <number of nodes per document>]\n";
               msg += " [-w]       (create a Wholedoc container)\n";
               msg += " [-2]       (use READ COMMITTED isolation)\n";

        System.out.println(msg);
        System.exit(-1);
    }  

Our main() and closeEnv() methods are exactly identical to the code that we presented in Transaction Example so we will not repeat that here. The entire rest of the update occurs in the openEnv() method.

Until now we have only eliminated things from the program. This is to be expected; after all, we need to collect less information in order to operate and so our code should be slightly simpler.

But now we need to start adding information to tell the Berkeley DB library that it must keep information in-memory only. We start by making the environment private; this causes all the region files to be kept in memory. (Additional code is in bold.)

Note that we also change EnvironmentConfig.setRunRecovery() to false. Because our containers, logs, and regions are maintained in-memory, there can never be anything to recover.

    private static void openEnv() throws DatabaseException {
        System.out.println("opening env");

        // Set up the environment.
        EnvironmentConfig myEnvConfig = new EnvironmentConfig();
        
        // Region files are not backed by the filesystem, they are
        // backed by heap memory.
        myEnvConfig.setPrivate(true);

        myEnvConfig.setAllowCreate(true);
        myEnvConfig.setInitializeCache(true);
        myEnvConfig.setInitializeLocking(true);
        myEnvConfig.setInitializeLogging(true);
        myEnvConfig.setRunRecovery(false);
        myEnvConfig.setTransactional(true);

Now we configure our environment to keep the log files in memory, increase the log buffer size to 10 MB, and increase our in-memory cache to 10 MB. These values should be more than enough for our application's workload.

        // Specify in-memory logging
        myEnvConfig.setLogInMemory(true);
        // Specify the size of the in-memory log buffer
        // Must be large enough to handle the log data created by
        // the largest transaction.
        myEnvConfig.setLogBufferSize(10 * 1024 * 1024);
        // Specify the size of the in-memory cache
        // Set it large enough so that it won't page.
        myEnvConfig.setCacheSize(10 * 1024 * 1024); 

Next, we open the environment and setup our lock detection. This is identical to how the example previously worked, except that we do not provide a location for the environment's home directory.

        // Indicate that we want to internally perform deadlock 
        // detection. Also indicate that the transaction that has
        // performed the least amount of write activity to
        // receive the deadlock notification, if any.
        myEnvConfig.setLockDetectMode(LockDetectMode.MINWRITE);

        try {
            // Open the environment
            myEnv = new Environment(null,    // Env home
                                    myEnvConfig);

        } catch (FileNotFoundException fnfe) {
            System.err.println("openEnv: " + fnfe.toString());
            System.exit(-1);
        } 

When we open our container, we provide an empty string for the container name. This causes the container to be kept entirely in memory. Notice that we also remove the code that causes our manager to delete the existing container. Obviously, because everything will be kept entirely in-memory, there will never be anything to delete.

        // Open the manager and container
        try {
            XmlManagerConfig managerConfig = new XmlManagerConfig();
            // Close the environment when the manager closes
            managerConfig.setAdoptEnvironment(true);
            mgr = new XmlManager(myEnv, managerConfig);

            // Open the container
            XmlContainerConfig containerConf = new XmlContainerConfig();
            containerConf.setTransactional(true);
            containerConf.setAllowCreate(true);
            // Declare the container type; that is, whether it is a 
            // node-storage or a whole doc container. If -w is specified
            // at the command line, the container is set to wholedoc,
            // otherwise node-storage is used.
            containerConf.setNodeContainer(ik.getIsNodeStorage());
            container = mgr.openContainer("", containerConf);
        } catch (XmlException xe) {
            System.err.println("TxnGuide: " + xe.toString());
            xe.printStackTrace();
        }
    } 

That completes the update to openEnv(). The only other thing left to do is to modify the parseArgs() method so that it no longer accepts the -h option, or tries to set the myEnvPath variable that we deleted earlier in this example. Once completed, parseArgs() looks like this:

    private static void parseArgs(String args[]) {
        for(int i = 0; i < args.length; ++i) {
            if (args[i].startsWith("-")) {
                switch(args[i].charAt(1)) {
                    case 't':
                        ik.setNumThreads(Integer.parseInt(args[++i]));
                        break;
                    case 'n':
                        ik.setNumNodes(Integer.parseInt(args[++i]));
                        break;
                    case 'w':
                        ik.setIsNodeStorage(false);
                        break;
                    case '2':
                        ik.setReadCommit(true);
                        break;
                    default:
                        usage();
                }
            }
        }
    }
}  

That completes the updates we must make in order to cause the application to keep its environment, container, and logs entirely in memory. The XMLWriter and InfoKeeper classes are left entirely unchanged.

If you would like to experiment with this code, you can find the example in the following location in your BDB XML distribution: