The following code provides a fully functional example of a multi-threaded transactional BDB XML application. The example creates multiple threads, each of which creates a set number of XML documents that it then writes to the container. Each thread creates and writes 10 documents under a single transaction before committing and writing another 10 documents. This activity is repeated 50 times.
From the command line, you can tell the program to vary:
The number of threads that it should use.
The number of nodes each XML document will contain.
Whether the container used by the program is of type Wholedoc or node storage.
Whether read committed (degree 2) isolation should be used for the container writes.
As we will see in Runtime Analysis each of these variables plays a role in the number of deadlocks the program encounters during its run time.
Of course, each writer thread performs deadlock detection as described in this manual. In addition, normal recovery is performed when the environment is opened.
This example program uses three classes:
TxnGuide.java
This is our main program. It opens and closes our environment, manager, and container and it spawns our worker threads for us.
This class is described in TxnGuide.java.
InfoKeeper.java
This is a utility class that we use to pass information to, and retrieve information from, our worker threads.
This class is described in InfoKeeper.java.
XMLWriter.java
This is our thread implementation. It performs all of our container writes for us. This class is where all of our transaction commit/abort code resides. We also perform deadlock handling in this class.
This class is described in XMLWriter.java.
This is our "main" class. We will use this class to open our environment, manager and container, and to spawn our worker threads.
To begin, we provide the normal package statement and the
necessary import statements.
// File TxnGuide.java package dbxml.txn; import com.sleepycat.dbxml.XmlContainer; import com.sleepycat.dbxml.XmlContainerConfig; import com.sleepycat.dbxml.XmlException; import com.sleepycat.dbxml.XmlManager; import com.sleepycat.dbxml.XmlManagerConfig; import com.sleepycat.db.DatabaseException; import com.sleepycat.db.Environment; import com.sleepycat.db.EnvironmentConfig; import com.sleepycat.db.LockDetectMode; import java.io.File; import java.io.FileNotFoundException;
Next we declare our class and we define the private data members that our class needs.
public class TxnGuide {
private static String myEnvPath = "./";
private static String containerName = "txn.dbxml";
// DBXML handles
private static Environment myEnv = null;
private static XmlManager mgr = null;
private static XmlContainer container = null;
private static InfoKeeper ik = null;
Now we implement our usage()
method. The information presented here is a duplicate of what we
have already described about this application in the preceding text.
In addition, we also describe the exact command line switches that
this application supports.
private static void usage() {
String msg = "\nThis program writes XML documents to a DB XML";
msg += "container. The documents are written using any\n";
msg += "number of threads that will perform writes\n";
msg += "using 50 transactions. Each transaction writes \n";
msg += "10 documents. You can choose to perform the ";
msg += "writes using default isolation, or using \n";
msg += "READ COMMITTED isolation. If READ COMMITTED ";
msg += "is used, the application will see fewer";
msg += "deadlocks.\n\n";
msg += "Note that you can vary the size of the documents ";
msg += "written to the container by defining the number\n";
msg += "of nodes in the documents. Up to a point, and";
msg += "depending on your system's performance,\n";
msg += "increasing the number of nodes will increase\n";
msg += "the number of deadlocks that your application\n";
msg += "will see.\n\n";
msg += "Command line options are: \n";
msg += " -h <database_home_directory>\n";
msg += " [-t <number of threads>]\n";
msg += " [-n <number of nodes per document>]\n";
msg += " [-w] (create a Wholedoc container)\n";
msg += " [-2] (use READ COMMITTED isolation)\n";
System.out.println(msg);
System.exit(-1);
}
Now for our main() method. To begin, we
instantiate an ParseArgs object (see
InfoKeeper.java
for information on
this object), and then we parse our command line arguments. As a
part of parsing those arguments, we will fill our
InfoKeeper object with relevant information
obtained from the command line. Once our argument list is parsed,
we open our environment, manager and container.
We parse our argument list and open our environment using methods private to this class. We describe these methods in a moment.
public static void main(String args[]) {
try {
ik = new InfoKeeper(5, // Num Threads
1, // Num Nodes
true, // Node storage container?
false); // Use ReadCommitted Isolation?
// Parse the arguments list
parseArgs(args);
// Open the environment and databases
openEnv();
After that, we need to spawn and join our worker threads. The
XMLWriter class that we use here is
described in
XMLWriter.java.
// Start the threads
XMLWriter[] threadArray;
threadArray = new XMLWriter[ik.getNumThreads()];
for (int i = 0; i < ik.getNumThreads(); i++) {
threadArray[i] = new XMLWriter(mgr, container, ik);
threadArray[i].start();
}
for (int i = 0; i < ik.getNumThreads(); i++) {
threadArray[i].join();
}
Once the threads have all completed, we print some statistics gathered during the program run.
// Report the run's results
System.out.println("\n\n\n");
System.out.println("Number of threads:\t\t"
+ ik.getNumThreads());
System.out.println("Number of doc nodes:\t\t"
+ ik.getNumNodes());
System.out.println("Using node storage:\t\t"
+ ik.getIsNodeStorage());
System.out.println("Using read committed:\t\t"
+ ik.getReadCommitted());
System.out.println("\nNumber deadlocks seen:\t\t"
+ ik.getNumDeadlocks());
Finally, we catch and manage all our exceptions. Notice that we
call another private member, closeEnv(),
in order to close our container and manager. We
describe this method next.
} catch (XmlException xe) {
System.err.println("TxnGuide: " + xe.toString());
xe.printStackTrace();
} catch (DatabaseException de) {
System.err.println("TxnGuide: " + de.toString());
de.printStackTrace();
} catch (Exception e) {
System.err.println("TxnGuide: " + e.toString());
e.printStackTrace();
} finally {
closeEnv();
}
System.out.println("All done.");
}
Having completed our main() method, we now
implement our private closeEnv()
method. The only thing to notice here is that we close the
container and manager, but not the environment. As we will see in
the next method, we allow our manager to adopt the environment.
This means that when the manager is closed, then so is the
environment.
private static void closeEnv() {
if (container != null) {
try {
container.close();
} catch (XmlException xe) {
System.err.println("closeEnv: container: " +
xe.toString());
xe.printStackTrace();
}
}
if (mgr != null) {
try {
mgr.close();
} catch (XmlException xe) {
System.err.println("closeEnv: mgr: " +
xe.toString());
xe.printStackTrace();
}
}
}
Now we can implement our openEnv()
method. As always, we begin by setting up our
EnvironmentConfig object. Here, we configure
the environment for transactional processing. We also allow the
environment to be created if it does not already exist and we cause
normal recovery to be run when the environment is opened.
private static void openEnv() throws DatabaseException {
System.out.println("opening env");
// Set up the environment.
EnvironmentConfig myEnvConfig = new EnvironmentConfig();
myEnvConfig.setAllowCreate(true);
myEnvConfig.setInitializeCache(true);
myEnvConfig.setInitializeLocking(true);
myEnvConfig.setInitializeLogging(true);
myEnvConfig.setRunRecovery(true);
myEnvConfig.setTransactional(true);
// EnvironmentConfig.setThreaded(true) is the default behavior
// in Java, so we do not have to do anything to cause the
// environment handle to be free-threaded.
We want to perform deadlock detection, so we configure that next. Here, we choose to resolve deadlocks by picking the thread with the smallest number of write locks. The thread with the smallest number of write locks is the one that has performed the least amount of work. By choosing this thread for the abort/retry cycle, we minimize the amount of rework our application must perform due to a deadlock.
myEnvConfig.setLockDetectMode(LockDetectMode.MINWRITE);
Now we simply open the environment.
try {
// Open the environment
myEnv = new Environment(new File(myEnvPath), // Env home
myEnvConfig);
} catch (FileNotFoundException fnfe) {
System.err.println("openEnv: " + fnfe.toString());
System.exit(-1);
}
Now we open the manager. Notice that when we configure the manager, we cause it to adopt the environment. As we stated above, this causes the manager to close our environment when it closes. We also remove our container if it happens to already exist; this program only ever begins with new, empty containers as a result.
try {
XmlManagerConfig managerConfig = new XmlManagerConfig();
// Close the environment when the manager closes
managerConfig.setAdoptEnvironment(true);
mgr = new XmlManager(myEnv, managerConfig);
// If the container already exists, delete it. We don't want
// naming conflicts if this program is run multiple times.
if (mgr.existsContainer(containerName) != 0) {
mgr.removeContainer(containerName);
}
Finally, we open our container. Notice how we define the
container's node storage type based on the information
recorded in our InfoKeeper
object. This information will be set based on data
collected from the command line when we parse our argument
list.
// Open the container
XmlContainerConfig containerConf = new XmlContainerConfig();
containerConf.setTransactional(true);
containerConf.setAllowCreate(true);
// Declare the container type; that is, whether it is a
// node-storage or a whole doc container. If -w is specified
// at the command line, the container is set to wholedoc,
// otherwise node-storage is used.
containerConf.setNodeContainer(ik.getIsNodeStorage());
container = mgr.openContainer(containerName, containerConf);
} catch (XmlException xe) {
System.err.println("TxnGuide: " + xe.toString());
xe.printStackTrace();
}
}
As a final bit of work, we implement our parseArgs()
method. Notice that we save most of the information collected here
to our InfoKeeper object for later usage and
reporting.
private static void parseArgs(String args[]) {
for(int i = 0; i < args.length; ++i) {
if (args[i].startsWith("-")) {
switch(args[i].charAt(1)) {
case 'h':
myEnvPath = new String(args[++i]);
break;
case 't':
ik.setNumThreads(Integer.parseInt(args[++i]));
break;
case 'n':
ik.setNumNodes(Integer.parseInt(args[++i]));
break;
case 'w':
ik.setIsNodeStorage(false);
break;
case '2':
ik.setReadCommit(true);
break;
default:
usage();
}
}
}
}
}
InfoKeeper is a simple utility class used to maintain information of
interest to the various threads in our example program. The information that it contains is either needed
by the various threads of control in order to know how to run, or is useful for reporting purposes at the
end of the program run, or both.
InfoKeeper is a trivial class that we present here only for
the purpose of completeness.
// File InfoKeeper.java
package dbxml.txn;
public class InfoKeeper
{
private static int numThreads; // Number of threads to use
private static int numNodes; // Number of nodes per
// document to generate.
private static boolean doNodeStorage; // Use node storage?
private static boolean doReadCommitted; // Use read committed
// isolation?
private static int deadlockCounter; // Number of deadlocks seen
// in this program run.
InfoKeeper(int nThreads, int nNodes, boolean nStorage,
boolean rCommit)
{
numThreads = nThreads;
numNodes = nNodes;
doNodeStorage = nStorage;
doReadCommitted = rCommit;
deadlockCounter = 0;
}
public synchronized void setNumThreads(int n) {
numThreads = n;
}
public synchronized void setNumNodes(int n) {
numNodes = n;
}
public synchronized void setIsNodeStorage(boolean n) {
doNodeStorage = n;
}
public synchronized void setReadCommit(boolean n) {
doReadCommitted = n;
}
public synchronized void incrementDeadlockCounter() {
deadlockCounter++;
}
public synchronized int getNumThreads() {
return numThreads;
}
public synchronized int getNumNodes() {
return numNodes;
}
public synchronized boolean getIsNodeStorage() {
return doNodeStorage;
}
public synchronized boolean getReadCommitted() {
return doReadCommitted;
}
public synchronized int getNumDeadlocks() {
return deadlockCounter;
}
}
XMLWriter is our thread implementation.
It is used to create a series of XML documents and then write
them to the container under the protection of a transaction.
The documents this class generates will vary in size depending
on the number of nodes specified at the command line. Beyond
that, however, each thread always performs 50 transactions and
a total of 10 documents are written to the container for each
transaction.
Each program run can also be varied somewhat in terms of the isolation level used by the transactions. Normally default isolation is used, but the user can relax this at the command line to use read committed isolation instead. Doing so can have positive implications for the program's throughput. See Runtime Analysis for more information.
To begin, we import our necessary classes. Note that
java.util.Random is used to generate
random data to place into our XML documents.
// File XmlWriter.java package dbxml.txn; import com.sleepycat.dbxml.XmlContainer; import com.sleepycat.dbxml.XmlException; import com.sleepycat.dbxml.XmlManager; import com.sleepycat.dbxml.XmlTransaction; import com.sleepycat.dbxml.XmlUpdateContext; import com.sleepycat.db.TransactionConfig; import java.util.Random;
Then we declare our class and we initialize several private data
members necessary to our class' operation. Notice the
MAX_RETRY variable. This is used for our deadlock
retry loop; our threads will never retry a transaction that deadlocks
more than MAX_RETRY times.
public class XMLWriter extends Thread
{
private Random generator = new Random();
private boolean passTxn = false;
private XmlManager myMgr = null;
private XmlContainer myContainer = null;
private InfoKeeper ik = null;
private static final int MAX_RETRY = 20;
Next we implement our class constructor. We set our BDB XML handles
here. We also get our InfoKeeper
object here, which is used to determine some important aspects of how
our thread is supposed to behave.
// Constructor. Get our DBXML handles from here
XMLWriter(XmlManager mgr, XmlContainer container, InfoKeeper info)
throws XmlException {
myMgr = mgr;
myContainer = container;
ik = info;
}
Next we implement our run() method. Every
java.lang.Thread class must implement
this method. This method is where our class will perform actual work,
so this is where we perform our transactional writes. This is also
where we will perform all our deadlock handling.
This method begins by defining a few variables of interest to us. Notice that this is where we discover how many nodes we should generate per XML document and whether read committed isolation should be used by our transactions.
public void run () {
Random generator = new Random();
XmlTransaction txn = null;
int numNodes = ik.getNumNodes();
boolean useRC = ik.getReadCommitted();
Then we begin our main transaction loop. As we do so, we initialize our
retry variable to true and our
retry_count variable to 0. So
long as retry_count does not exceed
MAX_RETRY, we will retry deadlocked transactions.
// Perform 50 transactions
for (int i=0; i<50; i++) {
boolean retry = true;
int retry_count = 0;
Now we enter our retry loop, and immediately the try
block that we use to perform deadlock handling and transaction aborts
(if required). At the top of the loop, we obtain an update context and
a TransactionConfig object. We use the
TransactionConfig object to identify the
whether we should use read committed isolation. Of course, this is
driven ultimately by information passed to the program from the command
line.
// while loop is used for deadlock retries
while (retry) {
// try block used for deadlock detection and
// general exception handling
try {
XmlUpdateContext context = myMgr.createUpdateContext();
// Configure whether the transaction will use Read
// Committed isolation. If -2 is specified on the
// command line, then Read Committed is used.
TransactionConfig tc = new TransactionConfig();
tc.setReadCommitted(useRC);
Now we (finally!) create our documents and write them to the container. Each document is written under the protection of a transaction. Each transaction is committed once 10 documents have been created and written.
// Get a transaction
txn = myMgr.createTransaction(null, tc);
// Write 10 records to the container
// for each transaction
for (int j = 0; j < 10; j++) {
// Get a document ID
String docID = getName() + i + j;
// Build the document
String theDoc = "<testDoc>\n";
for (int k = 0; k < numNodes; k++) {
theDoc += "<payload>" +
generator.nextDouble() + "</payload>\n";
}
theDoc += "</testDoc>";
// Put the document
myContainer.putDocument(txn,
docID,
theDoc,
context);
} // end inner for loop
// Commit
txn.commit();
txn = null;
retry = false;
To wrap things up, we have to perform error handling. First we look at
XmlException objects. In particular, we need to
look for deadlocks by examining the object to see if it encapsulates a
com.sleepycat.db.DeadlockException. If it does,
we will abort (this occurs in the finally clause,
and then retry if our retry count is low enough.
} catch (XmlException xe) {
retry = false;
// First, look for a deadlock and handle it
// if that is the cause of the exception.
if (xe.getDatabaseException() instanceof
com.sleepycat.db.DeadlockException) {
System.out.println(getName() +
" got deadlock exception!");
ik.incrementDeadlockCounter();
// retry if necessary
if (retry_count < MAX_RETRY) {
System.err.println(getName() +
" : Retrying operation.");
retry = true;
retry_count++;
} else {
System.err.println(getName() +
" : out of retries. Giving up.");
}
} else {
System.err.println("Error on txn commit: " +
xe.toString());
}
For general all-purpose exceptions, we simply refrain from retrying the operation.
} catch (Exception e) {
System.err.println(getName() +
" got general exception : " + e.toString());
retry = false;
Finally, we always abort the transaction if its handle is non-null. The
thread will then retry the transaction if the value of the
retry allows it.
} finally {
if (txn != null) {
try {
txn.abort();
} catch (Exception e) {
System.err.println("Error aborting txn: " +
e.toString());
e.printStackTrace();
}
}
}
} // end retry loop
} // end for 50 transactions loop
} // end of the run method
} // end of class
This completes our transactional example. If you would like to experiment with this code, you can find the example in the following location in your BDB XML distribution:
BDBXML_INSTALL/dbxml/examples/java/txn
In addition, please see Runtime Analysis for an analysis on the performance characteristic illustrated by this program.