Example Checkpoint Behavior

As shown in Sample Streams Output, the reason why we want to implement checkpoints is so that our streams application will not consume operations from the very beginning of the stream every time it is run. Now that we have implemented checkpoints, our application will begin streaming from the last saved checkpoint.

On the initial run, of 100 operations, the application's behavior is no different from the original application, with the exception of the checkpoints. (Output is truncated for brevity.)

> java pubsub.GSGStreamExample
Subscriber created to stream 100 operations.
Start stream 100 operations from table Users

Found a put. Row is:
UID: 0
	Quantity: 10
	myArray: [19,10,3,6,14,17,20,8,7,20]

Found a put. Row is:
UID: 1
	Quantity: 5
	myArray: [2,3,10,12,5]

Found a put. Row is:
UID: 2
	Quantity: 9
	myArray: [16,6,19,17,6,11,19,1,6]

... skipped ops for brevity ...

Found a put. Row is:
UID: 9
	Quantity: 1
	myArray: [2]
Finish checkpoint at position {kvstore(id=1500857641631): 
[rg1(vlsn=69)]}

Checkpoint succeeded after 10 operations at position 
{kvstore(id=1500857641631): [rg1(vlsn=69)]}, elapsed time in ms 36

Found a put. Row is:
UID: 10
	Quantity: 3
	myArray: [4,7,9]

Found a put. Row is:
UID: 11
	Quantity: 5
	myArray: [14,9,14,14,12]

... skipped ops for brevity ...

Found a delete. Row is:
Del OP [seq: 233, shard id: 1, primary key: {
  "uid" : 54
}]

Found a put. Row is:
UID: 88
	Quantity: 6
	myArray: [4,12,2,2,11,9]

Found a put. Row is:
UID: 89
	Quantity: 1
	myArray: [4]
Fail to checkpoint at position {kvstore(id=1500857641631): 
[rg1(vlsn=249)]}, cause: Cannot do checkpoint because there 
is a concurrently running checkpoint for subscriber 1_0
Finish checkpoint at position {kvstore(id=1500857641631): 
[rg1(vlsn=249)]}

Checkpoint succeeded after 100 operations at position 
{kvstore(id=1500857641631): [rg1(vlsn=249)]}, elapsed time in ms 42
Publisher closed normally.
All done.  

Notice in the previous output that at least one checkpoint failed to complete because there was already a concurrently running checkpoint. This happened because we are taking checkpoints far too frequently in this example. As a consequnce, we tried to take a checkpoint before the previous checkpoint finished. Extending the checkpoint interval to something more reasonable would eliminate the error situation.

Having completed one run of the example program, a subsequent run will begin where the previous run left off. In this example run, the previous stream left off on the database write that created the row with UID 89. The next run begins with the write operation that created row UID 90.

> java pubsub.GSGStreamExample
Subscriber created to stream 100 operations.
Start stream 100 operations from table Users

Found a put. Row is:
UID: 90
	Quantity: 3
	myArray: [3,1,8]

Found a put. Row is:
UID: 91
	Quantity: 4
	myArray: [2,9,6,13]

Found a put. Row is:
UID: 92
	Quantity: 6
	myArray: [2,3,9,9,7,3]


... skipped ops for brevity ...