Verification is a tool you can use to:
Perform general troubleshooting of the store.
Verification inspects all components of the store. It also checks whether all store services are available, and for those services it checks whether there are any version or metadata mismatches.
Check the status of a long-running plan
Some plans require many steps and may take some time to execute.
The administrator can verify plans to check on progress of the
plan, or as an aid to diagnose a plan that is in an ERROR state.
For example, if you can verify a Deploy Store
plan while it is running against many Storage Nodes. You can
watch the verify report at each iteration to see that more and
more nodes have created and have come online.
For more information on how to manage plans, see Plans.
Provide additional information about a plan that is in an ERROR state.
You run store verification using the verify
command in the CLI. It requires no parameters, and by default it
runs in verbose mode. For example:
> java -Xmx256m -Xms256m \
-jar KVHOME/lib/kvstore.jar runadmin -port <registry port>> \
-host <hostname>
kv-> verify configuration
Verify: starting verification of store mystore based upon
topology sequence #1008
1000 partitions and 3 storage nodes
Time: 2015-03-04 21:16:02 UTC Version: 12.1.3.2.15
See <nodeHostname>:/KVRT1/mystore/log/mystore_{0..N}.log for
progress messages
Verify: Shard Status: total:1 healthy:1 degraded:0 noQuorum:0 offline:0
Verify: Zone [name=baskin id=zn1 type=PRIMARY]
RN Status: total:3 online:3 maxDelayMillis:0 maxCatchupTimeSecs:0
Verify: == checking storage node sn1 ==
Verify: Storage Node [sn1] on <nodeHostname>:5000
Zone: [name=baskin id=zn1 type=PRIMARY] Status: RUNNING
Ver: 12cR1.3.2.15 2015-03-04 06:35:02 UTC Build id: 8e70b50c0b0e
Verify: Admin [admin1] Status: RUNNING,MASTER
Verify: Rep Node [rg1-rn1] Status: RUNNING,REPLICA
sequenceNumber:2,025 haPort:5011 delayMillis:0 catchupTimeSecs:0
Verify: == checking storage node sn2 ==
Verify: Storage Node [sn2] on <nodeHostname>:5100
Zone: [name=baskin id=zn1 type=PRIMARY] Status: RUNNING
Ver: 12cR1.3.2.15 2015-03-04 06:35:02 UTC Build id: 8e70b50c0b0e
Verify: Rep Node [rg1-rn2] Status: RUNNING,REPLICA
sequenceNumber:2,025 haPort:5110 delayMillis:0 catchupTimeSecs:0
Verify: == checking storage node sn3 ==
Verify: Storage Node [sn3] on <nodeHostname>:5200
Zone: [name=baskin id=zn1 type=PRIMARY] Status: RUNNING
Ver: 12cR1.3.2.15 2015-03-04 06:35:02 UTC Build id: 8e70b50c0b0e
Verify: Rep Node [rg1-rn3] Status: RUNNING,MASTER
sequenceNumber:2,025 haPort:5210
Verification complete, no violations.
A silent mode is available which shows only problems.
kv-> verify configuration -silent
Verify: starting verification of store mystore based upon
topology sequence #1008
1000 partitions and 3 storage nodes
Time: 2015-03-04 21:16:02 UTC Version: 12.1.3.2.15
See <nodeHostname>:/KVRT1/mystore/log/mystore_{0..N}.log for
progress messages
Verification complete, no violations.
Problems with the store are clearly reported. For example, if a Storage Node is unavailable, then in silent mode that problem is displayed in the following way:
kv-> verify configuration -silent
Verify: starting verification of store mystore based upon
topology sequence #1008
1000 partitions and 3 storage nodes
Time: 2015-03-04 21:16:02 UTC Version: 12.1.3.2.15
See <nodeHostname>:/KVRT1/mystore/log/mystore_{0..N}.log for
progress messages
Verification complete, 2 violations, 0 notes found.
Verification violation: [sn2] ping() failed for sn2 :
Connection refused to host: <nodeHostname>; nested exception is:
java.net.ConnectException: Connection refused
Verification violation: [rg1-rn2] ping() failed for rg1-rn2 :
Connection refused to host: <nodeHostname>; nested exception is:
java.net.ConnectException: Connection refused
In verbose mode, the above problem is shown in the following way:
kv-> verify configuration
Verify: starting verification of store mystore based upon
topology sequence #1008
1000 partitions and 3 storage nodes
Time: 2015-03-04 21:16:02 UTC Version: 12.1.3.2.15
See <nodeHostname>:/KVRT1/mystore/log/mystore_{0..N}.log for
progress messages
Verify: Shard Status: total:1 healthy:0 degraded:1 noQuorum:0 offline:0
Verify: Zone [name=baskin id=zn1 type=PRIMARY]
RN Status: total:3 online:2 maxDelayMillis:0 maxCatchupTimeSecs:0
Verify: == checking storage node sn1 ==
Verify: Storage Node [sn1] on <nodeHostname>:5000
Zone: [name=baskin id=zn1 type=PRIMARY] Status: RUNNING
Ver: 12cR1.3.2.15 2015-03-04 06:35:02 UTC Build id: 8e70b50c0b0e
Verify: Admin [admin1] Status: RUNNING,MASTER
Verify: Rep Node [rg1-rn1] Status: RUNNING,REPLICA
sequenceNumber:2,025 haPort:5011 delayMillis:0 catchupTimeSecs:0
Verify: == checking storage node sn2 ==
Verify: sn2: ping() failed for sn2 : Connection refused to host:
<nodeHostname>; nested exception is:
java.net.ConnectException: Connection refused
Verify: Storage Node [sn2] on <nodeHostname>:5100
Zone: [name=baskin id=zn1 type=PRIMARY] UNREACHABLE
Verify: rg1-rn2: ping() failed for rg1-rn2 : Connection refused to
host: <nodeHostname>; nested exception is:
java.net.ConnectException: Connection refused
Verify: Rep Node [rg1-rn2] Status: UNREACHABLE
Verify: == checking storage node sn3 ==
Verify: Storage Node [sn3] on <nodeHostname>:5200
Zone: [name=baskin id=zn1 type=PRIMARY] Status: RUNNING
Ver: 12cR1.3.2.15 2015-03-04 06:35:02 UTC Build id: 8e70b50c0b0e
Verify: Rep Node [rg1-rn3] Status: RUNNING,MASTER
sequenceNumber:2,025 haPort:5210
Verification complete, 2 violations, 0 notes found.
Verification violation: [sn2] ping() failed for sn2 : Connection
refused to host: <nodeHostname>; nested exception is:
java.net.ConnectException: Connection refused
Verification violation: [rg1-rn2] ping() failed for rg1-rn2 :
Connection refused to host: <nodeHostname>; nested exception is:
java.net.ConnectException: Connection refused
The verify output is only displayed in the shell when the
command is finished. You can tail or grep the Oracle NoSQL Database log
file in order to get a sense of how the verify is progressing.
Look for the string Verify. For example:
grep Verify /KVRT1/mystore/log/mystore_0.log