If the investigations in How to Troubleshoot Systems When IPsec Is Running fail to handle the problem, then the semantics of your configuration is the likely problem, rather than the syntax of your files or the service configuration.
If both the ike:default and ike:ikev2 service instances are enabled, ensure that the IKEv2 and IKEv1 rules do not overlap. Rules that apply to the same network endpoints can result in redundant IPsec SAs and could cause a lack of connectivity in certain situations.
If you change an IKE rule, read the rule into the kernel.
# ikeadm -v[1|2] read rule
If you are running IKEv1, make sure that the algorithm mechanisms in your rules are available on the IKEv1 system that you are connecting to. To view the available algorithms, run the ikeadm dump algorithms command on the system that does not support IKEv2:
# ikeadm dump groupsAvailable Diffie-Hellman groups # ikeadm dump encralgsAll IKE encryption algorithms # ikeadm dump authalgsAll IKE authentication algorithms
Correct both the IPsec and IKEv1 policy files to use algorithms that are available on both systems. Then, restart the IKEv1 service and refresh the IPsec service.
# svcadm restart ike:default; svcadm refresh ipsec/policy
If you are using preshared keys with IKEv1, and the remote IKEv1 system is rebooted, run the ipseckey flush command on the local system.
If you are using self-signed certificates, verify with the other administrator that a certificate with the same DN has not been re-created and that the hash values of your certificates match. For the verification steps, see Step 4 in How to Configure IKEv2 With Self-Signed Public Key Certificates.
If the certificate is updated, import the new certificate, then refresh and restart the IKEv2 service.
Use the ikeadm -v2 dump | get command to view the current IKEv2 configuration. For a usage summary, see Viewing IKE Information.
Use the kstat command to display IPsec-related statistics. For more information, see the kstat(1M) man page.
# kstat -m ipsecesp # kstat -m ipsecah # kstat -m ip
The kstat output in the following example indicates no problems in the ipsecesp module.
# kstat -m ipsecesp
module: ipsecesp instance: 0
name: esp_stat class: net
acquire_requests 18
bad_auth 0
bad_decrypt 0
bad_padding 0
bytes_expired 0
crtime 4.87974774
crypto_async 0
crypto_failures 0
crypto_sync 172
good_auth 86
keysock_in 135
num_aalgs 9
num_ealgs 13
out_discards 0
out_requests 86
replay_early_failures 0
replay_failures 0
sa_port_renumbers 0
snaptime 5946769.7947628
Use the snoop command to view the traffic that is not being protected. The Wireshark application can read snoop output. For an example of snoop output, see How to Verify That Packets Are Protected With IPsec.