4 Troubleshooting

This chapter contains information about troubleshooting security issue in Oracle Enterprise Manager.

4.1 Troubleshooting Authentication Issues in Enterprise Manager

Authentication can fail for a number of reasons. This section discusses ways to troubleshoot authentication failures. When Enterprise Manager is configured with external authentication, the LDAP/SSO WebLogic authentication providers authenticate the user. If authentication succeeds, the Enterprise Manager authentication layer checks if that user exists in Enterprise Manager repository. If authentication fails, the Enterprise Manager administrator should check ldap_trace.logATN located in the ../gc_inst/user_projects/domains/GCDomain' directory. This file contains authentication entries from the LDAP authenticator. If that file does not exist, you need to enable the WebLogic debug flag (DebugSecurityAtn).

4.1.1 Enabling the WebLogic Debug Flag

From the WebLogic Server Console, enable logging, set the WebLogic debug log level and then the enable the debug flag.

Navigate to Environment->Servers->EMGC_OMS1 and choose the Logging tab.
Under Rotation, set the file rotation size to 64000.
Expand Advanced.
Under Message destinations(s) (Log file: region) set the Severity level to Trace.
Navigate to the Environment->Servers->EMGC_OMS1 and choose the Debug tab.
Check WebLogic->security->atn->DebugSecurityAtn.
Click Save.

4.1.2 Debugging errors in ldap_trace.logATN file

In the Advanced section, set Minimum severity to log to Debug.
In Message destinations, set the Log file Severity level to Debug
Click Save/Activate changes.
Navigate to the Debug tab and enable debug for

weblogic->security->atn->DebugSecurityAtn
Click Save/Activate changes.

4.1.3 Invalid Credentials

Now, let's say user johndoe tries to log in with bad credentials. You should see something like the following in the ldap_trace.logATN. The non-zero resultCode of 49 indicates a bad password.

12:39:36.529 ldc=3 Connected to ldaps://<ldaphost>:3060
12:39:36.529 ldc=3 op=215 BindRequest {version=3, name=cn=orcladmin, authentication=********}
12:39:36.566 ldc=3 op=215 BindResponse {resultCode=0}
12:39:36.568 ldc=3 op=216 BindRequest {version=3, name=cn=johndoe,cn=Users,dc=us,dc=company,dc=com, authentication=********}
12:39:36.608 ldc=3 op=216 BindResponse {resultCode=49}

If user johndoe authenticates successfully by the LDAP provider but does not exist in the Enterprise Manager repository, then there should be no errors in the ldap_trace.logATN (you will see resultCode of 0) but in the Enterprise Manager log file emoms.log (under ../gc_inst/em/sysman/log) you should see something like the following:

2013-05-28 12:47:43,295 [[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'] WARN  auth.EMRepLoginFilter doFilter.457 - InvalidEMUserException caught in EMRepLoginFilter: Failed to login using external authentication for user: johndoe
oracle.sysman.emSDK.sec.auth.InvalidEMUserException: Failed to login using external authentication for user: johndoe
        at oracle.sysman.emSDK.sec.auth.EMLoginService._performLogin(EMLoginService.java:1269)
        at oracle.sysman.emSDK.sec.auth.EMLoginService._doSSOLogin(EMLoginService.java:754)
        at oracle.sysman.emSDK.sec.auth.EMLoginService.doSSOLogin(EMLoginService.java:727)
        at oracle.sysman.emSDK.sec.auth.EMLoginService.doLogin(EMLoginService.java:228)
        at oracle.sysman.emSDK.sec.auth.EMLoginService.doLogin(EMLoginService.java:256)

To obtain more detailed debug data in emoms.trc, you can enable the Enterprise Manager authentication logger via the following command:

emctl set property -name "log4j.category.oracle.sysman.core.security.auth" -value "DEBUG, emtrcAppender

emctl list properties will display something similar to the following output.

log4j.category.oracle.sysman.core.security.auth=DEBUG, emtrcAppender

Note:

Do not leave this debug logger enabled for performance reasons. Once the authentication issue has been diagnosed, turn this logger off using 'emctl delete property -name ".category.oracle.sysman.core.security.auth"

4.1.4 Timeout in LDAP Server'

There could be other non-zero resultCodes in ldap_trace.logATN that would indicate some other type of failure in the LDAP authentication layer. If the provider times out while trying to fetch results from the LDAP server, you should see something similar to the following in the file.

09:36:44.168 ldc=2 op=214 AbandonRequest {msgid=213}

To fix this issue, you can increase the Results time limit configuration parameter in the LDAP provider.

4.1.5 Errors Outside ldap_trace.logATN'

Sometimes, ldap_trace.logATN may not give the complete picture. In that case, check the diagnostic log EMGC_OMS1-diagnostic.log for errors/warnings from the configured external authentication provider.

Occasionally, There might be intermittent network issues between the LDAP server and the OMS host or the search base given as input might be too broad. You can use the 'ldapsearch' command that comes with the OS on your OMS host to validate connection/result retrieval timing issues. This command may not be available on all operating systems. You may use other LDAP tools. For example,

ldapsearch -h hostname -p 3060 -D cn=orcladmin -x -w xxxxxxx -b "cn=users,dc=us,dc=oracle,dc=com" -s one -l 5 -z 15

Where:

lh - hostname of the ldap server
p - port of the ldap server
D - bind dn
x - use simple authentication rather than SASL
w - password for the bind dn used above
l - gives a time limit in seconds for a search to complete
b - search base
s - specify scope search
z - specifies the limit of search result entries to be returned

You can give the user as well as the group base dn (that you specify in the emctl command) for the -b option and check if the appropriate results are returned and within the expected timeframe.

If you start seeing delays or timeouts, you should run this command from the same machine where Enterprise Manager is installed to ensure it is not LDAP server/network related.