WebLogic Server is Not in the Running Status

After upgrading, the WebLogic admin server does not change to the running status.

After upgrading to Weblogic Server to 12c (12.2.1.3.0), the admin server on WebLogic Server 12c (12.2.1.3) does not start. It does not display any relevant message for the current state in the admin server logs.

While tacking a thread dump we see:

"[STANDBY] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" #39 daemon prio=5 os_prio=0 tid=<tid#> nid=0x1c38a runnable [0x00007fc4348bc000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at weblogic.nodemanager.common.NMReader.readLength(NMReader.java:223)
at weblogic.nodemanager.common.NMReader.readChunk(NMReader.java:200)
at weblogic.nodemanager.common.NMReader.readLine(NMReader.java:41)
at weblogic.nodemanager.common.NMInputOutput.readLine(NMInputOutput.java:30)
at weblogic.nodemanager.common.DataFormat.readResponse(DataFormat.java:504)
at weblogic.nodemanager.client.NMServerClient.getResponseString(NMServerClient.java:900)
at weblogic.nodemanager.client.NMServerClient.checkResponse(NMServerClient.java:884)
at weblogic.nodemanager.client.NMServerClient.connect(NMServerClient.java:733)
at weblogic.nodemanager.client.NMServerClient.checkConnected(NMServerClient.java:668)
at weblogic.nodemanager.client.NMServerClient.checkConnected(NMServerClient.java:674)
at weblogic.nodemanager.client.NMServerClient.initState(NMServerClient.java:519)
- locked <0x00000000f58683a8> (a weblogic.nodemanager.client.PlainClient)
at weblogic.nodemanager.mbean.NodeManagerRuntime.initState(NodeManagerRuntime.java:600)
at weblogic.server.ServerLifeCycleRuntime.<init>(ServerLifeCycleRuntime.java:161)
at weblogic.server.ServerLifeCycleService.createServerLifeCycleRuntime(ServerLifeCycleService.java:263)
at weblogic.server.ServerLifeCycleService.start(ServerLifeCycleService.java:149)
at weblogic.server.AbstractServerService.postConstruct(AbstractServerService.java:76)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1287)
at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:333)
at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:375)
at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487)
at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:305)
at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:85)
at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126)
at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:116)
- locked <0x00000000e228e260> (a java.lang.Object)
at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:90)
at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1237)
at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1168)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:678)
at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:352)
at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:337)
at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:57)
at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)
at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:652)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:420)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:360

The admin server is connected to a node manager that is not running correctly or replying correctly.

As per stack trace we can see that communication to the node manager is established. By checking NMServerClient.java source, we see that it is still reading, but not progressing. This might be due to a network issue. In thread dumps, we can see that this is an on going process.

the same error may occur if the admin server is connecting to a node manager running on a different version of the WebLogic Server. In a clustered setup, this might occur if the node manager is running on any of the node in the cluster.

To solve this issue:

Ensure that the node manager on all nodes in the cluster setup is shut down. This allows the WLS admin Server to start-up without any issue.

Also, ensure that the node manager in the remote machine is of the same WebLogic Server version. In case of Weblogic upgrade, ensure that the node manager is not running on any of the nodes in the cluster.

For more information, see Doc ID 2431508.1.