FAILED_NOT_RESTARTABLE in WebLogic Server 11g

Hi everyone,

Today when I have go through my weblogic console I saw my one managed server was in FAILED_NOT_RESTARTABLE state. In my case after checking logs I find the following error message  

“weblogic.management.ManagementException: Unable to obtain lock on/u01/oid_domain/servers/wls_ods1/tmp/wls_ods1.lok. Server may already be running at weblogic.management. internal.ServerLocks. getServerLock (ServerLocks.java:159)”

This could be because of two reasons

1) WebLogic Server which you are trying to start is already running.

2) WebLogic Server which you are trying to start did not stop cleanly.

Normally When weblogic server starts, it creates two lock files

i) $DOMAIN_HOME/servers/<server_name>/tmp/<servermame>.lok

ii) $DOMAIN_HOME/servers/<server_name>/data/ldap/ldapfiles/EmbeddedLDAP.lok

When WebLogic server stops, it removes these two files. If you not stop server properly these lock files remains on that location and creating a problem when you attempt to start a server again and after more than one attempt your server goes in FAILED_NOT_RESTARTABLE state.

If you hit problem, first identify if server is running on port configured for WebLogic Server by using netstat.

netstat -an | grep <WebLogic_Server_Port> for LINUX

netstat -aon | findstr <WebLogic_Server_Port> for  WIN

If server is running then you should see output like

tcp        0      0 ::ffff:<IP>:<weblogic_port>      :::* LISTEN  

(Listen here identifies that it is listening)

Note: There could be other process using this port

How to identify which port is configured for weblogic server?

To identify port configured for weblogic server open weblogic configuration file

$DOMAIN_HOME/config/config.xml and search for listen-port

You should see entry like

<listen-port>7006</listen-port>

If server is not running then you can safely remove these lok files under WebLogic server. And if server is running then trying to stop it from console, backend or by killing the process id of the particular server then remove the .lok files.

Note: If this is weblogic managed server (not Admin Server) then you can safely remove entire managed server directory (including sub directories) $DOMAIN_HOME/servers/<serverName> or rename it as a backup. When you start managed server again, Admin Server will create these directories.

“Effort only fully releases its reward after a person refuses to quit.”

Regards,

Akhilesh B. Humbe