Sunday, 7 October 2012

FAILED_NOT_RESTARTABLE in WebLogic Server 11g

Hi everyone,

Today when I have go through my weblogic console I saw my one managed server was in FAILED_NOT_RESTARTABLE state. In my case after checking logs I find the following error message  

“weblogic.management.ManagementException: Unable to obtain lock on/u01/oid_domain/servers/wls_ods1/tmp/wls_ods1.lok. Server may already be running at weblogic.management. internal.ServerLocks. getServerLock (ServerLocks.java:159)”

This could be because of two reasons
1) WebLogic Server which you are trying to start is already running.
2) WebLogic Server which you are trying to start did not stop cleanly.

Normally When weblogic server starts, it creates two lock files
i) $DOMAIN_HOME/servers/<server_name>/tmp/<servermame>.lok
ii) $DOMAIN_HOME/servers/<server_name>/data/ldap/ldapfiles/EmbeddedLDAP.lok

When WebLogic server stops, it removes these two files. If you not stop server properly these lock files remains on that location and creating a problem when you attempt to start a server again and after more than one attempt your server goes in FAILED_NOT_RESTARTABLE state.

If you hit problem, first identify if server is running on port configured for WebLogic Server by using netstat.
netstat -an | grep <WebLogic_Server_Port> for LINUX
netstat -aon | findstr <WebLogic_Server_Port> for  WIN

If server is running then you should see output like
tcp        0      0 ::ffff:<IP>:<weblogic_port>      :::* LISTEN  
(Listen here identifies that it is listening)

Note: There could be other process using this port
How to identify which port is configured for weblogic server?
To identify port configured for weblogic server open weblogic configuration file
$DOMAIN_HOME/config/config.xml and search for listen-port
You should see entry like
<listen-port>7006</listen-port>

If server is not running then you can safely remove these lok files under WebLogic server. And if server is running then trying to stop it from console, backend or by killing the process id of the particular server then remove the .lok files.

Note: If this is weblogic managed server (not Admin Server) then you can safely remove entire managed server directory (including sub directories) $DOMAIN_HOME/servers/<serverName> or rename it as a backup. When you start managed server again, Admin Server will create these directories.

“Effort only fully releases its reward after a person refuses to quit.”

Regards,
Akhilesh B. Humbe
 

2 comments:

  1. We tried this, but it failed to resolve the "FAILED_NOT_RESTARTABLE" error.

    Tried removing the managed server (oim_server1 in our case), from the console. No such luck.
    Restarted the Admin server... no luck
    removed the oim_server1 directory ... no luck.

    So what else are we missing here?
    (rob.baldassano@accenture.com)

    ReplyDelete
    Replies
    1. Make Sure that the JVM size is set properly

      Delete

Popular Posts