ADMU3011E: Server launched but failed initialization

Hello Everyone,

Today I was facing this issue on one of my environment of Websphere Application server 8.5 ND. While try to start the server  I was facing this issue and not allow me to start the server/JVM. And showed the following error when execute ./startServer.sh  

ADMU3011E: Server launched but failed initialization. Server logs, startServer.log, and other log files under /usr/IBM/WebSphere85/profiles/AppSrv01/logs/server1 should contain failure information.

After net surfing I got the following solution.

It is not easy to conclude the reason of issue by the error message which shown above. There may a different reason for that issue like JVM/Heap Size, file permission etc., so for digging exact issue you need to check the JVM logs, startup logs or ffdc logs.

In my case it was not writing SystemOut.log and SystemErr.log so I finally got a detailes of error in startServer.log.

Error:

javax.naming.ServiceUnavailableException : A communication failure occurred while attempting to obtain an initial context with the provider URL: "iiop://localhost:2811/". Make sure that any bootstrap address information in the URL is correct and that the target name server is running. A bootstrap address with no port specification defaults to port 2809. Possible causes other than an incorrect bootstrap address or unavailable name server include the network environment and workstation network configuration.

Solution:

In my case I resolve the error by using following steps:

1. Login to the websphere admin console.
2. Go to the Application servers > Server_Name> Process definition > Java Virtual Machine

1. When I check Boot classpath and Generic JVM arguments I found some syntax error is there.
2. I repair the errors and take the restart of JVM/Server and it works.

Hope this will work in your case also. Kindly comment for your suggestion and quires.

 "Effort only fully releases its reward after a person refuses to quit.” 

 Regards, 
 Akhilesh B. Humbe

CWSIS1574E: The file store's permanent store file is full.

Hi All,

I was getting following error in my SystemOut.log and getting some errors on application level. When I search for this error I came across the concept of file store in Websphere Application server. 

Exception Message: 

CWSIA0067E: An exception was received during the call to the method JmsMsgProducerImpl.sendMessage (#4): com.ibm.ws.sib.processor.exceptions.SIMPResourceException: com.ibm.websphere.sib.exception.SIResourceException: CWSIP0002E: An internal messaging error occurred in com.ibm.ws.sib.processor.impl.ConsumerDispatcher, 1:787:1.280.5.25, com.ibm.ws.sib.msgstore.RollbackException: CWSIS1002E: An unexpected exception was caught during transaction completion. Exception: com.ibm.ws.sib.msgstore.PersistenceFullException: CWSIS1574E: The file store's permanent store file is full.

Cause:

This error is occurred due to the file store's permanent store file is full. The same can happened in case of  if file store's temporary store file is full or  if file store's log file is full.

Solution:

We can perform any of the following two steps, depending upon the scenario. 

1. We can clear/purge the  permanent store file,  temporary store file and  log file [if issue is not getting frequently ]
2. We can increase the size of permanent store file,  temporary store file and  log file [if issue is  getting frequently ]

Clear/Purge the File Store:

If you are not getting the issue frequently you can simply clear the logs to resolve it. We can perform the following steps to clear the store files.

1. First clear all the queues in the SIB bus which causes the problem. For that login to the WAS console and go to the location
Buses > Bus_Name > Messaging engines > Messaging engines_Name > Queue points

2. It will show you the list of all queues and current depth (Number of message it contain)
Click on the queue name > Runtime tab > Messages and it will list you all the messages, simply click on Delete all button to clear the queue

Similarly clear all the queues which contain messages.

3. Before clear the file store stop the message engine on required bus. For that go to the location Buses > Bus_Name > Messaging engines > Messaging engines_Name  select message engine and click on stop.

4. To Clear a file store, the below flat files from the server have to be backed up

a. Log
b. TemporaryStore
c. PermanentStore

The path of the file store files, on the server, can be obtained from the Buses > Bus_Name > Messaging engines > Messaging engines_Name >Message store

5. Once backed up a file store , you can delete  Log, TemporaryStore, PermanentStore or can make them empty.

6. Restart the Messaging engine. [if messaging engine doesn't start then restart the JVM]

Increase size of the File Store:

As I told previously if you are getting the same issue frequently you can increase the size of file store. We can perform the following steps for same.

Here in our case we are getting error CWSIS1574E: The file store's permanent store file is full. So we are increasing the size of permanent store only.

1. Login to the WAS admin console, the path of the file store files, on the server, can be obtained from the Buses > Bus_Name > Messaging engines > Messaging engines_Name >Message store

2. Here on this screen you can increase the value of Minimum permanent store size and Maximum permanent store size. In our case we increase it to minimum 500MB and maximum 750MB.

3. Clear/Purge the file store as discussed above.
4. Restart the Messaging engine. [if messaging engine doesn't start then restart the JVM]

This works in my case, hope will help you also. Kindly comment for your suggestion and quires.    

"Effort only fully releases its reward after a person refuses to quit.”

 Regards,

 Akhilesh B. Humbe

What is Message stores in WebSphere Application Server

Hi all,

When I came across SIB (Service Integration Bus) and it's working in Websphere Application Server. I think that it is very important to us to know about the some basic concept like Message Store so I just try to collect some information which is already there in IBM Knowledge Center.

Message stores

Message stores are important in the operation of messaging engines. To host queue-type destinations, a messaging engine includes a message store where, if necessary, it can hold messages until consuming applications are ready to receive them, or preserve messages in case the messaging engine fails. Each messaging engine has one and only one message store. This can either be a file store or a data store.
A message store enables a messaging engine to preserve operating information and to retain those objects that messaging engines need for recovery in the event of a failure.

A messaging engine preserves both volatile and durable data in its message store. Volatile data is lost when a messaging engine stops, in either a controlled or an uncontrolled manner. Durable data is available after the server restarts. A messaging engine stores various types of data, including messages, transaction states, and communication channel states.

When started, a messaging engine obtains configuration information from the WCCM ( WebSphere Application Server Common Configuration Model) repository. A messaging engine retrieves all other data from its own file store or data store.

There are currently no facilities available for migrating from a data store to a file store.
• File stores
• Data stores

File stores

File stores use a file system to preserve operating information and to persist the objects that messaging engines need for recovery in the event of a failure.
A file store is a type of message store that directly uses files in a file system through the operating system. The data storage in a file store is split into three levels:
1. Log file
2. Permanent store file
3. Temporary store file.

Log file :
This file contains information about currently active transactions and data that is not yet written to a store file. It is a circular log and its file size is static while a messaging engine is running, but can be changed if required. A restart of the messaging engine is required for the changes to take effect. The size of the log file limits the maximum size of a message that can be sent.

Permanent store file :
This file contains permanent data that is retained after the restart of the messaging engine, such as persistent messages, queue data, and information about the storage and transmission of persistent messages.
The permanent store file can be configured to have a maximum and minimum size, or to be unlimited in size. The file can grow from the minimum size (or as required in the unlimited case) but will never shrink (even if the maximum size is set lower than its current size). The file sizes can be changed in the administration console, but a restart of the messaging engine is required for the changes to take effect.
Similar to a file system, when data is deleted from the store, the data in the file is not deleted, only the directory information is updated. This means that if a message is consumed, the message data may still be present in the store file, but the directory information that includes this data in the store is updated to reflect the fact that it is deleted.

Temporary store file :
This file contains temporary data that is not retained after the restart of the messaging engine, such as nonpersistent messages that were spilled to the file store to release memory from the JVM heap. The temporary store file contents are truncated when the messaging engine starts.
The temporary store file can be configured to have a maximum and minimum size, or to be unlimited in size. The file can grow from the minimum size (or as required in the unlimited case) but will never shrink (even if the maximum size is set lower than its current size). The file sizes can be changed in the administration console, but a restart of the messaging engine is required for the changes to take effect.
Similar to a file system, when data is deleted from the store, the data in the file is not deleted, only the directory information is updated. This means that if a message is consumed, the message data may still be present in the store file, but the directory information that includes this data in the store is updated to reflect the fact that it is deleted.

You can configure where the file store files must be placed. By default, the file store uses a subdirectory in the following path:${USER_INSTALL_ROOT}/filestores/com.ibm.ws.sib/${ME_NAME}. The file store directory contains two other directories; the log directory that contains the log file and the store directory that contains both the PermanentStore and TemporaryStore files.

Data stores

A data store is a message store that uses a relational database. A messaging engine uses a data store to store operating information in the database, as well as to preserve essential objects that the messaging engine needs for recovery in the event of a failure.

A data store consists of the set of tables that a messaging engine uses to store persistent data in a database. See Data store tables for a list of the tables that comprise a data store. All the tables in a data store are held in the same database schema. You can create multiple data stores in the same database, provided that you use a different schema name for each data store.

The one-to-one relationship between a messaging engine and a data store means that every messaging engine must have its own data store. A messaging engine uses an instance of a JDBC data source to interact with the database that contains the data store for that messaging engine. The relationship between a messaging engine and its data store. illustrates these relationships.

A messaging engine uses a JDBC data source to manage the interaction with a data store, which is held in a database
All the tables in the data store must be stored in the same schema. You can create more than one data store in a database, provided that you use a different schema name for each data store. Although every messaging engine uses the same table names, its relationship with the schema gives each messaging engine exclusive use of its own tables.

Data store topologies :
You have several options for the relative location of a data store and its messaging engine. The topology also defines the relationship of a data store with other data stores.

The following options affect your choice of data store topology:
1. The data store can either run on the same node as its messaging engine or on a remote node.
2. The data store can either have a dedicated database or it can share a database with other data stores.


The contain of this blog is referred from IBM Knowledge Center.

I Just try to make it easy for readers by gathering information under single blog. Hope this will help you.

"Effort only fully releases its reward after a person refuses to quit.”

 Regards,

 Akhilesh B. Humbe