Uninstall WebSphere Application Server from Command line

Hello All,

It is a very common request to uninstall the WebSphere products from machine if you are working as an administrator. If it's a windows environment and you have installation manager console available then it's a very easy task to do. But if it's a UNIX OS and you don't even have a Xmanager to open a graphical user interface the it's critical task to uninstall WebSphere Application server.

In my case I have a Solaris as an OS and didn't have Xmanager available for graphical user interface. Then I use the command line to uninstall WebSphere application Server .  You can do this task in two ways which are described as below.

Procedure:

Uninstall WebSphere Environment Using IBM Installation Manager Command line console.

1. Go to the below location
cd /opt/InstallationManager/eclipse/tools

2. Execute the below command and you will get the command line console
./imcl -c

=====> IBM Installation Manager

Select:
      1. Install - Install software packages
      2. Update - Find and install updates and fixes to installed software packages
      3. Modify - Change installed software packages
      4. Roll Back - Revert to an earlier version of installed software packages
      5. Uninstall - Remove installed software packages

Other Options:
      L. View Logs
      S. View Installation History
      V. View Installed Packages
         ------------------------
      P. Preferences
         ------------------------
      E. Export Data for Problem Analysis
      A. About IBM Installation Manager
         ------------------------
      X. Exit Installation Manager

3. Type 5 to uninstall the installed software packages and press Enter.

4. Similarly by following the instructions given on the command line console you can uninstall all installed packages and Installation Manager also.

Uninstall WebSphere Environment Using IBM Installation Manager's ./imcl uninstall utility.

1. Go to the below location
cd /opt/InstallationManager/eclipse/tools/

2. Execute the below command to list down all installed packages.

./imcl listInstalledPackages

com.ibm.cic.agent_1.5.2000.20120223_0907
com.ibm.websphere.IBMJAVA.v70_7.0.1000.20120424_1539
com.ibm.websphere.ND.v85_8.5.2.20130327_1831
com.ibm.websphere.PLG.v85_8.5.1.20121017_1724

3. Execute the below commands to uninstall all installed packages including Installation manager if you want
[Note: Follow the below sequence for error free uninstall ]

bash-3.2$ ./imcl uninstall com.ibm.websphere.IBMJAVA.v70_7.0.1000.20120424_1539
Uninstalled com.ibm.websphere.IBMJAVA.v70_7.0.1000.20120424_1539 from the /opt/WebSphere85 directory.

bash-3.2$ ./imcl uninstall com.ibm.websphere.PLG.v85_8.5.1.20121017_1724
Uninstalled com.ibm.websphere.PLG.v85_8.5.1.20121017_1724 from the /opt/wasplugins directory.

bash-3.2$ ./imcl uninstall com.ibm.websphere.ND.v85_8.5.2.20130327_1831
Uninstalled com.ibm.websphere.PLG.v85_8.5.1.20121017_1724 from the /opt/WebSphere85 directory.

bash-3.2$ ./imcl uninstall com.ibm.cic.agent_1.5.2000.20120223_0907
Uninstalled com.ibm.websphere.IBMJAVA.v70_7.0.1000.20120424_1539 from the
/opt/InstallationManager directory.


Hope this will help you. Kindly comment for your suggestion and quires.  

"Effort only fully releases its reward after a person refuses to quit.”

 Regards,
 Akhilesh B. Humbe

Caused by: com.ibm.wsspi.management.bla.op.OpExecutionException: CWWMH0121E: Could not read configuration data for composition unit

Hello Everyone

We always think that installation/UnInstalltion and deletion of Websphere Application server  is easy task. It's just a thing like click on next button for some time and you are done with the work. But it is always not easy as we think. Today I was trying to delete one JVM from my Websphere Application server v8.5.0.0 ND environment and I was not able to delete it from the console. Then I tried the same thing using wsadmin tool but still getting some errors. When I was checking the logs I found bello errors in SystemOut.log. While resolving issue I came to know very interesting facts which I have noted down below.

Error :

Caused by: com.ibm.wsspi.management.bla.op.OpExecutionException: CWWMH0121E: Could not read configuration data for composition unit "WebSphere:cuname=<Application Name>,cuedition=BASE". Error data: com.ibm.wsspi.management.bla.op.OpExecutionException: Enterprise application data for composition unit "WebSphere:cuname=<Application_Name>,cuedition=BASE" could not be read. at com.ibm.ws.management.bla.model.CompositionUnitConfig.read(CompositionUnitConfig.java:293)

Cause: 

In my case I have deleted all the applications installed on the JVM which I want to delete and then try to delete that JVM. But as you can seen in above error it was saying that some application are still pointed to the JVMs which are not uninstall. But I never install these applications on my JVM may be did installed on some other JVMs in same domain but before some time, but currently these applications are not installed on any of the JVM in my domain. And when I search for the reason I came to know that when I install an application there are some folders in configuration which keep a track of these folders. These folders are as below.

WAS_INSTALL/profiles/ProfileName/installedApp/Cell_Name/
WAS_INSTALL/profiles/ProfileName/config/cells/Cell_Name/applications
WAS_INSTALL/profiles/ProfileName/config/cells/Cell_Name/applications/blas 
WAS_INSTALL/profiles/ProfileName/config/cells/Cell_Name/applications/cus 

When you delete the application the entries of application should be remove from the all these folders otherwise it will cause the error which we are facing. Most of the times the application entries are there in /blas and /cus folders.

We can solve this issue using below solution.

Resolution 

Perform the following procedure

1. Login to the WebSphere console and list down all the applications with exact version information which are currently install .
2. Stop the DeploymentManager/Adminagent
3. Go to the below listed folders location and delete all the listed application directories other than you noted down from admin console. WAS_INSTALL/profiles/ProfileName/installedApp/Cell_Name/ WAS_INSTALL/profiles/ProfileName/config/cells/Cell_Name/applications WAS_INSTALL/profiles/ProfileName/config/cells/Cell_Name/applications/blas  WAS_INSTALL/profiles/ProfileName/config/cells/Cell_Name/applications/cus 
4. Stop the DeploymentManager/Adminagent
5. Log back into the Admin Console
6. Perform a 'Full Resynchronize' operation on all nodes using the Admin Console
7. After re-sync fully completes for all nodes, attempt to delete the JVM/Cluster again using the Administrative Console, or the wsadmin utility

Hope this will help you. Kindly comment for your suggestion and quires.  

"Effort only fully releases its reward after a person refuses to quit.”

 Regards,
 Akhilesh B. Humbe

Step by Step Installation Of WebSphere Application Server

Today here we are going to discuss about the installation of WebSphere Application Server. Which is the very basic thing we need to know before learning to administrate any kind of technology. Here I have the trail version of WebSphere Application Server v8.5. You can also download this version from the Websphere Site

This one is just the trial version you can install any of  the WebSphere Application Server (Base/ND) product using these steps. For Installation in WebSphere Application Server v8.0 and onward we require a tool called Installation Manager. So here we divide the installation process in four sub processes and we will learn about the each process very briefly.

Process for Installing WebSphere Application Server.

For Installing WebSphere Application Server v8.0 and onwards we need to follow the below steps.

Step 1 : Installation of IMB Installation Manager .
Step 2 : Installation of WebSphere Application Server.
Step 3 : Creating Profile Definition. 
Step 4 : First Step towards WebSphere Application Server. 

Here we have windows environment. But the procedure is similar for all operating systems.

"Effort only fully releases its reward after a person refuses to quit.”

Regards,
Akhilesh B. Humbe

Installation of WebSphere Application Server.

 Step 1 : Installation of IMB Installation Manager .

Step 2 : Installation of WebSphere Application Server.

Installation of WebSphere Application Server using IBM Installation Manager is very user friendly and easy task. Here we are going to install the trail base version 8.5 using IBM Installation Manager. Below are the steps to install WebSphere Application Server trail base version.

Download the dump for WebSphere Application Server and  extract the .zip files. There are three .zip files for this installation.

Start IBM Installation Manager from start menu in windows. Or else you can start it using IBMIM command from location C:\Program Files\IBM\Installation Manager\eclipse . Here it's already started from Step 1

Before installation you need to register the product. For that click on the file menu in left corner of screen and click on preferences.

Click on Add Repository and  the  brows the location of repository.xml from disc1.

Click on apply and ok.

Then click on install in main screen and you will get the following window. Which will list you the products which you have registered. You can select one of then or all of then to install. Here we are selecting WebSphere Application Server trail v8.5.5.0 to install.

Accept the terms in license agreement and click on Next.

Select the installation Directory, you can keep it as default or can change according to you.

Select the language and click on next.

Select the products which you want to install. You can install the sample application for testing purpose.

Brows the Disc2 and then Disc3 location as asked and click on ok.

Review the summary information and click on install and I will start the installation.

After installation you can see the below screen  click on finish to start the profile management tool and start the profile creation.

Step 3 : Creating Profile Definition. 

Creating Profile Definition

Step 1 : Installation of IMB Installation Manager.
Step 2 : Installation of WebSphere Application Server.

Step 3 : Creating Profile Definition . 

After Installation profile creation is the step where we defile the definition for application environment. There are different profiles with there different significance.  Here we are creation stand alone application server profile as we have a base version installed. For managed profiles you can install the ND setup.
To create the profile definition you can follow the below steps.

Start the profile management tool from start menu in windows. Or else you can start it using command "C:\Program Files (x86)\IBM\WebSphere\AppServer\bin\ProfileManagement\pmt.bat".  Here it's already started from Step 2

Click on create for new profile creation.

Select the type of profile you want to create. Here we are creating Application Server Profile. Click on next.

In this window you can select any of the type, but when you use Advanced Profile Creation it allow you to change the default things. Here we are selecting Advanced Profile Creation.

Click on next in Optional Application Deployment window.

Here you can set the profile name and the profile directory path. Here we are using default one. Click on Next.

Now you can set Node name, Host name and server/jvm name. Here we are using default one. Click on Next.

In Administrative Security page you can enable or disable the security. And specify the user name and password for secure access. Click on Next.

Here you can configure SSL certificate with your profile. You can create one or import one. Here we are creating default one. Click on Next.

In this window you can see the default certificate details. Click on Next.

Here in this window you can check and change the port assign for profile.

This window will ask you that if you want to run the JVM as windows service.

If you have webserver already installed, here you can configure the webserver to application server. We don’t have webserver installed so we ignore this click on Next.

Check the profile creation summary and clock on Finish.

It will start the profile creation process.

Once you done with the profile creation process click on finish the first step console.

Step 4 : First Step towards WebSphere Application Server 

First Step towards WebSphere Application Server

Step 1 : Installation of IMB Installation Manager.
Step 2 : Installation of WebSphere Application Server.
Step 3 : Creating Profile Definition.

Step 4 : First Step towards WebSphere Application Server. 

First Step Console allow you to perform the very basic admin activities on newly created profile.

You can open the first step console from start menu in windows. Or else you can start it using command "C:\Program Files (x86)\IBM\WebSphere\AppServer\profiles\AppSrv01\firststeps\firststeps.bat". Here it's already started from Step 3

After profile creation and installation click on installation verification. It will verify your installation and start the default JVM.

From first step console you can stop the jvm and redirect to admin console.

It will brows you the admin console link and using username and password you can login to the Administration console.

Here we done with the installation of WebSphere Application Server

Hope this will help you. Kindly comment for your suggestion and quires.  

"Effort only fully releases its reward after a person refuses to quit.”

 Regards,
 Akhilesh B. Humbe

Installation of IMB Installation Manager

Step 1 : Installation of IMB Installation Manager

IBM installation Manager is the product/utility used for installation of WebSphere products. We are using this utility from WebSphere Application Server v8.0 and onwards. Installation Manager allow you to install/uninstall/update the products. Below are the steps to install IBM Installation Manager.

Download the IBM Installation Manager suitable for your operating system and extract the .zip file.

Run the file Install and it's good if you run it as an administrator.

You can see, it will start installing IBM Installation Manager.

You can see the version of Installation Manager in install package page. Click on Next to proceed.

Accept the terms in license agreement and click on Next.

Select the installation Directory, you can keep it as default or can change according to you.

Verify the summary and click on Install.

It will Initiate the installation process.

Once you get finish you will get the following page window.

Click on Restart Installation Manager to start the installation manager tool.

Step 2 : Installation of WebSphere Application Server 

SSLHandshakeE E SSLC0008E: Unable to initialize SSL connection

Hello Everyone,

Hope you all aware of SSL (Secure Socket Listener) . We normally face lot of issues during SSL configuration and after configuration while handshake. We have discussed many of them before also. Here today I was getting another one while handshake between client and server.

Error:

[4/10/15 10:44:20:024 EDT] 00000067 SSLHandshakeE E   SSLC0008E: Unable to initialize SSL connection.  Unauthorized access was denied or security settings have expired.  Exception is javax.net.ssl.SSLHandshakeException: Client requested protocol TLSv1 not enabled or not supported
        at com.ibm.jsse2.K.B(K.java:141)
        at com.ibm.jsse2.SSLEngineImpl.b(SSLEngineImpl.java:106)
        at com.ibm.jsse2.SSLEngineImpl.c(SSLEngineImpl.java:184)
        at com.ibm.jsse2.SSLEngineImpl.wrap(SSLEngineImpl.java:582)
        at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:16)
        at com.ibm.ws.ssl.channel.impl.SSLUtils.handleHandshake(SSLUtils.java:746)
        at com.ibm.ws.ssl.channel.impl.SSLConnectionLink.readyInbound(SSLConnectionLink.java:566)
        at com.ibm.ws.ssl.channel.impl.SSLConnectionLink.ready(SSLConnectionLink.java:295)
        at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)
        at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)
        at com.ibm.ws.tcp.channel.impl.WorkQueueManager.requestComplete(WorkQueueManager.java:558)
        at com.ibm.ws.tcp.channel.impl.WorkQueueManager.attemptIO(WorkQueueManager.java:608)
        at com.ibm.ws.tcp.channel.impl.WorkQueueManager.workerRun(WorkQueueManager.java:985)
        at com.ibm.ws.tcp.channel.impl.WorkQueueManager$Worker.run(WorkQueueManager.java:1074)
        at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1881)
Caused by: javax.net.ssl.SSLHandshakeException: Client requested protocol TLSv1 not enabled or not supported

Cause :

While Searching for this error I came across very important and helpful information. Here we consider that you all aware of the  SSL Handshake term or we will discuss it in brief later. In SSL hand shake the two communicators i.e. client and server initiate connection with each other to perform the transaction. For handshake we are having a different protocols based upon client and server. There is condition that for handshake need the communication protocols on both side should be compatible. Here in case WebSphere application server is our server and the requesting client may be different. It may be Webserver or your java application or you web browser or any internal service.

Here consider in my case the client which is asking foe handshake is a java application. Almost all HTTPS requesting traffic from application is Java generated. So the protocol requested by my java application is not enabled on server side so it's not supporting the handshake.

Let's check the below table for more clarification.

Here in my case the application is running using the java 6 means it will support the minimum SSLv3** and Maximum TSLv1 Version. Here in case if I disable the SSLv3** support from my server side the my application will use the higher version of SSL to handshake and will establish the connection for transaction. But I disable the higher version i.e. TSLv1 then it is not possible to establish the hand shake. So it will show the error Client requested protocol TLSv1 not enabled or not supported. Same thin will happen vise versa .

Solution:

In such case you can go and check on Websphere application server console which version of SSL handshake protocol your server is supporting. For that go to the
SSL certificate and key management > SSL configurations > NodeDefaultSSLSettings (your Custom setting ) > Quality of protection (QoP) settings and check. Here in my case it was showing TLSv1.2 which is not supported by a client.

So here I go and set the protocol to TLSv1, which one client asking for. And now it's working fine.

For this click on the drop box, select the suitable protocol, Apply the change and don't forget to save it.


Hope this will help you. Kindly comment for your suggestion and quires.  

"Effort only fully releases its reward after a person refuses to quit.”

 Regards,
 Akhilesh B. Humbe

referenced in VMStructs::localHotSpotVMStructs in the remote VM, was not present in the remote VMStructs

Hello Everyone,

Hope you all are doing well. Today here we are discussing about taking the heapdump using a jmap command. Here I have one command which successfully generate the heapdump for your running java process. While executing this command I had faced one error, which is very command and I think need to discuss with you people.

How to generate a heap dump using java process id (pid) :

The jmap is a very good utility to generate a heapdump. Using below command you can generate the heapdump for running java process

Command : jmap -dump:format=b,file=TestFile.hprof PID

Where TestFile.hprof is the dump file name you can change it accordingly.
And PID is process id of running java process for which you want to generate the heapdump

But sometime while executing this command we get the below error:

Error :

Attaching to process ID XXXX, please wait...
Exception in thread "main" java.lang.RuntimeException: Type "nmethodBucket*", referenced in VMStructs::localHotSpotVMStructs in the remote VM, was not present in the remote VMStructs::localHotSpotVMTypes table (should have been caught in the debug build of that VM). Can not continue.
        at sun.jvm.hotspot.HotSpotTypeDataBase.lookupOrFail(HotSpotTypeDataBase.java:345)
        at sun.jvm.hotspot.HotSpotTypeDataBase.readVMStructs(HotSpotTypeDataBase.java:236)
        at sun.jvm.hotspot.HotSpotTypeDataBase.<init>(HotSpotTypeDataBase.java:70)
        at sun.jvm.hotspot.bugspot.BugSpotAgent.setupVM(BugSpotAgent.java:544)
        at sun.jvm.hotspot.bugspot.BugSpotAgent.go(BugSpotAgent.java:476)
        at sun.jvm.hotspot.bugspot.BugSpotAgent.attach(BugSpotAgent.java:314)
        at sun.jvm.hotspot.tools.Tool.start(Tool.java:146)
        at sun.jvm.hotspot.tools.JMap.main(JMap.java:126)

Cause:

I was getting this error because my path to jmap wasn't the same as the path to the java process i.e. targeting two different versions. In such a case you need to set a proper path before executing the jmap command or execute command with full path of the java you are currently using.

i.e.  /opt/WebSphere85/java_1.7_64/bin/jmap -dump:format=b,file=TestFile.hprof 27693.

where /opt/WebSphere85/java_1.7_64 is my current version of java for webSphere server.

This command will generate you heapdump in your current working directory.

Hope this will help you. Kindly comment for your suggestion and quires.  

"Effort only fully releases its reward after a person refuses to quit.”

 Regards,
 Akhilesh B. Humbe

java.lang.OutOfMemoryError: PermGen space

Hello Everyone,

Today one of WAS 8.0.0.5 Server get crashed and was showing the below error in SystemOut.log

Error:

[04/03/15 10:27:47:004 GMT] 00013e2b servlet       E com.ibm.ws.webcontainer.servlet.ServletWrapper service SRVE0014E: Uncaught service() exception root caus
e spring-ws: org.springframework.web.util.NestedServletException: Request processing failed; nested exception is java.lang.OutOfMemoryError: PermGen space

Cause:

As it's a java application running of WebSphere Application server it allocate some memory to run. The size of memory is split into different areas including PermGen Space. The size of all these areas set during the JVM startup. If you do not set the sizes yourself, platform specific defaults will be used. So  the “java.lang.OutOfMemoryError: PermGen space” message indicates that the Permanent Size area in memory is exhausted. And there are lot of reasons for this.

All Java classes are loaded and stored in the Java PermGen.All of these are allocated to PermGen and stay in PermGen.As you can see, the PermGen size requirements depend both upon the number of classes loaded as well as the size of such class declarations. So it is easy to see the main cause for the “java.lang.OutOfMemoryError: PermGen space“: either too many classes or too big classes are being loaded to the permanent generation.

Solution:

The first solution to the OutOfMemoryError due to PermGen should be obvious. If we have exhausted the PermGen area in the memory we need to increase its size. This solution is indeed helpful if you just have not given your JVM enough below room. So alter your application launch configuration and add (or increase if present) the following:
-XX:MaxPermSize=512m (Suitable Size)
This configuration parameter is indicating to the JVM that PermGen is allowed to grow up to 512 MB before complaining in the form of OutOfMemoryError.

Second Solution  is you can take the heap dump at the time of issue and analyze it to find which objects causes the OutOfMemoryError in PermGen space

Third possibility is to allow GC to unload classes from PermGen. The standard JVM is rather conservative in this regard  Classes are born to live forever. So once loaded, classes stay in memory even if no one is really using them anymore. This can become a problem when the application is creating lots of classes dynamically and the generated classes are not needed for longer periods. In such a case, allowing JVM to unload class definitions can be helpful. This is achieved by adding again just one configuration parameter to your startup scripts:
-XX:+CMSClassUnloadingEnabled
By default this is set to false and so to enable this you need to explicitly set the following option in Java options. If you enable CMSClassUnloadingEnabled, GC will sweep PermGen too and remove classes which are no longer used. Keep in mind that this option will work only when UseConcMarkSweepGC is also enabled using the below option. So when running parallel or, God forbid, serial GCs, make sure you have set your GC to CMS by specifying:
-XX:+UseConcMarkSweepGC
But before calling it a night, be warned  more often than not usage of the recommended “quick fixes” means you are just masking the symptoms by hiding “java.lang.OutOfMemoryError: Permgen space” and are not tackling the underlying problem.

I personally recommend you to use the 1st and 2nd solution. And hope it will help you to resolve the issue.

"Effort only fully releases its reward after a person refuses to quit.”

 Regards,
 Akhilesh B. Humbe

HMGR0152W: CPU Starvation detected. Current thread scheduling delay is 65 seconds.

Hello Everyone,

Today discussing new issue. I was getting below error on WAS v6.1 environment. When I search for the error I got a very good note which I am sharing with you and adding some clarification.

Error :

HMGR0152W: CPU Starvation detected. Current thread scheduling delay is 65 seconds.
[3/24/15 12:01:10:162 EDT] 00000053 DiscoveryRcv  W   DCSV1115W: DCS Stack DefaultCoreGroup at MemberTestCell01\TestNode01\app_TestNode01_1: MemberTestCell01\TestNode01\nodeagent connection  was closed. Member will  be removed from view. DCS connection status is Discovery|Ptp, receiver closed.

Cause :

The HMGR0152W message indicate that JVM thread scheduling delays are occurring for this process.

The WebSphere Application Server high availability manager component contains thread scheduling delay detection logic, that periodically schedules a thread to run and tracks whether the thread was dispatched and run as scheduled. By default, a delay detection thread is scheduled to run every 30 seconds, and will log a HMGR0152W message if it is not run within 5 seconds of the expected schedule. The message will indicate the delay time or time differential between when the thread was expected to get the CPU, and when the thread actually got CPU cycles.

The HMGR0152W message can occur even when plenty of CPU resource is available. There are a number of reasons why the scheduled thread might not have been able to get the CPU in a timely fashion. Some common causes include the following

• The physical memory is overcommitted and paging is occurring.
• The heap size for the process is too small causing garbage collection to run too frequently and/or too long, blocking execution of other threads.
• There might simply be too many threads running in the system, and too much load placed on the machine, which might be indicated by high CPU utilization.

Note : If you have an admin agent in your environment. Then the JVM get restarted automatically.

Solution :

The HMGR0152W message is attempting to warn you that a condition is occurring that might lead to instability if it is not corrected. Analysis should be performed to understand why the thread scheduling delays are occurring, and what action(s) should be taken. Some common solutions include the following:

• Adding more physical memory to prevent paging.
• Tuning the JVM memory (heap size) for optimal garbage collection.
• Reducing the overall system load to an acceptable value.

If the HMGR0152W messages do not occur very often, and indicate that the thread scheduling delay is relatively short (for example, < 20 seconds), it is likely that no other errors will occur and the message can safely be ignored.

The high availability manager thread scheduling delay detection is configurable by setting either of the following 2 custom properties.

• IBM_CS_THREAD_SCHED_DETECT_PERIOD determines how often a delay detection thread is scheduled to run. The default value of this parameter is 30 (seconds).
• IBM_CS_THREAD_SCHED_DETECT_ERROR determines how long of a delay should be tolerated before a warning message is logged. By default this value is 5 (seconds).

These properties are scoped to a core group and can be configured as follows:

1. In the administrative console, click Servers > Core groups > Core groups settings and then select the core group name. 
2. Under Additional Properties, click Custom properties > New.
3. Enter the property name and desired value.
4. Save the changes.
5. Restart the server for these changes to take effect.

While it is possible to use the custom properties mentioned above to increase the thread-scheduling-detect-period until the HMGR0152W warning messages no longer occur, this is not recommended. The proper solution is to tune the system to eliminate the thread scheduling delays.

Hope this will work in your case also. Kindly comment for your suggestion and quires.  

"Effort only fully releases its reward after a person refuses to quit.”

 Regards,
 Akhilesh B. Humbe

DCSV1115W: DCS connection status is View|Ptp, receiver closed.

Hello Everyone,

Hope you are enjoying working with middleware. Getting error at the startup of the JVMs is a very common thing and I think we learn a lot from this. I was getting this error while starting one of my JVM/Server on WAS v8.5.0.0 environment.

Error:

W   DCSV1115W: DCS Stack DefaultCoreGroup at Member TestCell01\TestNode01\app_TestNode01_1: Member TestCell01\TestCellManager01\dmgr connection was closed. Member will  be removed from view. DCS connection status is View|Ptp, receiver closed.

Cause:

This is the general error that might be encountered during server start phase. While starting the server it's get hung and not roll any type of logs.Basic idea behind this is that, when you start the server, threads are getting initialized for your process/job that you want to run on server. That thread is waiting for few resources which helps them to run the process/job. But at that point of time thread may get hung, because of un-availability of resources.

Solution:

Way to fix it is  Kill the process of server from background. Again start the server.

1. In UNIX o/s you can find out the process and kill it by.

$ ps -ef | grep JVM_Name or 

$ ps -ef | grep java

$kill -9 process_id

2. In WINDOWS environment  you can find out the process and kill it by.

C:\>tasklist |findstr java

C:\>taskkill /F -PID process_id

3. Then start the JVM/Server using startServer.sh or startServer.cmd

Hope this will work in your case also. Kindly comment for your suggestion and quires.    

"Effort only fully releases its reward after a person refuses to quit.”

 Regards,

 Akhilesh B. Humbe

WASX7111E: Cannot find a match for supplied option

Hello Everyone.

Hope you are enjoying working with middleware. Today it was an interesting  error while deploying an application. It was .ear file and I was trying to deploy it on WAS 8.5.0.0. I have a script to deploy application which end with the error given below. Then I try to deploy it using console and got a same error.

Error: 

WASX7017E: Exception received while running file "/opt/scripts/deploy/was/install.py"; exception information:
com.ibm.ws.scripting.ScriptingException: WASX7111E: Cannot find a match for supplied option: "[Testapplication.war,
Testapplication.war,WEB-INF/web.xml,
WebSphere:cell=TestNode01Cell,node=TestNode01,server=Server1]"
 for task "MapModulesToServers". The supplied option must match with the existing task data in the application and the existing task data are:
"["Archetype Created Web Application" Testapplication.war,WEB-INF/web.xml]

It's an application error. But as middleware consultant we always need to find out the reason for an error to explain it from our side. When I search for the error I found some interesting facts.  Here  I am sharing cause and the solution for the error, which you can suggest to an application team. And if you are application team you can implement it.

Cause:

While installing application on WebSphere application server, it try to map application modules to target JVM. While mapping modules it first look into the web.xml for the module name. which we defines into the tab  <display-name></display-name>. If this tab is not present in web.xml then it will display the whole .war file name as module name. The error normally prompt when you have an incorrect entry in <display-name></display-name> tab or have any extra space or any case sensitivity issue.

In my case the application name was Testapplication.war and in display tab it was showing like <display-name>Archetype Created Web Application</display-name>
In such a case you can contact to your application development team to take an action or have a temporary solution given below

Solution:

You can modify the web.xml file and could just comment the entire <display-name> element out or remove it. It will successfully deploy your application.

Concern a application development team before doing this.

Hope this will work in your case also. Kindly comment for your suggestion and quires.  

"Effort only fully releases its reward after a person refuses to quit.”

 Regards,
 Akhilesh B. Humbe

Error cleaning old files: java.lang.NullPointerException

Hello Everyone.

Today going to discuss about the below error.

Error:

-bash-3.2$ /opt/WebSphere85/profiles/Agent01/bin/stopServer.sh adminagent
Password:
ADMU0116I: Tool information is being logged in file
           /opt/WebSphere85/profiles/Agent01/logs/adminagent/stopServer.log
ADMU0128I: Starting tool with the Agent01 profile
ADMU3100I: Reading configuration for server: adminagent
Error cleaning old files: java.lang.NullPointerException
com.ibm.rmi.ras.Utility.newWriter: could not write to orbmsg.12032015.1310.34.txt : java.io.FileNotFoundException: /home/XXX/orbmsg.12032015.1310.34.txt (Permission denied) 
com.ibm.rmi.ras.Utility.newWriter: could not write to orbtrc.12032015.1310.34.txt : java.io.FileNotFoundException: /home/XXX/orbtrc.12032015.1310.34.txt (Permission denied)


The error which is highlighted above is very common error we normally face while start up or stop of WebSphere application server, I am facing this error while stopping the adminagent.

As it's showing permission denied it means. I don't have permission on this file which needs to clean while stop. There may the following reasons of this thing

Cause:

1. We are trying to stop the adminagent using other user than WAS, which does not have a permission to read/write this file.
2. Some times using was user also getting same error. Means in this case you have started/stopped the adminagent [JVM] using root user previously, which changed the permissions of some files to root user and corresponding group. And now not allow WAS user to clean or replace these files.

Solution:

There is very simple solution for that.

1. Goto the  /opt/WebSphere85/profiles/ location and change the owner of the profile to was:was

2. In my case it is.

   /opt/WebSphere85/profiles/> chown -R  was:was Agent01
   Which will change the owner[user/group] of all file in Agent01 profile as was.

Hope this will work in your case also. Kindly comment for your suggestion and quires.  

"Effort only fully releases its reward after a person refuses to quit.”

 Regards,
 Akhilesh B. Humbe

1 Managed connections are already being used on this thread.

Hello everyone,

Today it was the below error in my WAS 7.0.017 ND environment..

[3/10/15 16:34:13:185 EDT] 000000cb PoolManager   W   Exceeded the number of allowable managed connection on thread 000000cb.  1 managed connections are already being used on this thread.    Managed connection being used on this thread 
MCWrapper id 1462303  Managed connection org.apache.activemq.ra.ActiveMQManagedConnection@1a4bf8a State:STATE_TRAN_WRAPPER_INUSE Thread Id: 000000cb Thread Name: WebContainer : 0 Handle count 0
     Start time inuse Tue Mar 10 16:34:13 EDT 2015 Time inuse 0 (seconds)
     Last allocation time Tue Mar 10 16:34:13 EDT 2015
     getConnection stack trace information:   org.apache.activemq.ra.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:67)
          org.springframework.jms.support.JmsAccessor.createConnection(JmsAccessor.java:184)
          org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:456)
          org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:534)
          org.springframework.jms.core.JmsTemplate.convertAndSend(JmsTemplate.java:612)

This as unexpected error occurred in my  SystemOut.log and at the same time my application performance was on it's worst. When I look for the error on google I found the one solution frequently for this  error and logically I think this is correct. I didn't implement this solution because in my case it resolved by the different thing. But here I am going to share the both things with you. The first one which I applied and second may also helps you.

1. Change Log Level:

The solution in my case was that I had to change/reduce the trace level of the AppServer back to *info from *finest , as the server was too busy writing logs besides serving a database connection request.

2. Increase maxNumberOfMCsAllowableInThread parameter for DataSource:

As in error it is showing that 1 managed connections are already being used on this thread means there is only one managed connection is allowed on each thread and it's in used now. But there may be scenario when the application is designed such that web layer (controller) makes database calls (read only) in addition to database calls (Read/Write) at the service layer in such a scenario we would assume we would have more than one connection involved per thread. So then the question is, is it possible to configure the WAS to increase the allowed size of managed connection to more than 1. And the answer is yes we can increase it using the below steps

Procedure:

Go to Data sources > DataSource_Name >Connection pools > Custom properties > New Create the new custom pool property  maxNumberOfMCsAllowableInThread setting it to the value you wish, In our case we set it to the 10.

Click on Apply and Save.

This worked in my case, hope will work for you also. Kindly comment for your suggestion and quires.  

"Effort only fully releases its reward after a person refuses to quit.”

 Regards,
 Akhilesh B. Humbe

ADMU3011E: Server launched but failed initialization

Hello Everyone,

Today I was facing this issue on one of my environment of Websphere Application server 8.5 ND. While try to start the server  I was facing this issue and not allow me to start the server/JVM. And showed the following error when execute ./startServer.sh  

ADMU3011E: Server launched but failed initialization. Server logs, startServer.log, and other log files under /usr/IBM/WebSphere85/profiles/AppSrv01/logs/server1 should contain failure information.

After net surfing I got the following solution.

It is not easy to conclude the reason of issue by the error message which shown above. There may a different reason for that issue like JVM/Heap Size, file permission etc., so for digging exact issue you need to check the JVM logs, startup logs or ffdc logs.

In my case it was not writing SystemOut.log and SystemErr.log so I finally got a detailes of error in startServer.log.

Error:

javax.naming.ServiceUnavailableException : A communication failure occurred while attempting to obtain an initial context with the provider URL: "iiop://localhost:2811/". Make sure that any bootstrap address information in the URL is correct and that the target name server is running. A bootstrap address with no port specification defaults to port 2809. Possible causes other than an incorrect bootstrap address or unavailable name server include the network environment and workstation network configuration.

Solution:

In my case I resolve the error by using following steps:

1. Login to the websphere admin console.
2. Go to the Application servers > Server_Name> Process definition > Java Virtual Machine

1. When I check Boot classpath and Generic JVM arguments I found some syntax error is there.
2. I repair the errors and take the restart of JVM/Server and it works.

Hope this will work in your case also. Kindly comment for your suggestion and quires.

 "Effort only fully releases its reward after a person refuses to quit.” 

 Regards, 
 Akhilesh B. Humbe