ERROR: The Installation Manager cannot be started.The registry information does not exist or does not match with this executable.

Hello Everyone,

Hope you all are enjoying working with middleware and doing well. Today's issue is bit different. When I was trying to install fixpack on one of my environment of WAS 8.5.5.2 I was getting the below error. The error was actually it's not allowing me to start the IBM installation manager to install the fixpack. Means this error is not related to the fix pack installation. We are getting this issue because improper installation of IBM installationmanager. It will not allow you to even execute the command ./imcl. When I was trying to execute just ./imcl command, getting below error. 

Error:

.............................................................

ADMU5002I: 1,160 files successfully backed up

ERROR: The Installation Manager cannot be started. The registry information does not exist or does not match with this executable. This may happen if you are trying to run Installation Manager installed by another user.

00:01.80 ERROR [main] com.ibm.cic.agent.internal.application.HeadlessApplication run

The Installation Manager cannot be started. The registry information does not exist or does not match with this executable. This may happen if you are trying to run Installation Manager installed by another user.

ADMU0116I: Tool information is being logged in file

Cause:

In Error itself it's showing  that the Installation Manager cannot be started because you may trying to run Installation Manager installed by another user. We had installed the Webpshere environment using wsadmin user and then after according to the security standard of our organization we changed user to was. It was working fine for all application server/JVMs and it started all the JVMs. But when I try to start installation manager it's showing an error because when Installation manager was installed by user wsadmin it's registry file i.e. InstallationManager.dat got stored in /export/localhome/wsadmin/etc/.ibm/registry location, which is the user directory of previous user.

To resolve the issue I need to copy the registry file i.e. InstallationManager.dat to new user directory location i.e. /export/localhome/was/etc/.ibm/registry.

Procedure: 

Copy the registry file i.e. InstallationManager.dat from user directory of previous user to the user directory of new user

Hope this will help you. Kindly comment for your suggestion and quires.   

"Effort only fully releases its reward after a person refuses to quit.”

 Regards,

 Akhilesh B. Humbe

Caused by: org.omg.CORBA.INTERNAL: CREATE_LISTENER_FAILED_4 vmcid: 0x49421000 minor code: 56 completed: No

Hello Everyone,

Hope you all are enjoying working with the Middleware .Today I am getting error while starting the JVM in WebSphere Application Server environment v8.5.0.2. and it's not getting started. When I checked the SystemOut.log found the below error description.

Error:

[11/8/16 0:11:44:222 EST] 00000001 WsServerImpl  E   WSVR0009E: Error occurred during startup

com.ibm.ws.exception.RuntimeError: org.omg.CORBA.INTERNAL: CREATE_LISTENER_FAILED_4  vmcid: 0x49421000  minor code: 56  completed: No

        at com.ibm.ws.runtime.component.ORBImpl.start(ORBImpl.java:490)

        at com.ibm.ws.runtime.component.ContainerHelper.startComponents(ContainerHelper.java:540)

        at com.ibm.ws.runtime.component.ContainerImpl.startComponents(ContainerImpl.java:627)

        at com.ibm.ws.runtime.component.ContainerImpl.start(ContainerImpl.java:618)

        at com.ibm.ws.runtime.component.ServerImpl.start(ServerImpl.java:540)

        at com.ibm.ws.runtime.WsServerImpl.bootServerContainer(WsServerImpl.java:311)

        at com.ibm.ws.runtime.WsServerImpl.start(WsServerImpl.java:224)

        at com.ibm.ws.runtime.WsServerImpl.main(WsServerImpl.java:697)

        at com.ibm.ws.runtime.WsServer.main(WsServer.java:59)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)

        at java.lang.reflect.Method.invoke(Method.java:613)

        at com.ibm.wsspi.bootstrap.WSLauncher.launchMain(WSLauncher.java:234)

        at com.ibm.wsspi.bootstrap.WSLauncher.main(WSLauncher.java:96)

        at com.ibm.wsspi.bootstrap.WSLauncher.run(WSLauncher.java:77)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)

        at java.lang.reflect.Method.invoke(Method.java:613)

        at org.eclipse.equinox.internal.app.EclipseAppContainer.callMethodWithException(EclipseAppContainer.java:587)

        at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:198)

        at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)

        at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)

        at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:369)

        at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)

        at java.lang.reflect.Method.invoke(Method.java:613)

        at org.eclipse.core.launcher.Main.invokeFramework(Main.java:340)

        at org.eclipse.core.launcher.Main.basicRun(Main.java:282)

        at org.eclipse.core.launcher.Main.run(Main.java:981)

        at com.ibm.wsspi.bootstrap.WSPreLauncher.launchEclipse(WSPreLauncher.java:401)

        at com.ibm.wsspi.bootstrap.WSPreLauncher.main(WSPreLauncher.java:164)

Caused by: org.omg.CORBA.INTERNAL: CREATE_LISTENER_FAILED_4  vmcid: 0x49421000  minor code: 56  completed: No

        at com.ibm.ws.orbimpl.transport.WSTransport.createListener(WSTransport.java:867)

        at com.ibm.ws.orbimpl.transport.WSTransport.initTransports(WSTransport.java:605)

        at com.ibm.rmi.iiop.TransportManager.initTransports(TransportManager.java:155)

        at com.ibm.rmi.corba.ORB.set_parameters(ORB.java:1274)

        at com.ibm.CORBA.iiop.ORB.set_parameters(ORB.java:1688)

        at org.omg.CORBA.ORB.init(ORB.java:364)

        at com.ibm.ws.orb.GlobalORBFactory.init(GlobalORBFactory.java:92)

        at com.ibm.ejs.oa.EJSORBImpl.initializeORB(EJSORBImpl.java:179)

        at com.ibm.ejs.oa.EJSServerORBImpl.<init>(EJSServerORBImpl.java:102)

        at com.ibm.ejs.oa.EJSORB.init(EJSORB.java:55)

        at com.ibm.ws.runtime.component.ORBImpl.start(ORBImpl.java:482)

        ... 34 more

Couse :

The main reason of this error is port conflict. It means the ports of JVM which you are trying to start are already being used by some other process on servers.

Solution:

First login to the console or from backend list down all the ports which your JVM is listening.

Console: login > Server > ServerName >Ports

Backend:  /IBM/WebSphere85/profiles/AppSrv01/config/cells/CellName/nodes/NodeName/serverindex.xml

Check if the ports are already listening.

netstat  -an | grep PortNo. 

If any port is already listening find out the process running on it.

netstat -tcpl | grep PortNo.

Kill the Process running on the port.

Kill -9 Process Id

Try to start the JVM again and this time it will start the JVM.

Hope this will help you. Kindly comment for your suggestion and quires.   

"Effort only fully releases its reward after a person refuses to quit.”

 Regards,

 Akhilesh B. Humbe

Log Purging in UNIX operating system.

Hello Every one,

Hope you all are doing well and having fun as Middleware admin. It's been freeze and no more issue today so lets talk about some other activities than the daily issue. As a Middleware consultant we have lot of other task than installation configuration and daily troubleshooting. Here we are discussing easy process of doing log purging.

We can set the log rotation and purging policy from Websphere console for the JVM logs. But some time you need to clean it manfully or need to set the policy according to the organization standers. In my case I need to purge the JVM logs, http logs and application logs also. In our environment we have policy where we archive the files older than 7 days(week) in .gz format and delete the archived files older than 30 day on monthly basis.

If I want to do this task manfully it will take a lot of time and manual efforts which we don't want to do. So here I am noting down some UNIX command which makes my work easy in file purging. Here in my case I am having SOLARIS as an operating system. The command may require some changes in other UNIX operating system or may be not.

Procedure:

1. Archive the files older than 7 days(week)

find . -name '*.log*' -mtime +7 -exec gzip {} \;


2. Delete the archive files older than 30 days.

find . -name '*.gz'  -mtime +30 -exec rm -f {} \;

Hope this will help you. Kindly comment for your suggestion and quires.  

"Effort only fully releases its reward after a person refuses to quit.”

 Regards,
 Akhilesh B. Humbe

SDK Interim Fix Installation for the WebSphere Application server in UNIX environment.

Hello Everyone,

Hope you all are enjoying working with the Middleware. Today's task is to install the interim fix pack on in Websphere environment. We have an example of installing interim fix pack for the SDK on WebSphere Application  Server v7. Same procedure you can use to install the interim fixpack for Application server/plugins and HIS also. Here we are installing interim fix pack from command line as we always don’t have an access to the GUI installation.

Procedure:

1. Before installing interim fixpack must take backup of installed java
    .i.e. /opt/IBM/software/WebSphere/java

2. Check the current java version and make a note of that for the confirmation after install.
     /opt/IBM/software/WebSphere/java/bin/java -version

3. Before installing the fix pack you need to stop all the JVMs running including dmgr, nodeagent and adminagent if it's there.

4. Once all java services get stopped create a response file with the below content.

/opt/IBM/software/WebSphere/interim7/install_interimfix_IFPI28439.txt (your location may be different)

################################################################################
#
# This is the silent install response file for installing maintenance packages
# using the update installer.
#
# A common use of an options file is to run the wizard in silent mode. This lets
# the options file author specify wizard settings without having to run the
# wizard in graphical or console mode. To use this options file for silent mode
# execution, *uncomment* and modify the parameters defined within.
#
# Use the following command line when running the wizard from the update
# installer directory:
#
#    update -options responsefiles/install.txt -silent
#
# Please enclose all values within a single pair of double quotes.
#
################################################################################
-W maintenance.package=/opt/IBM/software/WebSphere/interim7/7.0.0.0-WS-WASJavaSDK-LinuxX32-IFPI28439.pak  (it's a .pak file location and your location may be different)
-W product.location="/opt/IBM/WebSphere7"
-W update.type="install"

5. Go to the update installer home and run the below command.
    ./update.sh -silent -options /export/software/WebSphere/properties/install_interimfix_IFPI28439.txt

Note: always install the fix pack with the same user as product installed.

7. Check the current java version to confirm the install.
    /opt/IBM/software/WebSphere/java/bin/java -version

8. Start all java services as before.

Hope this will help you. Kindly comment for your suggestion and quires.  



"Effort only fully releases its reward after a person refuses to quit.”

 Regards,
 Akhilesh B. Humbe

Disable RC4 cipher suite For Security Vulnerability in IBM HTTP Server

Hello Everyone,

In Last blog we have seen how to disable SSLv3 for POODLE Vulnerability in IIBM HTTP Server and now today's task is disabling RC4 cipher suit in IBM HTTP Server.

The RC4 algorithm, as used in the TLS protocol and SSL protocol, could allow a remote attacker to obtain sensitive information. An attacker could exploit this vulnerability to remotely expose account credentials without requiring an active man-in-the-middle session. This vulnerability is commonly referred to as "Bar Mitzvah Attack"

Resolution:

IBM recommends disabling RC4 in IBM HTTP Server. To disable RC4, complete the steps below

Procedure:

1. First You can ask IHS to print out all its known ciphers with apachectl -t -DDUMP_SSL_CIPHERS, and it will tell you each virtual hosts configuration with apachectl -t -DDUMP_SSL_CONFIG.  For that you need to go to the <WebServer_Home>/bin and run the below command.

cd /opt/IBM/HttpServer/bin
./apachectl -t -D DUMP_SSL_CIPHERS
./apachectl -t -D DUMP_SSL_CONFIG

Output:

SSL server defined at: XXXXXXXXXXXXXXXXXXXXXXXXXX
Server name: XXX
SSL enabled: YES
FIPS enabled: 0
Keyfile: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Protocols enabled: SSLV2,SSLV3,TLSv10,TLSv11,TLSv12
Ciphers for SSLV2: (defaults)
Ciphers for SSLV3: (defaults) TLS_RSA_WITH_AES_128_CBC_SHA(2F),TLS_RSA_WITH_AES_256_CBC_SHA(35b),SSL_RSA_WITH_RC4_128_SHA(35),SSL_RSA_WITH_RC4_128_MD5(34),SSL_RSA_WITH_3DES_EDE_CBC_SHA(3A)
Ciphers for TLSv10: (defaults) TLS_RSA_WITH_AES_128_CBC_SHA(2F),TLS_RSA_WITH_AES_256_CBC_SHA(35b),SSL_RSA_WITH_RC4_128_SHA(35),SSL_RSA_WITH_RC4_128_MD5(34),SSL_RSA_WITH_3DES_EDE_CBC_SHA(3A)
Ciphers for TLSv11: (defaults) TLS_RSA_WITH_AES_128_CBC_SHA(2F),TLS_RSA_WITH_AES_256_CBC_SHA(35b),SSL_RSA_WITH_RC4_128_SHA(35),SSL_RSA_WITH_RC4_128_MD5(34),SSL_RSA_WITH_3DES_EDE_CBC_SHA(3A)
Ciphers for TLSv12: (defaults) TLS_RSA_WITH_AES_128_GCM_SHA256(9C),TLS_RSA_WITH_AES_256_GCM_SHA384(9D),TLS_RSA_WITH_AES_128_CBC_SHA256(3C),TLS_RSA_WITH_AES_256_CBC_SHA256(3D),TLS_RSA_WITH_AES_128_CBC_SHA(2F),TLS_RSA_WITH_AES_256_CBC_SHA(35b),SSL_RSA_WITH_3DES_EDE_CBC_SHA(3A)

2. Find the file where you have a setting for the SSL configuration. In most of the environment it's a httpd.conf but in some cases if you are using different file you can find the file using below command.

grep -i -r "SSLEnable" /opt/IBM/HTTPServer/
/opt/IBM/HTTPServer/ is Webserver home 

3. Take backup of the configuration file e.g. httpd.conf or ssl.conf

4. Edit the configuration file to add below line after the SSLEnable section

SSLCipherSpec ALL -SSL_RSA_WITH_RC4_128_SHA -SSL_RSA_WITH_RC4_128_MD5

5. It will look like below stanza

<VirtualHost *:${HTTPD_PORT}>
SSLEnable
SSLProtocolDisable SSLv3 SSLv2
SSLCipherSpec ALL -SSL_RSA_WITH_RC4_128_SHA -SSL_RSA_WITH_RC4_128_MD5
#SSLAttributeSet 471 1
#SSLClientAuth required
</VirtualHost>

6. Restart The IBM HTTP Server

7. For final confirmation once again run step one and check the output.

Hope this will help you. Kindly comment for your suggestion and quires.  

"Effort only fully releases its reward after a person refuses to quit.”

 Regards,
 Akhilesh B. Humbe

Disable SSLv3 for POODLE Vulnerability in IBM Http Server

Hello Everyone,

Hope you all are doing well. Today I had a task to disable the SSLv3 and SSLv2 protocol on IBM Http Server (HIS). SSLv3 is enabled by default in the Apache based IBM HTTP Server and which cause  a vulnerability that has been referred to as the Padding Oracle On Downgraded Legacy Encryption (POODLE). In which IBM HTTP Server could allow a remote attacker to obtain sensitive information, caused by a design error when using the SSLv3 protocol. A remote user with the ability to conduct a man-in-the-middle attack could exploit this vulnerability via a POODLE (Padding Oracle On Downgraded Legacy Encryption) attack to decrypt SSL sessions and access the plain text of encrypted connections. This vulnerability affects all versions and releases of IBM HTTP Server (IHS) component in all editions of WebSphere Application Server and bundling products.

Resolution:

To resolve this issue IBM recommends to disable SSLv3

Procedure:

As it's a process of disabling SSLv3 on Web Server we and have different option available on different Web servers and you may get confused about that. Here we are disabling SSLv3 on IBM Http Server, so we check the procedure to disable the SSlv3 protocol on HIS.

1. First you need to find the file where you have a setting for the SSL configuration. In most of the environment it's a httpd.conf but in some cases if you are using different file you can find the file using below command.

grep -i -r "SSLEnable" /opt/IBM/HTTPServer/
/opt/IBM/HTTPServer/ is Webserver home

2. Take backup of the configuration file e.g. httpd.conf

3. Edit the configuration file to add below line after the SSLEnable section

SSLProtocolDisable SSLv3 SSLv2

It will look like below stanza in configuration file

<VirtualHost *:${HTTPD_PORT}>
SSLEnable
SSLProtocolDisable SSLv3 SSLv2
#SSLAttributeSet 471 1
#SSLClientAuth required
</VirtualHost>

4. Restart The IBM Http Server


Hope this will help you. Kindly comment for your suggestion and quires.  

"Effort only fully releases its reward after a person refuses to quit.”

 Regards,
 Akhilesh B. Humbe

Caused by: java.lang.ClassNotFoundException: com.ibm.ws.javaee.dd.ejb.Method

Hello Everyone,

Hope you all are doing well. It's after five months I am writing this post. I was getting below error while starting the adminagent in one of my Websphere 7 environment. I am getting this error while startup and adminagent was not getting start. You may face the same error on dmgr or on application server instance. When I checked for the issue I got below details in SystemOut.log

Error :

************* End Display Current Environment *************
[4/15/16 18:16:15:386 EDT] 00000001 ManagerAdmin  I   TRAS0017I: The startup trace state is *=info.
[4/15/16 18:16:15:389 EDT] 00000001 ManagerAdmin  I   TRAS0111I: The message IDs that are in use are deprecated
[4/15/16 18:16:15:738 EDT] 00000001 ModelMgr      I   WSVR0800I: Initializing core configuration models
[4/15/16 18:16:16:612 EDT] 00000001 WsServerImpl  E   WSVR0009E: Error occurred during startup
java.lang.IllegalStateException: java.lang.IllegalStateException: java.lang.IllegalStateException: java.lang.reflect.InvocationTargetException
        at com.ibm.ws.config.ModelMgr.initializeModels(ModelMgr.java:355)
        at com.ibm.ws.config.ModelMgr.initialize(ModelMgr.java:162)
        at com.ibm.ws.runtime.WsServerImpl.start(WsServerImpl.java:221)
        at com.ibm.ws.runtime.WsServerImpl.main(WsServerImpl.java:697)
        at com.ibm.ws.runtime.WsServer.main(WsServer.java:59)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.ibm.wsspi.bootstrap.WSLauncher.launchMain(WSLauncher.java:234)
        at com.ibm.wsspi.bootstrap.WSLauncher.main(WSLauncher.java:96)
        at com.ibm.wsspi.bootstrap.WSLauncher.run(WSLauncher.java:77)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.eclipse.equinox.internal.app.EclipseAppContainer.callMethodWithException(EclipseAppContainer.java:587)
        at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:198)
        at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
        at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
        at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:369)
        at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.eclipse.core.launcher.Main.invokeFramework(Main.java:340)
        at org.eclipse.core.launcher.Main.basicRun(Main.java:282)
        at org.eclipse.core.launcher.Main.run(Main.java:981)
        at com.ibm.wsspi.bootstrap.WSPreLauncher.launchEclipse(WSPreLauncher.java:400)
        at com.ibm.wsspi.bootstrap.WSPreLauncher.main(WSPreLauncher.java:163)
Caused by: java.lang.IllegalStateException: java.lang.IllegalStateException: java.lang.reflect.InvocationTargetException
        at com.ibm.ws.config.ModelMgr.performInitialization(ModelMgr.java:620)
        at com.ibm.ws.config.ModelMgr.processInitializer(ModelMgr.java:577)
        at com.ibm.ws.config.ModelMgr.loadModelsViaEclipse(ModelMgr.java:512)
        at com.ibm.ws.config.ModelMgr.initializeModels(ModelMgr.java:351)
        ... 30 more
Caused by: java.lang.IllegalStateException: java.lang.reflect.InvocationTargetException
        at com.ibm.ws.config.ModelMgr.performInitialization(ModelMgr.java:647)
        at com.ibm.ws.config.ModelMgr.performInitialization(ModelMgr.java:617)
        ... 33 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.ibm.ws.config.ModelMgr.performInitialization(ModelMgr.java:644)
        ... 34 more
Caused by: java.lang.NoClassDefFoundError: com/ibm/ws/javaee/dd/ejb/Method
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClassCond(ClassLoader.java:637)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:621)
        at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.defineClass(DefaultClassLoader.java:188)
        at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.defineClass(ClasspathManager.java:580)
        at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findClassImpl(ClasspathManager.java:550)
        at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLocalClassImpl(ClasspathManager.java:481)
        at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLocalClass_LockClassLoader(ClasspathManager.java:469)
        at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLocalClass(ClasspathManager.java:449)
        at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.findLocalClass(DefaultClassLoader.java:216)
        at org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(BundleLoader.java:393)
        at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:469)
        at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:422)
        at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:410)
        at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(DefaultClassLoader.java:107)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        at org.eclipse.jst.j2ee.internal.J2EEInit.init(J2EEInit.java:123)
        at org.eclipse.jst.j2ee.commonarchivecore.internal.helpers.ArchiveInit.invokePrereqInits(ArchiveInit.java:100)
        at org.eclipse.jst.j2ee.commonarchivecore.internal.helpers.ArchiveInit.init(ArchiveInit.java:81)
        at org.eclipse.jst.j2ee.commonarchivecore.internal.helpers.ArchiveInit.init(ArchiveInit.java:45)
        at com.ibm.websphere.models.config.init.ReadOnlyConfigInit.initializeCommonArchive(ReadOnlyConfigInit.java:110)
        at com.ibm.websphere.models.config.init.ReadOnlyConfigInit.init(ReadOnlyConfigInit.java:90)
        ... 39 more
Caused by: java.lang.ClassNotFoundException: com.ibm.ws.javaee.dd.ejb.Method
        at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:506)
        at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:422)
        at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:410)
        at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(DefaultClassLoader.java:107)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        ... 61 more

Cause:

There are several reasons for this Error. After an upgrade , it is possible that the class cache's are still holding onto previous versions of classes. Or improper stop start, It is also possible that the caches became corrupted.

Solution:

You need to clear the two types of the cache, the JVM's cache and the OSGi cache. The server has to be stopped before clearing the cache

To clear the OSGi class cache:
For UNIX platforms, run the following script in each profile:
<WebSphere_HOME>/profiles/profile_name/bin/osgiCfgInit.sh

To clear the JVM's class cache:
For UNIX platforms, run the following script:
<WebSphere_HOME>/bin/clearClassCache.sh

And clear the following directory contents:
<WinUsers_home>\Local Settings\ApplicationData\javasharedresources\

Procedure:

1. The server has to be stopped before clearing the cache

2. You must be logged in as the user who started WebSphere Application Server. In our case we are executing the commands for adminagent.

3. Execute the command <WebSphere_HOME>/profiles/profile_name/bin/osgiCfgInit.sh

Example:
bash-3.2$ su - was
bash-3.2$ cd /IBM/WebSphere7/profiles/Agent01/bin
bash-3.2$ ./osgiCfgInit.sh
OSGi profile cache successfully cleaned for /IBM/WebSphere7/profiles/Agent01/.
OSGi server cache successfully cleaned for /IBM/WebSphere7/profiles/Agent01/servers/adminagent.

4. Execute the command <WebSphere_HOME>/bin/clearClassCache.sh 

Example:
bash-3.2$ ./clearClassCache.sh
bash-3.2$ ./startServer.sh adminagent
ADMU0116I: Tool information is being logged in file
          /IBM/WebSphere7/profiles/Agent01/logs/adminagent/startServer.log
ADMU0128I: Starting tool with the Agent01 profile
ADMU3100I: Reading configuration for server: adminagent
ADMU3200I: Server launched. Waiting for initialization status.
ADMU3000I: Server adminagent open for e-business; process id is NNNN


Hope this will help you. Kindly comment for your suggestion and quires.  

"Effort only fully releases its reward after a person refuses to quit.”

 Regards,
 Akhilesh B. Humbe

java.lang.OutOfMemoryError: GC overhead limit exceeded

Hello Everyone,

Last time we discussed the issue of java.lang.OutOfMemoryError: Java heap space for Apache Tomcat server. Similarly today we are having another JVM related issue for WebSphere application server. Here my users are complaining about the some kind of slowness while application access and when I check for the error I am getting below error string in logs.

Error:

java.lang.OutOfMemoryError: GC overhead limit exceeded

Cause:

Before solution we need to know what is GC and why we are getting this error called  GC overhead limit exceeded.
Java runtime environment contains a built-in Garbage Collection (GC)  daemon process. Whenever a particular space in memory is no longer used, a process called Garbage Collection clears that memory space and GC do this work on regular basis. There are different algorithms and parameter of GC which defines how GC will work. And if in case GC is not performing well the it will write the error code similar to what we get.

The java.lang.OutOfMemoryError: GC overhead limit exceeded error is displayed when your application has exhausted pretty much all the available memory and GC has repeatedly failed to clean it. The java.lang.OutOfMemoryError: GC overhead limit exceeded error is the JVM’s way of signaling that your application spends too much time doing garbage collection with too little result. By default the JVM is configured to throw this error if it spends more than 98% of the total time doing GC and when after the GC only less than 2% of the heap is recovered. This means that the small amount of heap the GC is able to clean will likely be quickly filled again, forcing the GC to restart the cleaning process again. This forms a vicious cycle where the CPU is 100% busy with GC and no actual work can be done and end users of the application face extreme slowdowns

Solution: 

You can resolve this issue by using below options.

Step 1
Disable the error check using -XX:-UseGCOverheadLimit. You can disable the error check by adding the option  -XX:-UseGCOverheadLimit to the Generic JVM arguments. This will help you to stopping these error alerts. Please keep in mind that this error is very likely to be a symptom of a JVM Heap / tuning problem so my recommendation to you is always to focus on the root cause as opposed to the symptom instated disabling the error alert .

Step 2
You may also try to change Garbage Collector Policy. You can add the below policy of Garbage collector option to the Generic JVM arguments.
-XX:+UseConcMarkSweepGC 
-XX:+UseConcMarkSweepGC this can help you in the space issue as it compact all fragment free space after GC run and make you available more contiguous space.
-XX:+UseParallelGC
This -XX:+UseParallelGC parameter help you to run the GC continuously instated of reaching to the memory space threshold.

Step 3
If the option two not works you can  Increase heap size using -Xmx switch in Generic JVM arguments

Step 4
If option three not works add -XX:+HeapDumpOnOutOfMemoryError to Generic JVM arguments which will automatically generate heap dump in case of out of memory issue. Then analyze the dump with IBM Memory Analyzer looking for memory leaks. On the basis of analysis you can work with the development team, optimize your code to use less memory and/or reuse objects instead of creating new ones thus reducing the number of times the garbage collector runs.

Sometime dealing with the OutOfMemoryError is a log process, you need to analyze lot of things for perfect tuning and sometime it just resolved by JVM parameter setting.

Hope this will help you. Kindly comment for your suggestion and quires.  

"Effort only fully releases its reward after a person refuses to quit.”

 Regards,
 Akhilesh B. Humbe

java.lang.OutOfMemoryError: Java heap space

Hell Everyone,

After a very long time I am working on Apache tomcat. While accessing application login page user facing some issue and getting some errors on the web browser. When I check the tomcat output logs I seen the below error.

Log Location:

/usr/local/apache-tomcat-5.5.23/logs/catalina.out

Error:

Jan 14, 2016 10:40:36 AM org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor processChildren
SEVERE: Exception invoking periodic operation:
java.lang.OutOfMemoryError: Java heap space
        at java.util.HashMap.<init>(HashMap.java:203)
        at java.util.LinkedHashMap.<init>(LinkedHashMap.java:178)
        at org.apache.catalina.startup.HostConfig$DeployedApplication.<init>(HostConfig.java:1314)
        at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:898)
        at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:883)
        at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:492)
        at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1206)
        at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:293)
        at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:120)
        at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1306)

Solution:

As you can see in error it's a java.lang.OutOfMemoryError: Java heap space means we don’t have enough heap memory available  to write new objects of an application. There is not a single specific way to resolve this issue. There may be a different reasons of OutOfMemoryError. You can find out that by generating and analyzing heap dump.

In my case the allocated heap memory for running the application was too less as compared to my requirement. So I increase the Java Heap memory  and it resolve my issue.

How to increase the java heap space in Apache tomcat ?

When you initiate the command to start the Apache tomcat instance it take the configuration from the file catalina.sh. If you want to increase the java heap space you need to manually edit this file. For increasing the java heap space increase the value of -Xmx parameter for the CATALINA_OPTS
E.g. 
cd /usr/local/apache-tomcat-5.5.23/bin/
vi catalina.sh

JAVA_HOME=/usr/java
CATALINA_OPTS="-Xmx512m " [In my case previously it was -Xmx126m]

Save the Configuration changes and start restart the Apache tomcat service.

/usr/local/apache-tomcat-5.5.23/bin/shutdown.sh
/usr/local/apache-tomcat-5.5.23/bin/startup.sh

Note: Do not Increase the heap size blindly, it will not help you every time in case of the Out Of Memory Error. Make a habit to analyze the heap dump and try to find out the best suitable JVM  arguments for your environment.

Important:

How to take a Heap Dump.
Best Guide for Garbage Collection
Best Guide for JVM tuning.

Hope this will help you. Kindly comment for your suggestion and quires.  

"Effort only fully releases its reward after a person refuses to quit.”

 Regards,
 Akhilesh B. Humbe

CRIMC1066E ERROR: Unable to acquire lock on file .imlock''.

Hello Everyone,

Happy New Year 2016 to all readers. It's 1st week of the new year and I am getting new error while initiating IBM Installation Manager console.

CRIMC1066E ERROR:   Unable to acquire lock on file  /export/localhome/was/var/ibm/InstallationManager/.imlock''.

Problem:

While executing command ./imcl -c to launch the command line console of IBM installation manager it displays the following error message.

CRIMC1066E ERROR:   Unable to acquire lock on file  /export/localhome/was/var/ibm/InstallationManager/.imlock''.

The installation manager could not be started because another installation manager instance is already running. Only a single installation manager may be running at any time.

Cause:

We are getting this error if the installation manager is already running by some other user. When we start Installation manager the user create lock file .imlock which not allow other user to run installation manager to other location. This file get deleted when you close the installation manager. In some case if installation manager process got killed accidentally the lock file remains as it is and not allow you to start the installation manager next time and showing the error code. 

Resolution:

1. Ensure that there is no running Installation Manager instance.

2. If there is an Installation manager process running, delete that process

Unix: ps -ef | grep imcl or  ps -ef | grep java (Run these commands to find the process)

Windows: Go to the Task Manager and terminate all the java.exe, javaw.exe and IBMIM.exe processes.

3. Re-launch Installation Manager.

If the above steps do not resolve the issue, follow these steps:

1. Go to location list in the error code and manually delete .imlock file.

Unix: ls -a (use this command to list down the hidden files)

2. Hope this will not require, but if necessary, restart the system.

Hope this will help you. Kindly comment for your suggestion and quires.   

"Effort only fully releases its reward after a person refuses to quit.”

 Regards,

 Akhilesh B. Humbe