Wednesday, September 24, 2014

Heap Analysis using Eclipse Memory Analyzer

Download Eclipse Memory Analyzer from Download Link (Click on the Download link)

The Memory Analyzer is able to work with HPROF binary heap dumps, IBM system dumps (after preprocessing them), and IBM portable heap dumps (PHD) from a variety of platforms.
Typical information which can be found in heap dumps (once more - depending on the heap dump type) is:
  • All Objects - Class, fields, primitive values and references
  • All Classes - Classloader, name, super class, static fields
  • Garbage Collection Roots - Objects defined to be reachable by the JVM
  • Thread Stacks and Local Variables - The call-stacks of threads at the moment of the snapshot, and per-frame information about local objects
What do  we have in a heap dump:
 
A heap dump is a snapshot of the memory of a Java process at a certain point of time. There are different formats for persisting this data, and depending on the format it may contain different pieces of information, but in general the snapshot contains information about the java objects and classes in the heap at the moment the snapshot was triggered. Usually a full GC is triggered before the heap dump is written so it contains information about the remaining objects.
The Memory Analyzer is able to work with HPROF binary heap dumps, IBM system dumps (after preprocessing them), and IBM portable heap dumps (PHD) from a variety of platforms.
Typical information which can be found in heap dumps (once more - depending on the heap dump type) is:
  • All Objects
    Class, fields, primitive values and references
  • All Classes
    Classloader, name, super class, static fields
  • Garbage Collection Roots
    Objects defined to be reachable by the JVM
  • Thread Stacks and Local Variables
    The call-stacks of threads at the moment of the snapshot, and per-frame information about local objects
A heap dump does not contain allocation information so it cannot resolve questions like who had created the objects and where they have been created.
 
Launch the application - Start your application with Java 6, then start <jre6>/bin/jconsole.exe and select the running application (in this case Eclipse).
 


Then, select the operation dumpHeap from the com.sun.management.HotSpotDiagnostic MBean. The first parameter p0 is the full path to the heap dump file. Make sure you give it the file extension .hprof. The second parameter p1 should be left at true as we are only interested in live objects. 
 
 
Click on File -> Open Heap Dump

 


 
On the right, you'll find the size of the dump and the number of classes, objects and class loaders.
Right below, the pie chart gives an impression on the biggest objects in the dump. Move your mouse over a slice to see the details of the objects in the object inspector on the left. Click on any slice to drill down and follow for example the outgoing references.
 
Shallow Size: Shallow Size is the memory consumed by one object.  An object needs 32 or 64 bits (depending on the OS architecture) per reference, 4 bytes per Integer, 8 bytes per Long, etc. Depending on the heap dump format the size may be adjusted (e.g. aligned to 8, etc...) to model better the real consumption of the VM. 
 
Retained set of X is the set of objects which would be removed by GC when X is garbage collected.
Retained heap of X is the sum of shallow sizes of all objects in the retained set of X, i.e. memory kept alive by X.
Generally speaking, shallow heap of an object is its size in the heap and retained size of the same object is the amount of heap memory that will be freed when the object is garbage collected.
The retained set for a leading set of objects, such as all objects of a particular class or all objects of all classes loaded by a particular class loader or simply a bunch of arbitrary objects, is the set of objects that is released if all objects of that leading set become unaccessible. The retained set includes these objects as well as all other objects only accessible through these objects. The retained size is the total heap size of all objects contained in the retained set.
 
 
The Memory Analyzer displays by default the retained size of individual objects. However, the retained size of a set of objects - in this case all instances of a particular class - needs to be calculated.
To approximate the retained sizes for all rows, pick Calculate retained size icon from the tool bar. Alternatively, select a couple rows and use the context menu.
 
Select calculate retained sizes from the tool bar
 
Using the context menu , you can drill-down into the set of objects which the selected row represents. For example, you can list the objects with outgoing or incoming references. Or group the objects by the value of an attribute. Or group the collections by their size. Or or or...
  
One thing that makes the Memory Analyzer so powerful is the fact that one can run any action on any set of objects. Just drill down and slice your objects the way you need them. 

  Drill down via the context menu
 
Another important feature is the facility to group any histogram by class loader, packages or superclass .
 
Group the histogram by class loader or package via the tool bar
   
Any decent application loads different components by different class loaders. The Memory Analyzer attaches a meaningful label to the class loader - in the case of OSGi bundles it is the bundle id. Therefore it becomes a lot easier to divide the heap dump into smaller parts.
 

Analyze Class Loader

Class loaders load classes into the memory of the JVM. When analyzing the heap, class loaders are very important for two reasons: First, applications typically load components using separate class loaders. Second, the loaded classes are usually stored in a separate space (e.g. the perm space) which can also be depleted.

Class Loader Explorer

To get an overview, run the Query Browser > Java Basics > Class Loader Explorer inspection on the heap dump.
  • The Memory Analyzer attaches a meaningful label to the class loader - in the case of OSGi bundles it is the bundle id. Looking out for duplicate entries!
  • Next to the class loader name, the table contains the defined classes and the number of live instances. If one and the same component is loaded multiple times, the number of live instances can indicate which class loaders is more alive and which one should be garbage collected.
 
Class Loader Explorer
 
 
Histogram grouped by class loader
 
Grouping the histogram by packages allows to drill-down along the Java package hierarchy.
 
Histogram grouped by packages
 
Grouping the histogram by superclass provides an easy way to find for example all the subclasses of java.util.AbstractMap, etc...
 
 
Histogram grouped by superclass
 
Concept of Domination Tree: It is useful to determine which objects are taking major part of heap memory and dependencies within the objects can be easily identified.
 
for e.g. If we have 2 objects (A&B), if all paths going to B needs to go via A then A is called the parent of B.
 
Memory Analyzer provides a dominator tree of the object graph. The transformation of the object reference graph into a dominator tree allows you to easily identify the biggest chunks of retained memory and the keep-alive dependencies among objects. Bellow is an informal definition of the terms.
An object x dominates an object y if every path in the object graph from the start (or the root) node to y must go through x.
The immediate dominator x of some object y is the dominator closest to the object y.
A dominator tree is built out of the object graph. In the dominator tree each object is the immediate dominator of its children, so dependencies between the objects are easily identified.
The dominator tree has the following important properties:
  • The objects belonging to the sub-tree of x (i.e. the objects dominated by x ) represent the retained set of x .
  • If x is the immediate dominator of y , then the immediate dominator of x also dominates y , and so on.
  • The edges in the dominator tree do not directly correspond to object references from the object graph.
 
The Leak Report
The Memory Analyzer can inspect the heap dump for leak suspects, e.g. objects or set of objects which are suspiciously big.
  
Run the leak report

Friday, July 11, 2014

Encode and Decode WAS Password

Hi All,

Please follow below method to encode/decode password:

Encode:

$WAS_BINARIES/java_1.7_64/jre/bin/java -Djava.ext.dirs=$WAS_BINARIES/plugins:$WAS_BINARIES/lib com.ibm.ws.security.util.PasswordEncoder abcdef

Output:

decoded password == "abcdef", encoded password == "{xor}Pj08Ozo5"






Decode:

$WAS_BINARIES/java_1.7_64/jre/bin/java -Djava.ext.dirs=$WAS_BINARIES/plugins:$WAS_BINARIES/lib com.ibm.ws.security.util.PasswordDecoder {xor}Pj08Ozo5

output:

encoded password == "{xor}Pj08Ozo5", decoded password == "abcdef"




Add Certs to WAS Truststore

1)     Go to Websphere Console
2)     Click on Security -> SSL Certificate and Key Mgmt -> Manage endpoint security configurations
3)     Click on node 1 (inbound/outbound - NodeDefaultSSLSettings) -> Check and note down“Inherited SSL Configuration Name”
4)     Click on Security -> SSL Configurations -> “Inherited SSL Configuration Name”
5)     Note “Trust Store name”
6)     Click on Key Stores and certificates
7)     Take backup of trust.p12 path mentioned in front of “Trust Store name”.
8)     Click “Trust Store name” -> Signer Certficates
9)     Place the cert on that node at any location.
10)  Click on Add
11)  Name it anything (Preferred – name of the URL)
12)  Give complete path of the cert.
13)  Restart all Nodeagents and app servers for changes to take place.

Monday, June 2, 2014

Liberty Profile

Introduction to Liberty Profile:



  • The Liberty profile provides a simplified stand-alone run time for web applications, supporting a subset of the programming model and run time features available with the full profile. Any application that runs on the Liberty profile will also run on the full profile.
  • It is a good option for developers who are building web applications that do not require the full Java EE environment of traditional enterprise application server profiles. For production environments, enterprise qualities of service, such as security and transaction integrity, are enabled as required.
  • Liberty profile instances are stand-alone servers similar to application server profiles;however, they are characterized by a dynamic runtime based on the OSGi model. This design allows you to configure the server runtime stack and features in real time. This design also results in a small server footprint and fast startup time because the server uses only feature that are needed by the applications. For example, if an application requires just a servlet engine, Liberty profile can be configured to only start the WebSphere Application Server kernel, an HTTP endpoint for connection, and a web container. If the application needs additional features, such as database connectivity, the Liberty profile configuration can be dynamically modified to include the JDBC feature without the need of a server restart.
  • The Liberty profile includes the following key features:
    • A dynamic and flexible run time to load only what the application needs
    • A quick startup time (under 5 seconds with simple web applications)
    • A simplified configuration that uses a single configuration file or modular configuration
    • Support for deploying applications developed in the Liberty profile to run in the full profile
    • Support of web applications, OSGi applications, and Java Persistence API
    • (New in V8.5.5) Support for web services
    • (New in V8.5.5) Support for Java Messaging Support (JMS)
    • A secure server environment. User registry options include single or (New in V8.5.5) federated LDAP registries, role based authorization, SSL, single sign-on, custom login modules, (New in V8.5.5) OAuth support, and more.
    • (New in V8.5.5) Support for the use of MongoDB, a NoSQL database system. The server and client drivers are not shipped with WebSphere Application Server.
    • Ability to deploy an application and configured server as a package
    • (New in V8.5.5) Ability to extend the Liberty profile with custom features
    • (New in V8.5.5) Ability to cluster servers for high availability and scalability
    • (New in V8.5.5) Centralized operational management of groups of Liberty servers and of Liberty clusters.
    • Managed, centralized deployment for many nodes of a packaged application and server using the Job Manager.
    • Availability of WebSphere Application Server Developer Tools as Eclipse plug-ins for broad tool support
    • Support for z/OS platform native features such as System Authorization Facility (SAF),
    • Resource Recovery Services (RRS), and z/OS Workload Manager (WLM)
  • The Liberty profile comprises the Liberty profile kernel and any number of optional features that run inside of a single Java Virtual Machine (JVM) process. Most of the kernel runs as OSGi bundles within an OSGi framework. The Liberty profile requires Java Runtime Environment version 6 or higher.
  • When the server is started, the launcher bootstraps the kernel and starts the OSGi framework. The server configuration is parsed, and features are loaded by the feature manager. The kernel makes extensive use of OSGi services to provide a highly-dynamic runtime:
    • System configuration is managed by the OSGi configuration admin service.
    • The OSGi declarative services component is used to manage the lifecycle of system services.
    • Application and configuration file changes are detected by the file monitor service. The file monitor service detects changes that are then reflected in real-time updates.

Configuration: 

1) You can obtain the latest Liberty profile package from the following WASdev community website:

click on WASDevCommunity

2) Configuring JRE

  • The Liberty profile requires a Java Runtime Environment to run in. By default, it does not share the Java Runtime Environment with the WebSphere Application Server, so you must specify the JRE location. The JVM on the system PATH is used by default.

  • The Liberty profile runtime searches for the java command in the following order of properties:
      • JAVA_HOME, JRE_HOME and PATH.
      • Additionally, you can use the Liberty profile server.env configuration file to set up a specified Java Runtime. To configure the Java runtime using this file, add the following line into the file:
      • JAVA_HOME=C:\IBM\jre6
      • If the file does not exist, you must create it manually in the Liberty profile etc directory.
    • Installing the Liberty profile is as simple as running a self-extracting archive. The WASdev web site provides three archives for V8.5.5:
      • Liberty profile: wlp-developers-runtime-8.5.5.0.jar - This archive contains the Liberty core features (equivalent to the V8.5.0 Liberty profile) plus the features that complete the web profile support.
      • Extended content for Liberty profile: java -jar wlp-developers-extended-8.5.5.0.jar - This archive includes web services, messaging, and MongoDB support. 
      • Embeddable EJB container: java -jar wlp-developers-extras-8.5.5.0.jar - This archive includes the embeddable EJB container and JPA client.
      • To manually install a Liberty profile archive:
        • java -jar wlp-developers-runtime-8.5.5.0.jar
        • Press x to skip reading the license terms, or press Enter to view them.
        • Press Enter to view the license agreement.
        • Press 1 if you agree to the license terms and want to proceed.
        • Provide the installation path for the Liberty profile, for example C:\IBM\WebSphere, and press Enter.
    • Server:
      • Run following Command to create the server: server create server1
      • Run following Command to start the server: server run server1
      • Starts the server in debug mode: server debug server1
      • Stop: server stop server1
      • Status: server status server1
      • Configuration:


        • server description should include Server Name
        • feature - what feature we need to enable
        • host - hostname on which the JVM is installed
        • port - WC Default Host and WC Default Host Secure
        • Application ID - Name of the WAR
        • location - Full path of war file
        • name - Name of the WAR 
        • type - WAR
        • if we need to add Data Source Config: We can use include statement as demonstrated below:


    • The bootstrap.properties file is the second file that can be used for the Liberty profile server configuration. Changes to this file require the server to be restarted. By default this file is not generated when you create the Liberty profile server instance. If you want to use this file, you first have to create it. Place this file in the main server directory along side the server.xml file. The bootstrap properties are generally used to initialize the runtime environment for a particular server.
    • Data Source Configuration:

    • Parameters:
      Attribute name How the configuration update is applied
      beginTranForResultSetScrollingAPIs The update is effective immediately.
      beginTranForVendorAPIs The update is effective immediately.
      commitOrRollbackOnCleanup The update is effective immediately.
      connectionManagerRef All connections and the connection pool are destroyed. The data source is then managed by the new connection manager
      connectionSharing The update is applied with each first connection handle in a transaction.
      isolationLevel The update is applied with new connection requests. Current connections retain their isolation level
      jdbcDriverRef All connections and the connection pool are destroyed. The new JDBC driver is then used.
      jndiName All connections and the connection pool are destroyed. The new JNDI name is then used.
      propertiesRef If the data source is Derby Embedded, all connections and the connection pool are destroyed before new properties go into effect. For other JDBC drivers, the new properties go into effect with new connection requests.
      queryTimeout The update is effective immediately.
      statementCacheSize The statement cache is resized upon next use.
      supplementalJDBCTrace All connections and the connection pool are destroyed. The new setting is then used.
      syncQueryTimeoutWithTransactionTimeout The update is effective immediately.
      transactional The update is applied to new connections and existing connections not in use from the connection pool.
      type All connections and the connection pool are destroyed. The new setting is then used.
      agedTimeout The update is effective immediately.
      connectionTimeout The update is effective immediately.
      maxIdleTime The update is effective immediately.
      maxNumberOfMCsAllowableInThread The update is effective immediately.
      maxPoolSize The update is effective immediately.
      minPoolSize The update is effective immediately.
      numConnectionsPerThreadLocal The update is effective immediately.
      reapTime The update is effective immediately.
      purgePolicy The update is effective immediately.
      numConnectionsPerThreadLocal The update is effective immediately.

    • Integrating Messaging Queues

    • The wasJmsServer-1.0 feature configures the Liberty server to support the JMS server run time that provides the capabilities for connections, transactions, persistence, security, and so on.
    • The wasJmsClient-1.1 configures the Liberty server to support JMS client connectivity. This feature provides the resource adapter support that allows JMS clients to perform synchronous and asynchronous messaging activities.
    • The Liberty profile server also supports the use of message-driven beans (MDB). The jmsMdb-3.1 feature provides support for deploying and configuring the JMS resources that are required for the MDB to run within the Liberty profile. This feature enables MDB to interact with either the embedded Liberty messaging or WebSphere MQ.
    • If you want to perform a JNDI lookup for JMS resources, then you must also add the jndi-1.0 feature.
    • Enable JMS in Binding Mode: 
      <wmqJmsClient nativeLibraryPath="/opt/mqm/java/lib64"/>
    Implementing Security (LDAP)
    • The Liberty profile provides support for securing the server runtime environment and web applications by using user registries, authentication, and authorization. For secure communication between the client and the server, you can enable SSL for the Liberty profile. A minimal or detailed configuration can be done by adding the ssl-1.0 server feature to the server configuration file.
    • Implementing LDAP:





    Sunday, June 1, 2014

    Installing Liferay on WAS 8.5

    When the application server binaries have been installed, start the Profile Management Tool to create a profile appropriate for Liferay.

    1. Click on Create…. Choose Application Server. Click Next.
    2. Click the Advanced profile creation option and then click Next. Why Advanced? You can specify your own values for settings such as the location of the profile and names of the profile, node and host. You can assign your own ports. You can optionally choose whether to deploy the administrative console and sample application and also add web-server definitions if you wish. Web server definitions are used with IBM HTTP Server. For more information about these options, please see the WebSphere documentation.
      websphere-01-profile.png
    3. Check the box Deploy the administrative console. This gives you a web-based UI for working with your application server. Skip the default applications. You’d only install these on a development machine. Click Next.
    4. Set profile name and location. Ensure you specify a performance tuning setting other than Development, since you’re installing a server for production use. Please see the WebSphere documentation for further information about performance tuning settings. Click Next.
    5. Choose node, server, and host names for your server. These will be specific to your environment. Click Next.
    6. Administrative security in WebSphere is a way to restrict who has access to the administrative tools. You may want to have it enabled in your environment so that a user name and password are required to administer the WebSphere server. Please see WebSphere’s documentation for further information. Click Next.
    7. Each profile needs a security certificate, which comes next in the wizard. If you don’t have certificates already, choose the option to generate a personal certificate and a signing certficate and click Next.
    8. Once the certificates are generated, set a password for your keystore. Click Next.
    9. Next, you can customize the ports this server profile uses. Be sure to choose ports that are open on your machine. When choosing ports, the wizard detects existing WebSphere installations and if it finds activity, it increments ports by one.
    10. Choose whether you want this profile started when the machine starts. Click Next.
    11. WebSphere ships with IBM HTTP Server, which is a rebranded version of Apache. Choose whether you want a web server definition, so that this JVM receives requests forwarded from the HTTP server. Please see WebSphere’s documentation for details on this. When finished, click Next.
    12. The wizard then shows you a summary of what you selected, enabling you to keep your choices or go back and change something. When you’re satisfied, click Next.
    websphere-03-summary.png
     

    WebSphere then creates your profile and finishes with a message telling you the profile was created successfully. You’re now ready to install Liferay!

    Copying portal dependencies

    Liferay ships with dependency .jars it needs to have on the global classpath. These should be copied to WebSphere’s global folder provided for this purpose:

    [Install Location]/WebSphere/AppServer/lib/ext
    
    If you have a JDBC database driver .jar, copy it to this location as well. Once you’ve copied the .jars, start the server profile you created for Liferay. Once it starts, you’re ready to configure your database.

    Database Configuration

    If you want WebSphere to manage the database connections, follow the instructions below. Note this is not necessary if you’re planning on using Liferay’s standard database configuration; in that case, skip this section. You’ll set your database information in Liferay’s setup wizard after the install.

    02-websphere-jdbc-providers.png

    1. Start WebSphere.
    2. Open the Administrative Console and log in.
    3. Click Resources → JDBC Providers.
    4. Click New.
    5. For name, enter the name of JDBC provider (e.g. MySQL JDBC Provider).
    6. For Implementation Class Name, enter:
      com.mysql.jdbc.jdbc2.optional.MysqlConnectionPoolDataSource
      
    7. Click Next.
    8. Clear any text in classpath. You already copied the necessary .jars to a location on the server’s class path.
    9. Click Next.
    10. Click Finish.
    11. Click Data Sources under Additional Properties.
    12. Click New.
    13. Enter a name: liferaydatabasesource.
    14. Enter JNDI: jdbc/LiferayPool.
    15. Everything else should stay at the default values. Save the data source.
    16. When finished, go back into the data source and click Custom Properties and then click the Show Filter Function button. This is the second from last of the small icons under the New and Delete buttons.
    17. Type user into the search terms and click Go.
      02-modifying-data-source-properties-in-websphere.png
    18.  
    19. Select the user property and give it the value of the user name to your database. Click OK and save to master configuration.
    20. Do another filter search for the url property. Give it a value that points to your database. For example, the MySQL URL would be: jdbc:mysql://localhost/lportal?useUnicode=true&characterEncoding=UTF-8&useFastDateParsing=false. Click OK and save to master configuration.
    21. Do another filter search for the password property. Enter the password for the user ID you added earlier as the value for this property. Click OK and save to master configuration.
    22. Go back to the data source page by clicking it in the breadcrumb trail. Click the Test Connection button. It should connect successfully.
    Once you’ve set up your database, you can set up your mail session.

    Mail Configuration

    If you want WebSphere to manage your mail sessions, use the following procedure. If you want to use Liferay’s built-in mail sessions, you can skip this section.

    1. Click Resources → Mail → Mail Providers.
    2. Click the Built-In Mail Provider for your node and server.
    3. Click Mail Sessions and then click the New button.
    4. Give it a name of liferaymail and a JNDI name of mail/MailSession. Click OK and save to master configuration.
    5. Click Security → Global Security and de-select Use Java 2 security to restrict application access to local resources if it is selected. Click Apply.
    Great! Now you’re ready to deploy Liferay.

    Deploy Liferay


    1. Click ApplicationsNew ApplicationNew Enterprise Application.
    2. Browse to the Liferay .war file and click Next.
    3. Leave Fast Path selected and click Next. Ensure that Distribute Application has been checked, and click Next again.
    4. Choose the WebSphere runtimes and/or clusters where you want Liferay deployed. Click Next.
    5. Map Liferay to the root context (/) and click Next.
    6. Ensure that you have made all the correct choices and click Finish. When Liferay has installed, click Save to Master Configuration.
    You’ve now installed Liferay, but don’t start it yet. If you wish to use PACL, you have one more thing to configure.

    Enabling Security for Portal Access Control Lists

    In the administrative console, go to Security $rarr; Global Security. Check the box to enable Java 2 security, and click Apply. Save to the master configuration.
    websphere-05-liferay-enable-security.png
    Next, you need to configure security for the Liferay profile you created. This requires editing a text file, which can be found nested several folders deep in WebSphere’s profiles directory. The exact path depends on how you’ve named your profile, but it will be something like this:

    [profile_root]/config/cells/[cell_name]/nodes/[node_name]/app.policy
    
    First, in each existing grant section, replace the content with java.security.AllPermission;. Then add the following lines to the bottom of the file:

    grant codeBase "file:${was.install.root}/lib/-" {
      permission java.security.AllPermission;
    };
    
    grant codeBase "file:${was.install.root}/plugins/-" {
      permission java.security.AllPermission;
    };
    
    grant codeBase "file:${server.root}/-" {
      permission java.security.AllPermission;
    };
    
    Save the file. You should now stop the profile and restart it. Once it comes up, you’re ready to start Liferay.

    Start Liferay


    1. If you plan to use Liferay’s setup wizard, skip to the next step. If you wish to use WebSphere’s data source and mail session, create a file called portal-ext.properties in your Liferay Home folder. Place the following text in the file:
      jdbc.default.jndi.name=jdbc/LiferayPool
      mail.session.jndi.name=mail/MailSession
      setup.wizard.enabled=false
      
    2. Select the Liferay application and click Start.
      websphere-04-liferay-starting.png
       

    3. In the setup wizard, select and configure your database type. Click Finish when you’re done.
      Liferay then creates the tables it needs in the database.
    Congratulations! You’ve installed Liferay on WebSphere!