Feature

The WebSphere Application Server Network Deployment version 5.0 (WAS ND) provides an infrastructure for you to centrally administer multiple WAS servers, resources, and other elements of your topology. Your managed topology can include support for clustered servers with workload management and failover. WAS ND's support for centrally administering topologies provides significant benefits for both large-scale and small-scale topologies.

This article provides tips to help you better exploit the administrative capabilities of WAS ND. We discuss options for deciding on the scope of your topology, and advice on administering the topology. If you are planning or administering a topology based on WAS ND (or WAS Enterprise), these tips, which are based on lab experiences with large ND deployments, should be helpful.

Tips for Planning Cells

Applications and the resources they require are the key elements in any topology. WAS ND provides flexibility for spreading your applications across cells, nodes, and application servers. There is no magic formula that tells you exactly how to build your cell topology, but your solution needs to combine performance and scalability with your site requirements and administrative processes.

WAS ND allows for many nodes, with multiple application servers on each node and multiple applications in each server. In fact, the WAS ND architecture allows as many as you want. Lab experiments have achieved hundreds of nodes and server elements without experiencing any "hard" limits (see Figure 1).

While large configurations are impressive, they may not be entirely practical for you. As you add elements to your topology, it's logical that you'll experience effects of a larger topology. When you plan your topology, consider the practical limits. In terms of the number of nodes and servers, practical limits include complexities in administering a large topology, as well as physical constraints such as memory and network bandwidth available.

Let's take a look at a few tips for you to consider when you plan your topology.

Place the Deployment Manager on a Dedicated System

The deployment manager for your cell can be co-located on a system with a node agent and application servers. But, sharing a system can introduce contention for system resources and interfere with administration performance. Plus, remember that your deployment manager is your central point of administration for your entire cell, and anything that compromises your deployment manager process also compromises your ability to manage and monitor your cell. Therefore, consider placing the deployment manager on a dedicated system.

Put One or a Small Handful of Applications in Each Application Server

While it is entirely possible to put many, many applications in each server, the more typical and preferred deployment is a single application, or perhaps a small set of closely related applications. Keeping applications separated is best from a standpoint of security, isolation, manageability, and so on.

Provide for Fail-Over

Though we won't go into all of the various fail-over and high availability alternatives and options for your site, you'll typically want at least two nodes (at a minimum), to provide for fail-over. Using WAS ND cluster capabilities, each application is set up and managed as a cluster across the nodes.

Meet Each Application's Performance and Scalability Requirements

While many application performance and scalability requirements will be satisfied by just two application servers across two nodes, others have additional needs. In terms of your performance needs, make sure you consider them in case of maintenance or failures. For example, if one server in a two-node cluster is brought down, your site is immediately at 50% capacity. If two nodes are required to meet performance goals, a three-node cluster is the appropriate number to maintain ongoing capacity.

Next, consider how many servers you need to handle your user load. If two servers are insufficient, the easiest scalability solution is to add additional nodes, taking care to scale any shared resources such as databases and directory servers.

Maximize Each Server's Hardware Utilization

Once you have determined how many nodes you need for each application, and therefore, how many application servers you'll need, you can turn your focus to maximizing the hardware resources at your disposal. The first step is to utilize the processing capacity of your servers. Under expected load conditions, check the processor utilization. If the processor is already heavily utilized, do not add more application servers. Adding more servers to an already utilized machine does not improve performance. Typically, if you have a small amount of hardware (2-4 CPUs), running multiple application servers is not necessary for hardware utilization.

However, if you have many applications, but these applications aren't heavily used, you may want to put multiple application servers on a single node to extend the use of the hardware. When doing this, pay special attention to the physical memory discussion that follows.

Keep Application Servers in Physical Memory

Determining the number of application servers per node requires an understanding of the available memory. Never create more application servers than the available system memory can support. This condition leads to paging that adversely affects performance across the system. The default WAS Java Virtual Machine (JVM) maximum heap size is 256 MB. However, when an application server starts up, it does not use its maximum 256 MB heap size. Thus, application servers exceeding available memory may not be apparent at startup time but can result in excessive paging as the servers increase their heap utilization under load.

For planning purposes, recognize that the heap size grows. As the heap size grows, the Java process for the server uses more system memory. The total memory used is equal to the current heap size plus the Java interpreter. The size of the Java interpreter depends on the heap size. For example, a 256MB heap typically has approximately 64 MB additional from the interpreter, whereas a 128MB heap will have approximately 42 MB. Therefore, an application server with a maximum heap size of 256MB can grow to approximately 320MB of physical memory. A 128MB node agent will grow to approximately 170MB.

To plan the maximum number of application servers for a single node, include the memory requirements for the operating system, node agent, deployment manager (if co-resident), and the maximum size of the application servers (including the Java interpreter). The calculations are shown in Table 1. Leaving extra memory is safer than exceeding server memory capacity, so we recommend including a buffer (our calculations used 0.8 as the buffer, but you can increase the value for an even larger margin of safety).

With an available 4GB of physical memory, the calculations show that memory requirements permit up to eight servers (256MB maximum heap) on this system. You should not have more (and as discussed earlier, you may not need to use eight).

Know When to Stop

After you understand what each application requires, how to utilize each server, and your fail-over and maintenance strategy, you still need to determine exactly where to put your applications. If you have a lot of applications, how many is too many to cram into a cell? When do you split across multiple cells? There is no right or wrong answer here. There are both performance and administrative considerations.

With ND-centralized administration, each node has a node-agent process that communicates with the deployment manager to provide status on servers and applications. In addition, the deployment manager and node agents continuously compare notes on the configuration. Background communication between the deployment manager and the node agent is lightweight and has very little effect on application-server performance, but a large number of nodes in the topology can result in longer response times for routine administrative actions and monitoring.

In addition, a large number of servers increases administration-console navigation. By default, the administration console lists 20 servers per page. If there are 200 servers, the administrator traverses 10 pages to get to server 200. As we will discuss later, features exist to make this task easier, such as server filtering by name or node.

Our personal preference was to limit the number of servers in the cell to approximately 60, either using quite a few small servers, with just a couple of application servers on each (i.e., 20 nodes with 3 servers each) or a few larger servers, with quite a few application servers (i.e., 3 nodes with 20 servers each). The choice of approach here depends on hardware availability, operating system choices, cost, etc. Find the right balance to meet your needs by understanding your administrative processes and performance requirements.

Tips for Administering Cells

No matter what size WAS ND system you have, here are a few tips that will make administering it easier and faster.

DEFINE RESOURCES AT CELL LEVEL
You can take advantage of the WAS ND cell architecture by defining resources at the cell level. Resources defined at the cell level are available to application servers throughout the cell. When a resource can be used by multiple application servers, defining it at the cell level enables configuration changes or updates to that resource to be made only once and apply across the cell. For example, often all members in a cluster use the same JDBC driver and data source. Defining the JDBC driver and data source at the cell level enables you to make tuning changes once at the cell level, rather than once per application server.

Use Naming Conventions

To assist in administering multiple servers, establish a good naming convention to keep better track of your topology. For example, include a number in the hostnames of your nodes, such as System1 and System2, and so forth. Then, name your cluster members ClusterMember1 and ClusterMember2.

Also consider renaming your log files with the corresponding number suffix, such as SystemOutNA1.log and SystemOutNA2.log, and SystemOutCM1.log and SystemOutCM2.log for your node agents and cluster members respectively. The larger your deployment, the more important it is to establish a good naming convention.

USE FILTERS TO NAVIGATE ADMINISTRATIVE CONSOLE
With a large number of applications, default settings on the administrative console require you to navigate with next and view entries one page at a time. Faster navigation is possible by setting a larger number of applications per page or filtering the application servers on this list. For maximum usability, we preferred navigating no more than three pages by keeping the total number of application servers listed less than 60.

USE A SCRIPT TO IMPROVE STARTUP FOR MULTIPLE, NON-CLUSTERED SERVERS
The $AdminControl startServer script command issues a request to start a server, and by default, waits for up to 10 minutes for the request to complete. This command works fine for starting one or two servers, but becomes a bottleneck when starting many individual servers; the next server cannot start until the previous server starts.

To speed up startup for multiple servers, you can use a script with a reduced wait time. A shorter wait time parameter setting returns control to your wsadmin script where you can start additional servers while previous start requests are processed. Note that setting a wait time of zero actually produces the same 10-minute wait time as the default.

In our 60-application server, single-node test environment, lowering the wait time to 1 second decreased server startup from over 40 minutes to under 5 minutes. A small wait time enables server start requests to process in parallel, which takes advantage of multiple CPUs in a system, or multiple systems. Starting multiple servers in parallel is the technique WebSphere uses when clusters are started.

Listing 1 includes a sample startup script. Logic is also provided to check server status.

Use a Script to Improve Stop Time for Multiple, Non-Clustered Servers

The $AdminControl stopServer script command issues a request to stop an application server and waits for the request to complete. This command works when stopping a few servers but can be a long process when stopping a large number of servers.

Using the stop method of the server MBean is faster. This method returns control to your wsadmin script without waiting. This approach enables multiple stop requests to proceed in parallel.

Figure 2 shows the time it takes to stop from 5 to 100 application servers. As shown, the server MBean stop method is much faster than using the $AdminControl stopServer request. Stopping multiple servers in parallel is the same technique used by WAS when clusters are stopped. Listing 2 provides a sample stop script as well as logic to check for server shutdown.

DON'T AUTOSTART APPLICATIONS UNNECESSARILY
By default, WebSphere server startup processing starts all applications on the server. Choosing to defer the startup of the applications accelerates the initial server startup.

If there are applications installed that do not need to run until later, disable autostart for these applications and start the applications as needed. For example, you might want WebSphere samples available on a development server, but they are not required to continuously run. In our test of application server startup with 20 simple applications, startup time is reduced by half as a result of disabling autostart on the applications. Remember that time saved at startup is a cost later when the applications are started. To disable autostart, change the target mapping of the application to enabled=false.

Using the Administrative Console:
(1) Select the application: Applications > Enterprise Applications and then select an application to change.
(2) On the Configuration tab, go to Additional Properties and select Target Mappings
(3) Select the target and clear the checkbox to disable the target mapping

Using wsadmin script:

set deployments [$AdminConfig getid /Deployment:$appName/]
set deploymentObject [$AdminConfig showAttribute $deployments deployedObject]
set targetMappings [lindex [$AdminConfig showAttribute $deploymentObject
targetMappings] 0]
$AdminConfig modify $targetMappings {{enable false}}
$AdminConfig save

Conclusion
WAS ND v5 provides a new infrastructure to manage and administer your servers and applications. Improvements in the architecture create a faster and more flexible administration experience when managing large WebSphere environments. New scripting capabilities provide you with the capability to quickly and efficiently manage your entire topology with automation and reliability. The administrative console can be used to check on your topology or modify it from any Web browser.

WAS ND v5 contains no hard limits on what you can do with your topology, but you should consider practical limits such as memory and administrative complexity as important elements in optimizing your network deployment performance.

References

Joines, Stacy; Willenborg, Ruth; and Hygh, Ken. (2002). Performance Analysis for Java Web Sites. Addison-Wesley.

Alcott, Tom. "J2EE Application Deployment: One or Many Applications per Application Server." www.software.ibm.com/wsdd.