The successful development of scalable Web services on WebSphere (or any other application server) requires thorough performance testing. Applying a well-designed, consistent performance testing methodology throughout the development life-cycle is key in satisfying a Web services application's performance requirements.

This article provides strategies for creating load test scenarios that reflect tendencies of the real world environment. To help you apply these strategies, it introduces best practices for organizing and executing load tests, and suggests how these best practices fit into a Web services application's development life-cycle. It also demonstrates how to utilize performance monitors offered by WebSphere and the operating system, as well as how to analyze load testing results. The bulk of the discussion assumes that WSDL is used to describe the Web service's interface, and that SOAP and HTTP are used for the messaging and transport layers, respectively.

Creating a Test Plan

Incorporating Performance Testing into the Web Service Development Life-Cycle
Although performance and scalability are critical parts of any non-trivial Web service, performance testing is often left for the later stages of the development process. Introducing performance testing at the early stages of development, or even prototyping, brings the following benefits:

• Performance-related bugs (such as concurrency problems, memory leaks, and unreleased resources) are caught immediately after they are introduced, which helps isolate the culprit to the recently checked-in code or recent infrastructure changes.
• New features' performance costs can be evaluated and quantified.
• Project management has constant awareness of the application performance and has more confidence in meeting the release dates. This, in turn, greatly reduces the risk of missed deadlines and overspending on expensive enterprise application profiling tools and consultants.

Creating Load Test Scenarios
Performance tests should be started after the functional tests execute successfully. The best way to perform load testing is to have the load test application's virtual users run the complete functional test suite. Load tests should identify and report all transport, message, and application errors.

The usefulness of the performance test results is directly related to how closely the tests emulate request streams that the Web services application will be handling once it is deployed in the production environment. To design load test scenarios with real world value mixes, request types, sequences, and intensities, it is helpful to consider the load test scenario categories shown in Figure 1.

Type of Use (Level 3 head)
Depending on the type of deployment, your Web services can be exposed to various types of SOAP clients. These clients may produce unexpected, erroneous, and even malicious types of requests. Your load test scenarios should include profiles that simulate this type of user. The more your Web service is exposed to the outside ("uncontrolled") world, the greater the probability of a non-regular usage. The "misuse" and "malicious use" categories may include invalid SOAP requests as well as valid requests with unusual or unexpected values or request sizes.

For example, if your service uses arrays or complex types, examine your WSDL and create load test scenarios that emulate requests with expected average and maximum possible element counts, as well as element counts that exceed the allowed maximum.

<xsd:complexType name="IntArray">
  <xsd:sequence>
    <xsd:element name="arg" type="xsd:int" maxOccurs="100"/>
  </xsd:sequence>
</xsd:complexType>

Measure service performance with various sizes of client requests and server responses. If the expected request sizes and their probabilities are known (for example, based on log analysis), then create the request mix accordingly. If such data is unavailable, test with the best-, average-, and worst-case scenarios to cover the full performance spectrum.

Emulation Mode (Level 3 head)
A Web service may or may not support the notion of a user. More generically, it may be stateful or stateless. Your decision to use either "virtual user" or "request per second" request emulation mode should be based on this criteria. For example, the load of a stateless search engine exposed as a Web service is best expressed in terms of a number of requests per second, because the notion of a virtual user is not well-defined in this case. A counter example of a stateful Web service is one that supports customer login, such as a ticket reservation service. In this context, it makes more sense to use virtual user emulation mode.

If your Web service is stateless and you have chosen the request per second approach, make sure that you select a load test tool that supports this mode. If a load test tool can sustain only the scheduled number of users, the effective request injection rate may vary substantially based on the server response times. Such a tool will not be able to accurately emulate the desired request sequence. If the number of users is constant, the request injection rate will be inversely proportionate to the server processing time and will be likely to fluctuate, sometimes dramatically, during the test. Web services load test tools such as Parasoft SOAPtest offer both "virtual user" and "request per second" modes.

When load testing stateful Web services, such as services that support the notion of a user, make sure that you are applying appropriate intensity and concurrency loads. Load intensity can be expressed in request arrival rate; it affects system resources required to transfer and process client requests, such as CPU and network resources. Load concurrency, on the other hand, affects system resources required to keep the data associated with logged-in users or other stateful entities - session objects in memory, open connections, or used disk space. A concurrent load of appropriate intensity could expose synchronization errors in your Web services application. You can control the ratio between load intensity and concurrency by changing the virtual user think time in your load test tool.

Content Type (Level 3 head)
When load testing Web services, it is easy to overlook the fact that SOAP clients may periodically refresh the WSDL, which describes the service, to get updates on the service parameters it is about to invoke. The probability of such updates may vary depending on the circumstances. The test team can analyze access logs or make reasonable predictions based on the nature of the service. If the WSDL access factor (the probability of WSDL access per service invocation) is high and WSDL size is compatible with the combined average size of request and response, then network utilization will be noticeably higher in this scenario, as compared to the one without WSDL refresh. If your Web service's WDSLs are generated dynamically, the high WSDL access factor will affect server utilization as well. On the other hand, if your WSDLs are static, you can offload your application server by moving the WSDL files to a separate Web server that is optimized for serving static pages. Such a move creates increased capacity for processing Web service requests.

Type of Load (Level 3 head)
To ensure that your Web services application can handle the challenges it will face once it is deployed in production, you test its performance with various load intensities and durations. Performance requirement specifications should include metrics for both expected average and peak loads. After you have run average and peak load scenarios, conduct a stress test. A stress test should reveal the Web services application's behavior under extreme circumstances, which would cause your application to start running out of resources, such as database connections or disk space. Your application should not experience an unrecoverable crash under this stress.

Performance degradation - even dramatic degradation - is acceptable in this context, but the application should return to normal after the load has been reduced to the average. If the application does not crash under stress, verify that the resources utilized during the stress have been released. A comprehensive performance testing plan will also include an endurance test that verifies the application's ability to run for an extended period of time. Endurance tests are run for hours or days, and could reveal slow resource leaks that are not noticeable during regular tests. Slow memory leaks are among the most common - if they are present in a Java environment, these leaks could lead to a java.lang.OutOfMemoryError and the crash of the application server instance.

Creating a Real World Value Mix
To better verify the robustness of your Web service, you should use your load test tool to generate a wide variety of values inside SOAP requests. This mix can be achieved, for example, by using multiple value data sources (such as spreadsheets or databases), or by having the values of the desired range dynamically generated (scripted) and then passed to virtual users that simulate SOAP clients. By using this approach in load tests of sufficient duration and intensity, you can test your Web service with an extended range and mix of argument values that will augment your functional testing. Depending on the circumstances, it may be advisable to run the mixed request load test after all known concurrency issues have been resolved. If errors start occurring after the variable request mix has been introduced, inspect error details and create functional tests using the values that caused your Web service to fail during load testing. These newly created functional tests should become part of your functional test suite.

Web Services Design Considerations
The load test scenario creation practices previously mentioned help reveal performance problems that have already been introduced into a Web services application. However, it's equally important to prevent potential performance problems at the design and development stages. By focusing on prevention as well as detection, you reduce the number and severity of performance problems that you would have to identify and repair later, which in turn streamlines the development process and conserves development/testing resources. This section offers some tips on designing Web services in a way that prevents performance problems.

Minimize the Number of Service Requests and Accumulated Traffic Size
Many current Web services development software packages, including WebSphere Studio Application Developer, offer tools that greatly automate the exposure of Java code via Web services. Although these tools are effective development instruments, indiscriminate use of such tools can cause performance problems. For instance, assume that we want to use the following Java interface for customer registration:

public interface Customer {
	void setName(String first,String last);
	void setAddress(Address address);
	void setTelephone(String phone);
	void setAccountNo(String account);
}

If the Customer interface is exposed as a Web service, it will take four SOAP calls to complete the customer registration operation. We can reduce the number of remote calls and accumulated request processing time and traffic size by introducing a single register operation as follows:

void register
(String firstName,String lastName,Address address,String phone,String account);

An even better approach would be to introduce a CustomerRegistrationData type that will hold all of the parameters necessary for registration:

void register(CustomerRegistrationData data);

When designing a Web services interface, it is advisable to define a set of high-level actions that the Web service user would want to perform. Then, create a set of SOAP calls that perform the desired high-level actions with the minimal request quantity and traffic volume between the client and the server.

Minimize Potential Performance Problems
As previously discussed, your service can be exposed to erroneous or malicious types of requests. It's often possible to reduce the performance degradation caused by such requests. For example, assume that your Web service's WSDL was auto-generated and has complex types that allow an unlimited number of elements as a result of a default conversion of a Java array, or a vector.

<xsd:complexType name="stringArray">
  <xsd:sequence>
    <xsd:element name="id" type="xsd:string" maxOccurs="unbounded"/>
  </xsd:sequence>
</xsd:complexType>

Your application business logic, however, may have a well-defined limit of the number of elements in the complex type:

<xsd:element name="id" type="xsd:string" maxOccurs="20"/>

Setting a limit will cause erroneous or malicious messages to be rejected during the early validation stages of request processing, thereby minimizing the performance penalty for servicing such requests.

Other Considerations

In certain circumstances, you can gain performance benefits using another optimization technique: turning off XML schema validation and performing custom validation and parameter deserialization. If you choose such custom SOAP message processing, SAX-style parsers are a preferred choice for most situations, but especially for processing large messages. SAX parsers are faster and more memory-efficient than their DOM equivalents because they do not create a tree object model of the XML file in memory, but rather represent the XML document as a series of events.

Following a layered application design approach offers several advantages. One such advantage is that it helps you isolate the components responsible for the performance drop by turning on or off various application layers (such as security and access control) or persistence layers by stubbing out database calls.

What Are We Measuring?

Making Sense of Response Time Measurements
Response time is one of the major parameters that characterize Web service quality. Although measuring a time interval may seem straightforward, it may be trivial to decide where to place this measurement, how to interpret it, and how to estimate its accuracy. Most load testing tools measure the elapsed time between sending a request and receiving a response from the server. Although measuring response time on the client side better reflects simulated user experience, it might depend heavily on the network environment, where the particular load testing experiment is performed and can yield different results with different network configurations. For this reason, it is beneficial to monitor the server processing time as well. WebSphere Application Server offers such monitors (see Figure 2). Comparing server processing times with the virtual user recorded response times gives a good estimate of the delays caused by the network.

Eliminating Sources of Incorrect Results on the Test Client
Load testing tools that emulate hundreds to thousands or more virtual users may cause heavy resource utilization - or even resource starvation - on test client machines. This can not only cause measurement errors, but also distort the stream of requests generated by the load test tool. For example, CPU saturation, paging, and long JVM garbage collection cycles (if the load test client is a Java application) may cause bursts of requests instead of the desired uniform - Poisson - or another request distribution that the load test tool is supposed to generate. To prevent anomalies and measurement distortions caused by resource overutilization on load-generating machines, you can monitor major performance parameters, such as CPU, memory, and network utilization. If your monitoring efforts reveal that resource utilization goes higher than 75%, add more hardware to avoid those anomalies.

Overutilization of load test client resources can also result in test failures, which can be erroneously interpreted as server faults. Figure 3 presents an example of a load test run that has errors caused by socket starvation on the client, which ran on a Windows XP machine. The load test ran without errors for approximately 140 seconds, then errors started occurring in bursts of up to 7 errors per second. Error details indicate network problems:

java.net.BindException: Address already in use: connect

Adding TCP monitors on the test client revealed that the upper port on the client machine grows for the first half of the test, reaches the value of 5000, then stays at that value for the remainder of the test. The graph also shows a correlation between the upper TCP port reaching the maximum value and the beginning of errors. Further investigation revealed that, for Windows 2000 and higher, the operating system configuration restricts attempts to set up TCP connections from port 5000 and above. The upper port value is controlled by a Registry Key setting. After the Key value was increased to the maximum value of 65534, the load test ran without errors.

Conclusion

By following performance testing best practices, you can achieve more comprehensive coverage of possible performance issues that could arise during the production deployment of your Web service and increase its overall quality.