Enterprise portals have become the most popular method of offering a common user interface to a suite of services across the enterprise. Offering business visibility, flexibility, and knowledge management, portals promise users the ability to monitor, search, and manage business activity across the enterprise.

However, as portals mature from basic offerings to more complex sets of services, they will require a robust, agile, and pervasive integration network to provide critical business services to portal users in real time. The Enterprise Service Bus (ESB) not only provides the necessary infrastructure to reliably link services with the portal across heterogeneous environments, disparate organizations, and geographic boundaries - it also has the capability to provide or enable more advanced services, such as integrated business process management and business activity monitoring.

In part 1 of this series (WLDJ, Vol. 2, issue 9), we discussed how the ESB can provide a reliable "services network" for a portal, providing reliable transport, intelligent routing, transformations, and the ability to operate in a highly distributed and federated environment. Strictly speaking, Gartner has defined the ESB has a "lightweight integration platform composed of message-oriented middleware (MOM), transformations, intelligent routing, and Web services." However, once coarse-grained services are accessible from the bus, services can be recombined in multiple ways to form unified business processes that span the enterprise. As business processes execute over days, months, or even years, a new wealth of information can be extracted to provide deeper business awareness and insight. These types of capabilities are the next evolution of the ESB and are critical to providing the "missing link" in a service-oriented architecture (SOA).

In this article, we'll examine two use cases that demonstrate the extended power of the ESB, beyond what a single application server or portal platform can provide on its own. The first use case discusses the notion of a "Worklist Waypoint," a step in a long-running business process that needs manual intervention. A portal offers the user a "human interface" accessible from virtually anywhere, making it a good choice for users to interact with a business process. The second use case shows how portals can provide a window into the operational aspects of the business processes, typically referred to as "Business Activity Monitoring." To illustrate these points, we will once again enhance the "Avitek Medical Records" tutorial shipped with BEA WebLogic 8.1. The tutorial is modeled around a portal (a J2EE Struts application) that allows patients, physicians, and administrators to view an aggregated set of medical records.

Worklist Waypoint
One of the great opportunities of SOA is the ability to create a set of common business processes that bind together services in different applications, technology domains, and even organizational domains. But with this opportunity come significant challenges. What type of infrastructure is required to allow business process execution to operate over long durations (days, weeks, months, or even years)? How will manual steps be intermixed with automated steps, and how will the business process execution interact with my various presentation tiers, both now and in the future?

Let's consider that Avitek would like to automate the process of scheduling lab work for a patient. Through the Avitek portal, a doctor will be able to refer a patient to one of several labs for a test. The business process might look something like Figure 1.

There are a few characteristics that should be observed from this diagram. The first, and primary item of note is that the business process must be long running. The business process combines manual steps (shaded in Figure 1) with automated steps (services on the ESB) that may take an undetermined amount of time. In addition to the latency caused by manual steps, the availability of services themselves may also extend the duration of a business process. Given the long-running nature of this type of business process, the ESB provides a perfect foundation to manage long-running business processes. Leveraging the guaranteed delivery characteristics of the underlying Java Message Service (JMS)-based transport, business process orchestration is a natural extension to the bus.

However, further analysis of the business process reveals a more complex requirement. The manual steps require some type of human interaction to satisfy the conditions of the business process. An open, standards-based, flexible interaction model is required to allow any presentation tier interface to interact with the business process from anywhere on the ESB. For example, Figure 1 shows two different interface types providing input to the process: a portal (in our case, the Avitek portal), and a custom .NET application (designed specifically for the lab technicians to provide results back to the process). The interfaces should be flexible enough to interact with other types as well, such as Interactive Voice Response (IVR), Web services, or even some type of handheld device used by a lab technician. A loosely coupled approach, such as Web services/XML or JMS/XML, is an appropriate choice to satisfy these requirements. The ESB supports both of these natively.

In business process terminology, manual steps are handled through a technique known as a "Worklist." A Worklist is a specialized "inbox" for tasks that need to be accomplished by an "Actor" in a business process. A task correlates to a specific instance of a business process definition. For example, a lab technician may have seven open lab requests that need fulfillment. In this context, a Worklist would contain seven process instances of the "Lab Referral Process" (from Figure 1). The first required service is a "Worklist Summary" that returns the list of tasks for a specific user (see Figure 2).

Once the task lists have been retrieved from the Worklist Summary service, the user may select a task, perform an operation on the task, and submit the task back to the process instance to continue executing. To illustrate the process, the presentation tier first extracts the task (using a process id embedded in the Worklist). The task is represented in XML and returned to the presentation tier (see Figure 3).

The presentation tier would then allow the user to operate on the task (XML). Once the user has completed the necessary editing, the presentation tier submits the modified task (XML) back to the process instance to continue. It's important to note here that the presentation tier submits only a modified XML document back to the process, and does not specifically invoke operations within the process itself. This simplifies process maintenance by keeping the presentation tier unaware of the definition of the business process (see Figure 4).

One common theme throughout these design patterns is the use of XML to represent the list of tasks, as well as the tasks themselves. XML provides the most widely accepted means of data interchange and will have the most adaptability for different presentation tiers. However, a significant point of concern is the number of different XML schemas that may be used to support the different business processes within the ESB. To understand the full nature of the problem, let's consider the Worklist interaction points from the Lab Referral Process shown in Figure 1:
1.  When a lab technician makes a "Worklist Summary Request," they might need to know the patient ID, the type of lab work, and the due date.
2.  When a lab technician makes a "Worklist Task" request for more information, they might need to know the name of the patient, the doctor, relevant history for the lab, and some detailed information related to the type of lab.
3.  Finally, when the lab technician submits the task back to the process, they may need to fill out data specific to the type of lab work performed.

In this simple discreet use case, it is clear that there is a need to support a very contextual set of information to the presentation tier to provide meaningful interaction. When one starts considering the numbers of potential business processes and Worklist requirements within a real-world ESB, it begins to highlight some critical technical requirements on the infrastructure to provide a flexible, adaptive environment:

1.  The Worklist Inbox Must Efficiently Manage XML
Imagine the prospect of manually constructing relational database tables and implementing marshaling code to move XML between the tables for every business process on the ESB. The manual effort required to implement the solution would outweigh the benefits of a service-oriented approach.

For this reason, a natural extension to the ESB is a native XML database, allowing XML of any schema to be persisted, indexed, retrieved, queried, and aggregated, without requiring support from a database administrator. The XML database features a schema-less design, allowing multiple XML schemas to co-exist in the database without prior knowledge or administration.

2.  The Worklist Interaction Model Must Support a Standards-Based, Open, Flexible XML Query Mechanism
To accommodate the presentation needs of various business processes, allowing any data to be retrieved and manipulated at the presentation tier, a flexible XML query mechanism should be used to automate integration with the presentation tier. In addition to the sheer numbers of business processes that may need to be accommodated, imagine also the different needs of the presentation tiers. An IVR system may request a subset of the information, while a "thick" .NET client may have the ability to request all of the information.

Luckily, a W3C standard known as XQUERY is being developed for this purpose. An XQUERY facility that natively operates with the ESB's native XML database provides a standards-based approach for different presentation tiers on the ESB to interact with any business process, regardless of the XML schemas in use.

The "Lab Results" service could be quickly integrated with the Avitek Portal by constructing a JSP that invokes the service using XML parameters (the principle would be a useful parameter) and extracts the response in XML. The results are returned in XML and rendered in HTML back to the browser. The Java Standard Tag Library could be used to automate the process of parsing the XML and rendering the results to HTML, as well as any other type of XML- to-HTML mapping utility.

The service invocation mechanism could be any loosely coupled type of mechanism, such as JMS or HTTP/Web services. BEA WebLogic 8.1 provides automatic pooling of JMS connections, making the run-time invocation of JMS-based services extremely efficient and fast.

While the XQUERY statement could certainly be coded directly in the JSP and passed as a parameter to the "Lab Results" service, it may be better to embed the XQUERY into the service itself, providing better abstraction of presentation components from the underlying implementation details (see Listing 1).

Business Activity Monitoring
As critical business processes across the enterprise are moved into a loosely coupled, service-oriented framework on the ESB, new information can be unlocked that can have significant business value. Traditional application implementations typically capture the "end result" of a business process with business intelligence reporting tools. The ESB can capture the intermediate steps of a business process, allowing critical usage patterns to be identified and integrated into decision support systems in real time. Usage patterns can be identified to streamline business processes, or even head off problems before they cause irrevocable damage.

To illustrate the idea of business activity monitoring, let's once again consider the "Lab Referral" process outlined above. In this simple business process, what types of information could be derived?
1.  As doctors refer patients to the lab, hospital administrators may want to be notified if the number of referrals in a given week exceeds the capacity of the lab. This might allow administrators to change work schedules for the lab technicians to accommodate the peak in activity.
2.  As doctors refer patients to the lab, the CDC may want to know if the number of referrals of a certain type exceeds a certain threshold. Certain lab tests may indicate some type of new outbreak that needs to be dealt with quickly.
3.  Administrators may want to monitor overall lab processing times and perform some analysis to optimize processes in the future. For example, they may want to determine the processing time or failure rates for individual lab technicians.

These use cases highlight some core capabilities that can be embedded directly into an ESB: XML data generation, capture, aggregation, and triggering. Once these basic facilities are inplace, the portal can harness these core services to provide interactive dashboards using standard XML facilities such as XQUERY and XSLT. It's important to recognize that there is no "one-size fits all BAM solution." Just as business models differ between organizations, it would stand to reason that the BAM solutions must also differ. The ESB takes the approach of providing a core set of building blocks that provide users with the ability to build their own BAM solutions quickly and in the context of their own business models at a lower cost than dedicated stand-alone platforms. To appreciate the infrastructure behind the portal to support these features, let's take a deeper look at the five layers of BAM.

Generate
In order to support a highly flexible BAM environment, the ESB must provide quick and efficient mechanisms to generate the core data that will feed the BAM message store, poviding business users the ability to perform real-time analytics on the data. There are primarily two sources of BAM data:
1.  Business processes: As business processes execute, the ESB must be able to extract intermediate messages between services in the process, service start and stop times, and process start and stop times. The ESB must also provide the ability to determine information on processes in the aggregate, such as the total, average, and min/max values of critical process data.

This is where ESB providers differ in implementation. The ideal ESB infrastructure will allow process tracking and intermediate data capture without refactoring the existing business processes. Tracking of BAM data should be a transparent deployment-time operation to streamline the process of tracking business data.

2.  Systems on the Bus: As services are invoked within systems on the bus, there may be data internal to the systems that should be communicated out in an event-driven fashion. The ESB provides this feature based on its inherent underpinnings of JMS. As a standards-based asynchronous transport, JMS provides the best (and widely accepted) method of generating events from any system on the bus (see Figure 5).

Capture
Once data is generated from anywhere on the ESB, it must be intelligently routed to BAM Collection Points for real-time analysis. While the ESB provides an efficient means of routing through a distributed infrastructure, it must also provide efficient means of storing XML natively. Again, the concept of a native XML storage mechanism is crucial to a flexible BAM infrastructure. The schema-less design of the native XML storage provides the ability to quickly introduce new types of BAM data without the overhead of marshaling the XML into custom-defined relational tables.

Equally important is the ability to distribute the XML storage anywhere along the ESB. This allows deployment engineers to provide BAM collection points close to the point of origination, streamlining the flow of data traffic in the ESB. Imagine a case where the Lab System (from Figure 5) was generating a lot of event data. If a slow network or a WAN separated the portal and the Lab System, it would be better to capture and aggregate the data locally while also allowing aggregation of BAM data across all of the collection points.

Aggregate/Threshold/Trigger
Once the data is efficiently captured, there must be ways of aggregating the XML data to create useful reports of trends over time. XQUERY and XSLT can be used to aggregate information into summary XML documents, which can subsequently be dashboarded on a portal interface.

Once the data has been aggregated, business rules need to be executed on the aggregated data to determine if critical thresholds have been reached. Again, an XQUERY statement can be run to compare the data against critical business thresholds. If the thresholds are exceeded, the XQUERY can construct a "Business Alert" message to the ESB. Once the alert is on the ESB, it can be routed and handled in any method appropriate. For example, the alert can be routed to an e-mail service to notify one or more users of the event.

Event Processing
As Business Alerts are generated from the BAM Message Stores, they can be routed back to the ESB and handled in arbitrarily complex ways. The alerts may even kick off new, long-running business processes that run over the ESB. One interesting application of business alerts with respect to a portal is the notion of "feedback loops." Feedback loops provide the ability for the portal to modify its own behavior (within a set of constraints) based on external events.

Using the example "Lab Referral" process, imagine the case where the overall lab processing times exceeded a specific time threshold. In this case, the portal could provide additional content informing the doctor of the delay in lab processing times, and possibly even reroute the request to a secondary lab for processing.

Dashboard
Once the BAM data is in an agile XML database, the ability to view the data in a variety of ways on any presentation device (including a portal) becomes important. As executives begin to see the business value the BAM data, the requests for more visualization will increase. There are couple of challenges to consider:
1  An open, generic service architecture is important to integrate with different presentation devices. For this purpose, again the XQUERY and XML is a good choice.
2.  For rapid integration, flexible methods of moving the XML to the presentation layer are required. In the case of a portal, XSLT is a great way to quickly move XML data into HTML visualization.

Conclusion
As the notion of SOA gains traction, Enterprise Portals will be expected to rapidly integrate new services reliably with the portal, providing users with instant access to these services. The ESB will be a critical component of the portal infrastructure, providing scalablity, reliability, and the ability to harness the complexity of very large services networks. However, there are still critical infrastructure services necessary to provide integrated business process management and business activity monitoring. As ESB's gain in popularity, core features such as native XML storage and integrated business process management will be considered standard and necessary features for any portal.