SOA

There has been a lot of buzz and hype, some factual, some not so well-founded, surrounding the opportunities presented by Service-oriented Architectures (SOA) and their implementation as Web services. Analysts have predicted, pundits have professed, professors have lectured. And companies have scurried to sell what they had as SOA products, often missing the point that SOA is not a product.

SOA is about bridging the gap between business and IT through a set of business-aligned IT services using a set of design principles, patterns, and techniques. This article discusses the highlights of service-oriented modeling and architecture, and the key activities needed for the analysis and design required to build a Service-Oriented Architecture (SOA). It stresses the importance of addressing the techniques required for the identification, specification and realization of services, their flows and composition, as well as the enterprise-scale components needed to realize and ensure the quality of services required of an SOA.

A huge demand exists for the development and implementation of SOAs. Gartner has predicted that by 2008, for example, more than 60 percent of enterprises will use SOA as a "guiding principle" when creating mission-critical applications and processes. But if SOA is not just about the products and standards that help realize it-through Web services, for example-then what additional elements are needed to realize an SOA?

Start with Modeling
Service-oriented modeling requires additional activities and artifacts that are not found in traditional object-oriented analysis and design (OOAD). Additional important considerations exist that need to be taken into account. For one thing, current OOAD methods do not address the three key elements of an SOA: services, flows, and components realizing services. You also must be able to explicitly address the techniques and processes required for the identification, specification and realization of services, their flows and composition, as well as the enterprise-scale components needed to realize and ensure the quality of services required.

Second, a paradigm shift needs to occur in order to appreciate the distinct requirements of the two key roles in an SOA: the service provider and service consumer. Third, applications assumed to be built for one enterprise or line of business must now aspire to be used in a supply chain and be exposed to business partners who might compose, combine, and encapsulate them into new applications. This is the notion of the service ecosystem.

This is a slight step up from just "distributed objects." SOAs need to be about the value created through the network effect. An example would be when parties leverage a combination of Amazon.com with Google's search services and combine them with eBay services to build their own hybrid solutions. Or when a travel agency reaches deep into an airline reservation system and coordinates it with a rental car agency and hotel chain, updating their records and sending the itinerary to your electronic organizer.

It's also critical to understand what you are aiming to produce: what is an SOA, and what does it look like? After defining the notion and concepts behind an SOA, you need to understand the layers of an SOA. You also need to know to record key architectural decisions about each layer that help you in building an SOA blueprint for your project, line of business, enterprise-wide effort, or value-chain that you are trying to integrate, and come up with a set of services, flows, and components that implement the SOA.

Whatever the application, you need much more than just good tools and standards to successfully create an SOA. You need some prescriptive steps to support your SOA life cycle; techniques for the analysis, design, and realization of services, flows, and components. Therefore, for anyone interested in enterprise application development, it's crucial to understand the detailed steps involved in service-oriented modeling and architecture.

Conceptual SOA Model
A valid SOA conceptual model is based on an architectural style that defines an interaction model between three primary parties:

• The service provider, who publishes a service description and provides the implementation for the service
• A service consumer, who can either use the uniform resource identifier (URI) for the service description directly, or can find the service description in a service registry and bind and invoke the service.
• The service broker, to provide and maintain the service registry (although nowadays public registries are not in vogue.)

A meta-model showing these relationships is depicted in Figure 1.

An SOA is an enterprise-scale IT architecture for linking resources on demand. In an SOA, resources are made available to participants in a value net, enterprise, line of business (typically spanning multiple applications within an enterprise or across multiple enterprises). It consists of a set of loosely-coupled, business-aligned IT services that collectively fulfill an organization's business processes and goals. The architecture style defining an SOA describes a set of patterns and guidelines for services that, because of the separation of concerns between description, implementation, and binding, provide unprecedented flexibility in responsiveness to new business threats and opportunities.

A service is a software resource (discoverable) with an externalized service description. This service description is available for searching, binding, and invocation by a service consumer. The service provider realizes the service description implementation and also delivers the quality of service requirements to the service consumer. Services should ideally be governed by declarative policies and thus support a dynamically re-configurable architectural style.

You can choreograph these services into composite applications and invoke them through standard protocols, as shown in Figure 2.

Business agility is gained by IT systems that are flexible, primarily by separation of interface, implementation, and binding (protocols) offered by an SOA, allowing the deferral of the choice of which service provider to opt for at a given point in time based on new business requirements, functional and non-functional (for example, performance, security, and scalability.)

You can reuse the services across internal business units or across the value chains among business partners in a fractal realization pattern. Fractal realization refers to the ability of an architectural style to apply its patterns and the roles associated with the participants in its interaction model in a composite manner. You can apply it to one tier in an architecture and to multiple tiers across the enterprise architecture. Among projects, it can be between business units and business partners within a value chain in a uniform and conceptually scalable manner.

Another facet that must be addressed is service-oriented modeling-a service-oriented analysis and design (SOAD) process for modeling, analyzing, designing, and producing an SOA that aligns with business analysis, processes, and goals. You first need to look at what you intend to build: namely an SOA and its layers. Then you can address the main activities and techniques needed to create an SOA.

Let's assume that you need to migrate a portion of a banking line of business that has a self-service accounting system to an SOA. In order to migrate to the SOA, you need some additional elements that go beyond service modeling. These include:

• Adoption and maturity models. Where is your enterprise at in the relative scale of maturity in the adoption of SOA and Web Services? Every different level of adoption has its own unique needs.
• Assessments. Do you have some pilots? Have you dabbled into Web services? How good is your resulting architecture? Should you keep going in the same direction? Will this scale to an enterprise SOA? Have you considered everything you need to consider?
• Strategy and planning activities. How do you plan to migrate to an SOA? What are the steps, tools, methods, technologies, standards, and training you need to take into account? What is your roadmap and vision, and how do you get there? What's the plan?
• Governance. Should existing API or capability become a service? If not, which ones are eligible? Every service should be created with the intent to bring value to the business in some way. How do you manage this process without getting in the way?
• Implementation of best-practices. What are some tried and tested ways of implementing security, ensuring performance, compliance with standards for interoperability, designing for change?

In addition to identification, specification, and realization, the service-oriented modeling approach includes the techniques required for deployment, monitoring, management, and governance required to support the full SOA life cycle.

So the above discussions on migration to SOA and the additional activities after realization deserve an article of their own, which I will get to in a subsequent column in this series. For now, let's assume that you scoped the project, and a focal point for transformation of existing systems or services to a new set of systems and services has been defined. You can now start service-oriented modeling to build your service-oriented architecture.

An Architectural Template
An abstract view of SOA depicts it as a partially-layered architecture of composite services that align with business processes. Figure 3 depicts a representation of this type of architecture. The relationship between services and components is that enterprise-scale components (large-grained enterprise or business line components) realize the services and are responsible for providing their functionality and maintaining their quality of service. Business process flows can be supported by a choreography of these exposed services into composite applications. An integration architecture supports the routing, mediation, and translation of these services, components, and flows using an Enterprise Service Bus (ESB). The deployed services must be monitored and managed for quality of service and adherence to non-functional requirements.

For each of these layers, you must make design and architectural decisions. Therefore, to help document your SOA, you might want to create a document consisting of sections that correspond to each of the layers. Table 1 contains a template for your SOA architecture document.

SOA Layer Descriptions
Now, let's describe each layer in greater detail and discuss the composition of each of these layers.

Layer 1: Operational systems layer. This consists of existing custom built applications, otherwise called legacy systems, including existing CRM and ERP packaged applications, and older object-oriented system implementations, as well as business intelligence applications. The composite layered architecture of an SOA can leverage existing systems and integrate them using service-oriented integration techniques.

Layer 2: Enterprise components layer. This is the layer of enterprise components that are responsible for realizing functionality and maintaining the QoS of the exposed services. These special components are a managed, governed set of enterprise assets that are funded at the enterprise or the business unit level. As enterprise-scale assets, they are responsible for ensuring conformance to SLAs through the application of architectural best practices. This layer typically uses container-based technologies such as application servers to implement the components, workload management, high-availability, and load balancing.

Layer 3: Services layer. The services the business chooses to fund and expose reside in this layer. They can be discovered or be statically bound and then invoked, or possibly, choreographed into a composite service. This service exposure layer also provides for the mechanism to take enterprise scale components, business unit specific components, and in some cases, project-specific components, and externalizes a subset of their interfaces in the form of service descriptions. Thus, the enterprise components provide service realization at runtime using the functionality provided by their interfaces. The interfaces get exported out as service descriptions in this layer, where they are exposed for use. They can exist in isolation or as a composite service.

Level 4: Business process composition or choreography layer. Compositions and choreographies of services exposed in Layer 3 are defined in this layer. Services are bundled into a flow through orchestration or choreography, and thus act together as a single application. These applications support specific use cases and business processes. Here, visual flow composition tools such as IBM WebSphere® Business Integration Modeler or Websphere Application Developer Integration Edition can be used for the design of application flow.

Layer 5: Access or presentation layer. Although this layer is usually out of scope for discussions around an SOA, it is gradually becoming more relevant. I depict it here because there is an increasing convergence of standards, such as Web Services for Remote Portlets Version 2.0 and other technologies, that seek to leverage Web services at the application interface or presentation level. You can think of it as a future layer that you need to take into account for future solutions. It is also important to note that SOA decouples the user interface from the components, and that you ultimately need to provide an end-to-end solution from an access channel to a service or composition of services.

Level 6: Integration (ESB). This layer enables the integration of services through the introduction of a reliable set of capabilities, such as intelligent routing, protocol mediation, and other transformation mechanisms, often described as the ESB. Web Services Description Language (WSDL) specifies a binding, which implies a location where the service is provided. On the other hand, an ESB provides a location independent mechanism for integration.

Level 7: QoS. This layer provides the capabilities required to monitor, manage, and maintain QoS such as security, performance, and availability. This is a background process through sense-and-respond mechanisms and tools that monitor the health of SOA applications, including the all-important standards implementations of WS-Management and other relevant protocols and standards that implement quality of service for an SOA.

Combining Top-Down with Bottom-Up
When developing an SOA, you also need to be able to combine a top-down, business-driven approach with a bottom-up approach, leveraging legacy investments. The service-oriented modeling approach provides modeling, analysis, design techniques, and activities to define the foundations of an SOA. It helps by defining the elements in each of the SOA layers and making critical architectural decisions at each level. It does so using a combination of a top-down, business-driven manner of service identification coupled with a stream of work that conducts service identification through leveraging legacy assets and systems.

In this way, high-level business process functionality is externalized for large-grained services. Smaller-grained services-those that help realize the higher level of services-are identified by examining the existing legacy functionality and deciding how to create adaptors and wrappers, or componentizing the legacy systems to externalize the desired functionality often locked within the system. Finally, using goal-service modeling, you use a cross-sectional approach to cut down the sheer number of candidate services that might already be identified.

A more judicious approach would be to first do top-down, then goal-service modeling, and finally bottom-up legacy analysis of existing assets. The message is: the faster you scope the project down to a manageable and realistic set, the sooner you can realize value by focusing on key services to expose with service descriptions that form the cornerstone of the SOA.

This combination of functional business aspirations and leveraging of existing investments in legacy systems provide a potent solution to organizations that want to have quick wins and migrate their enterprise to a modern SOA. Consolidation of software applications through service-oriented integration thus becomes possible.

Service-oriented integration is an evolution of Enterprise Application Integration (EAI) in which proprietary connections are replaced with standards-based connections over an ESB notion that is location transparent and provides a flexible set of routing, mediation, and transformation capabilities.

Analysis and Design of Services
Unlike with your comfortable world of objects, you need to take into account two perspectives in an SOA, that of the service consumer and service provider. (The service broker is currently not mainstream and will be covered in a later venue.)

The design strategy for an SOA does not start from the bottom-up as is often the case with a Web services-based approach. You must remember that SOA is more strategic and business-aligned. Web services are a tactical implementation of SOA. A number of important activities and decisions exist that influence not just integration architecture but enterprise and application architectures as well. They include the activities from the two key views of the consumer and provider described in Figure 4. This figure shows the activities that are typically conducted by each of the roles of provider and consumer. Note that the provider's activities are a superset of the consumer's activities. For example, the provider would also be concerned with service identification, categorization, etc.

In many cases, the differentiation of the roles comes from the fact that the consumers specify the services they want, often search for it, and once they are convinced of the match between the specification of the service they are looking for, and that provided by a service provider, they bind and invoke the service as needed.

The provider, in turn, needs to publish the services they are willing to support; both in terms of functionality and most importantly in terms of the QoS that consumers will require. This implicit contract between consumer and provider might mature into an explicit contract in terms of SLAs, negotiated either electronically or through business and legal venues.

The activities described above can be depicted to flow within the service-oriented modeling and architecture method, as shown in Figure 5.

The process of service-oriented modeling and architecture consists of three general steps: identification, specification and realization of services, components and flows (typically, choreography of services).

Service identification. This process consists of a combination of top-down, bottom-up, and middle-out techniques of domain decomposition, existing asset analysis, and goal-service modeling. In the top-down view, a blueprint of business use cases provides the specification for business services. This top-down process is often referred to as domain decomposition, which consists of the decomposition of the business domain into its functional areas and subsystems, including its flow or process decomposition into processes, sub-processes, and high-level business use cases. These use cases often are very good candidates for business services exposed at the edge of the enterprise, or for those used within the boundaries of the enterprise across lines of business.

In the bottom-up portion of the process or existing system analysis, existing systems are analyzed and selected as viable candidates for providing lower cost solutions to the implementation of underlying service functionality that supports the business process. In this process, you analyze and leverage API's, transactions, and modules from legacy and packaged applications. In some cases, componentization of the legacy systems is needed to re-modularize the existing assets for supporting service functionality.

The middle-out view consists of goal-service modeling to validate and unearth other services not captured by either top-down or bottom-up service identification approaches. It ties services to goals and sub-goals, key performance indicators, and metrics.

Service classification or categorization. This activity is started when services have been identified. It is important to start service classification into a service hierarchy, reflecting the composite or fractal nature of services: services can and should be composed of finer-grained components and services. Classification helps determine composition and layering, as well as coordinates building of interdependent services based on the hierarchy. Also, it helps alleviate the service proliferation syndrome in which an increasing number of small-grained services get defined, designed, and deployed with very little governance, resulting in major performance, scalability, and management issues. More importantly, service proliferation fails to provide services, which are useful to the business, that allow for the economies of scale to be achieved.

Subsystem analysis. This activity takes the subsystems found above during domain decomposition and specifies the interdependencies and flow between the subsystems. It also puts the use cases identified during domain decomposition as exposed services on the subsystem interface. The analysis of the subsystem consists of creating object models to represent the internal workings and designs of the containing subsystems that will expose the services and realize them. The design construct of the subsystem will then be realized as an implementation construct of a large-grained component realizing the services in the following activity.

Component specification. In the next major activity, the details of the components that implement the services are specified:

• Data
• Rules
• Services
• Configurable profile
• Variations

Messaging and events specifications and management definition occur at this step.

Service allocation. This consists of assigning services to the subsystems that have been identified so far. These subsystems have enterprise components that realize their published functionality. Often you make the simplifying assumption that the subsystem has a one-to-one correspondence with the enterprise components. Structuring components occurs when you use patterns to construct enterprise components with a combination of mediators, façade, rule objects, configurable profiles, and factories.

Service allocation also consists of assigning the services and the components that realize them to the layers in your SOA. Allocation of components and services to layers in the SOA is a key task that require the documentation and resolution of key architectural decisions that relate not only to the application architecture but to the technical operational architecture designed and used to support the SOA realization at runtime.

Service realization. This step recognizes that the software that realizes a given service must be selected or custom-built. Other options that are available include integration, transformation, subscription and outsourcing of parts of the functionality using Web services. In this step you make the decision as to which legacy system module will be used to realize a given service and which services will be built from the "ground-up". Other realization decisions for services other than business functionality include: security, management and monitoring of services.

In reality, projects tend to capitalize on any amount of parallel efforts to meet closing windows of opportunity. Therefore, I recommend conducting three streams in parallel.

Top-down domain decomposition-process modeling and decomposition, variation-oriented analysis, policy and business rules analysis, and domain specific behavior modeling using grammars and diagrams-is conducted in parallel with a bottom-up analysis of existing legacy assets that are candidates for componentization (modularization) and service exposure. To catch the business intent behind the project and to align services with this business intent, goal-service modeling is conducted.