Data is king in today's information-driven economy, which is why organizations are willing to spend tens or even hundreds of thousands of dollars on data integration frameworks and applications. These organizations understand two critical truths: they have yet to capitalize on the potential business value stored in relational databases, EDI, flat files, and XML-based systems; and they must seamlessly connect with customers, suppliers, and business units - all of which may store and process data in different formats - to remain competitive.

Open standards-based technologies like XML promise to unify enterprise data and enable advanced Web Services and SOA. But while XML may be standards-based, most existing enterprise data isn't, nor is it easily extensible. Complicating matters further is the fact that many large enterprises rely on EDI systems to exchange business information with their partners. EDI is generally not interoperable with other systems.

XML gives organizations the ability to leverage existing systems and increase their usefulness by adding the flexibility required for real-time data exchange. Furthermore, it can facilitate the exchange across departmental and geographical boundaries and through system and programmatic constraints. But XML is not, in and of itself, a cure-all for data integration. Successfully integrating XML with other data formats requires applications that integrate system interfaces and map between data structures.

No Two Data Formats Are Alike
There are various formats for storing and exchanging data in use today and the fact that no two are alike add to the challenges of information accessibility and data integration. Let's take a look at the most popular formats and what makes each of them unique.

Relational databases
This is the dominant storage mechanism for structured enterprise data, an efficient means of storing, searching for, and retrieving information from large collections of data. Relational databases specialize in relating individual data records grouped by type in tables. Records can be joined together as needed using SQL and presented to business users as meaningful information.

The technology is mature, and so the sheer volume of information stored in relational databases and the number of hours invested developing structures and specialized systems make them valuable assets. But their flexibility remains limited when it comes to integrating with other systems. Also, the differences between major commercial implementations can make data integration difficult.

Electronic data interchange (EDI)
Long before the Internet made business-to-business electronic trade a standard practice, there was EDI. This set of widely used formats allows for the electronic exchange of information, and was developed to enable independent organizations to reliably exchange various types of data, including purchase orders, invoices, shipping notices, medical and insurance claims, and the like.

EDI has proven valuable for supplanting paper-based business processes. It has also enabled organizations to exchange large amounts of information with partners and other companies quickly in a fairly standardized interaction.

Many larger organizations have substantial investments in EDI technology. But smaller companies have been less likely to invest in EDI technology partly because the implementations are infrastructure-, training-, and maintenance-intensive. To partner with larger enterprises, however, these smaller companies must find a way to handle EDI-based business transactions otherwise they risk missing lucrative business opportunities.

Flat files
Many legacy enterprise systems and popular applications, including accounting, banking, CRM, and spreadsheet software, support flat-file formats. They are frequently used as an interchange format for transferring information between applications, including databases. However, flat files generally require additional processing to interoperate with common data formats such as EDI or XML and can be cumbersome when dealing with large amounts of information.

XML
In the world of information interchange, the use of XML has grown steadily, and it now plays a central role in data management, transformation, and exchange. It has gotten widespread support from leading software, server, and database vendors, and has become the language of choice for lowering the cost of processing, searching, exchanging, and reusing data and information.

The openness of XML allows it to be exchanged between virtually any hardware, software, or operating system, and allows for information interchange without restriction. XML and XML-based technologies such as XML Schema, XSL, WSDL, and SOAP are all open standards that can be used in conjunction with any programming language or platform. Thus, XML technologies and Web services can be used on and between virtually any combination of database, application runtime, and operating system - a characteristic that's essential for integration with heterogeneous systems.

Developing Data Integration Applications
Data integration applications and frameworks offer the potential to unify business data while capitalizing on the particular strengths of relational databases, EDI, flat files, and XML systems, and there are several approaches an organization can take to developing these solutions - each with its own advantages and disadvantages.

Middleware or server-based platforms, for example, tend to be proprietary, closed solutions that are extremely expensive to purchase, implement, and maintain. For some the more viable option is to build customized data integration applications that are flexible enough to adapt to changes and don't force businesses to lock into a particular system vendor. Yet despite the advantage of flexibility, customized data integration applications are often extremely complex, expensive, and time-consuming to develop.

Alternative, More Cost-Effective Approaches
With complicated ESB or EAI installations, expensive server deployments, hundreds of hours of programming, consulting and training; clearly the cost and complexity of an effective data integration application can add up, and fast. But there are alternatives that can simplify the development of data integration applications and meet the needs of individual integration challenges.

When timeliness and budget make byzantine enterprise solutions impractical, data integration development tools like Altova MapForce can be used to build customized applications very quickly and at a fraction of the cost. Such tools are easy to use and sell for under $1,000 - far less than an ESB or EAI solution. Most companies with specific data integration requirements will find using these tools a more manageable and cost-effective investment.

If a data integration tool is the right choice for your organization, there are certain characteristics you should look for to be certain the tool is up to the task.

Visual data integration and mapping
The advantages of a visual data integration tool can't be overstated. Because the tool reads and writes all the native file formats, from relational databases and EDI to flat files and XML, and even established Web services, one well-rounded programmer is all it takes to accomplish what otherwise would require a specialized team of experts. A visual interface lets the developer design a data mapping without having to understand the specific details of how to programmatically access the data formats that are being integrated. (see sidebar)

Multiple sources and targets
Mapping shouldn't be limited to one-to-one relationships. Look for a tool that lets developers mix multiple sources and multiple targets to map any combination of different data sources and targets in a mixed environment.

A data integration tool should also provide a comprehensive library of advanced data processing functions, and let developers specify mappings based on conditions - Boolean logic, string operations, filters, mathematical computations, and so on. It should also let developers save complex functions for use at other stages of data processing to save time and effort.

Automatic generation of royalty-free code
The ability to auto-generate code in various languages (Java, C#, C++, XSLT, XQuery) means you get reliable code faster and with less effort. Instead developers can focus on the all-important business logic of the application while leaving the generation of low-level infrastructure code to the tool. Make sure the code that the tool generates can be used royalty-free and doesn't require any proprietary deployment adapters.

Furthermore, some tools, such as Altova MapForce, also have the capability to process transformations internally, letting you preview the output of your mapping and ensure accuracy before generating code. This feature is also useful for doing periodic or light-duty integration tasks on the fly.

A Way To Achieve Data Integration Today
The value of an organization's business information is directly proportional to its ability to share the information internally and externally, which is why organizations can no longer afford to let data storage and exchange systems operate in a vacuum.

Fortunately, the arrival of simple cost-effective data integration and development tools means that sharing data with customers, partners, and business units doesn't have to cost companies a small fortune for a largely unnecessary enterprise solution.