By SW Staff
Data integration tools allow enterprises to combine technical and business processes to merge data from siloed sources. This creates a data source that enables businesses to understand, monitor, cleanse, transform, and deliver meaningful information. These tools also facilitate a better understanding between corporations and their IT department.
According to Gartner’s 2016 Gartner Magic Quadrant for Data Integration Tools research, at the end of 2015, the data integration tool market was worth approximately $2.8 billion in constant currency. This is an increase of 10.5 percent since the end of the prior year.
Data integration tools improve enterprise data management, but they require modern accessibilities to combat challenges. Gartner research lists data integration tool market leaders as Actian, IBM, Informatica, Oracle, SAP, SAS, and Talend.
Enterprise Data management
Data integration tools provide enterprise data management while allowing organizations to validate data from multiple sources. It is then transformed and converted into a common format for decision making and analysis. Data integration tools allow the synchronization of business data across multiple systems and databases.
These tools are critical for successful enterprise data management. “They’ve been the key to blending data from multiple silos across the organization into one cohesive view with various degrees of success and accuracy,” says Matthew Magne, global product marketing manager, SAS.
Deepak Singh, CTO, Adeptia, says “data is the lifeblood of any organization and data integration and management allows businesses to keep a handle on their data to efficiently run their business operations.”
Beth Adams, product marketing manager, Information Builders, adds that data integration tools help users focus on data and less on structural and technical details. Additionally, these tools streamline cloud-based CRM applications.
“Around 70 percent of corporate data still originates from or stored on mainframe computers—without data integration solutions, that would be inaccessible,” says Tendu Yogurtcu, GM, Syncsort. Yogurtcu explains that these tools master the data environment and allow users to analyze and find patterns to help users solve problems.
Businesses today use modern data platforms that provide flexibility and scalability to propel themselves into a data-driven transformation. “The more current data management platforms brought a new level of sophistication and automation to data integration—making the steps involved repeatable, extensible, and scalable. This allows for more agile collaboration between teams and enables faster updates, which is essential for companies that need to scale and keep with new data demands,” points out Isabelle Nuage, big data solutions director, Talend.
Integration Requirements
Business users once relied on IT to create integration connections because data integration was the exclusive domain of the IT staff and developers. Singh says business users are now savvy enough to do basic, simpler data integration tasks without it.
User interfaces are now web based and provide visibility and collaboration for users. Singh says early generation data integration tools were desktop applications that required developer tools to design and create integration dataflows. These early tools were also database focused and designed to extract, transform, and load—or ETL—data. Now, data integration tools access data in various formats and support real-time web services.
Modern data integration tools have powerful data mapping capabilities. “Modern data integration tools should support rich data mapping and transformation capabilities to allow data merging from multiple sources into one target, splitting data and many-to-many mapping,” says Singh.
Magne explains that other requirements for modern data integration revolve around ingesting increasingly exogenous third party data streams, the velocity of those streams, and blending traditional relational technologies with emerging open source technologies like Apache Hadoop.
Yogurtcu summarizes the three most important requirements for modern data integration as agility, all-inclusiveness, and ability to insulate organizations from rapidly evolving environments. “Data integration solutions must be able to move massive amounts of data from disparate sources, both batch and streaming, on premise, or in the cloud,” says Yogurtcu. Solutions insulate applications from the technology stack and integrate with highly scalable and distributed frameworks. Enterprises look for user friendly applications to ramp up emerging platforms without the need for complex coding skills to leverage existing teams.
Solving Problems
Data professionals need to access data quickly for decision making from more sources at greater speeds. To combat this challenge, Magne says that business users need self-service access to the data so that IT is free for other activities. “Emerging technologies, like Hadoop, create a gap in skills needed to manage data,” he adds. Companies are relieved when the required skills for data integration tasks are minimized.
Managing data where it lives is an effective tool. Generating a cleansed view of data without relocation is complete by data virtualization. Self-service data preparation tools provide business users with access to data. “Plain, old-fashioned ETL tools can be brought to bear to do the heavy batch lifting and blending of data that make up the lion’s share of organizations data integration tools,” says Magne.
Multiple ecosystems sometimes lead to difficulties within infrastructure. “Integration technologies can smooth out the difference among multiple ecosystems, like SAP-based infrastructure, Relational Databaase Management Systems (RDBMS)-based infrastructure, and Hadoop-based infrastructure,” says Adams. Some data integration tools work across a variety of different versions and technologies inside the ecosystem. This relieves cluster management, versioning, module installation, and other processes that drain productivity.
Available Tools
A variety of tools assist in data integration.
The Adeptia Integrations Suite features a web browser-based interface for creating, managing, and running data integration flows. After it is configured and setup by IT, it is intended for business use. Singh says Adeptia’s data integration tools supports the widest range of data formats and protocols to handle all situations. It supports cloud-based applications, on premise applications, and a variety of databases for accessibility ease and deliverable data. Its process engine provides flexibility for designing complex data flows including notifications and data reprocessing. “Adeptia is the only data integration tool to allow user tasks to be designed in the data flow to review the data, make decisions, make corrections, and send the data forward to next steps,” explains Singh. With this process, Adeptia handles human-to-system flows and system-to-system data flows.
Information Builders offers the iWay Big Data Integrator to provide an interface that marshals Hadoop resources and standards to natively ingest, transform, clean, and manage data lakes. Informatica’s Omni-Gen generates artifacts needed for integration using business definitions. This includes data cleansing and MDM to ensure rapid cycle times based on business input. The iWay Service Manager allows businesses to create, manage, and respond to internal and business to business (B2B) events. The company also features the B2B Integration and Trading Partner Manager to manage trading partner agreements and activities. “This product dramatically improves the way partner transactions, as well as those performed over external exchanges, are planned, executed, and tracked,” shares Adams.
SAS features SAS Data Integration Studio, a self-service data preparation for business users using an SAS Data Loader, SAS Event Stream that streams data, and SAS Federation Server for data virtualization tools that share common metadata with analytics and reporting solutions like SAS Visual Analytics for simpler reporting and integration. “Whether it’s running in-stream, in-database, in-memory, inside Hadoop, or in the cloud, SAS helps organizations manage data where it lives, pushing processing to the database for improved performance, and governance,” says Magne. SAS also provides governance and metadata lineage tools for calculating data relations and impacts.
Syncsort’s DMX-h to allows organizations to access and integrate large volumes of data from legacy mainframes and RDBMs while providing a cross platform metadata lineage. Yogurtcu says this software is essential for audit and compliance among heavily regulated industries like banking, insurance, and healthcare. DMX-h allows users to access mainframe data and grasp data insights quickly for business agility and innovation. Its purpose is to bridge the gap between legacy systems and emerging big data environments while liberating data from mainframes for advanced analytics. “With increasing numbers of connected devices, the need for data governance and security will continue to grow,” says Yogurtcu.
Talend combines all of its products into a set of easy-to-use tools for batch, data, or application integration and data management on the cloud or on premises. “Talend Data Fabric is a modern, open, and flexible data architecture that solves for the continuously evolving market challenges companies face today as they drive towards becoming more data-driven organizations,” says Nuage. The company’s products have over 900 components and connect and support big data and IoT scenarios. The software generates native code for Spark and MapReduce with no overhead on Hadoop clusters and has a visual interface. “It helps companies increase data project agility and speed time to market with a solution that is up to seven times faster and one fifth the price of legacy, proprietary integrations solutions,” she adds.
Managing Data
Data integration tools allow corporations to collect and merge into a system that is easy to comprehend. Modern data integration processes alleviate IT’s work because the processes are simple enough for business users to handle. These tools unite multiple ecosystems and solve difficulties within the corporation’s infrastructure.
Feb2017, Software Magazine
