By Cassandra Balentine
Enterprise data is increasingly important and complex. Integration tools are used to move data between systems, the next step is interpreting them.
According to Gartner, the market for data integration is composed of tools rationalizing, reconciling, semantically interpreting, and restructuring data between diverse architectural approaches, specifically to support data analytics leaders in transforming data access and delivery in the enterprise.
Data integration tools tie together data sources and best-of-breed cloud applications (apps), as well as on-premise legacy systems to achieve better business outcomes faster, offers Ben Thrift, director of engineering, Dell Boomi.
“Once data is accurate, complete, trusted, and compliant, it must be widely available to users across the enterprise in a secure and self-service fashion. This democratizes data and enables organizations to accelerate data-driven initiatives for digital transformations,” offers John Haddad, VP, product and technical marketing, Informatica.
Data Needs
Today’s business landscape is all about the data. “Every company with a data-centric strategy needs a data integration tool if they are going to succeed. Nearly every business is undergoing a digital transformation—if it hasn’t yet already transformed. We’re seeing customers across every industry—from nonprofits to major distribution and shipping companies to airlines,” says Anthony Brooks-Williams, CEO, HVR Software.
While every organization benefits from data integration, larger businesses tend to gain the most from data integration tools as they have complex and multiple data sources that need to be fed into data warehouses, clouds, or lakes. “Modern data integration helps create a unified view of enterprise data by allowing users to configure, monitor, and manage all replication tasks throughout the IT environment,” explains Itamar Ankorion, SVP and marketing manager, enterprise data integration, Qlik.
Dynamic industries with changing markets and competitive conditions also tend to have more data integration needs, as the modern data integration tools provide the ability to leverage data faster to make decisions, become more efficient, and deliver new solutions to their customers.
Haddad says it is easy to say that all organizations in all industries—in every department and every organization—benefit from intelligent data-driven collaboration. But to put a finger on it, he breaks it down, explaining that sales and marketing gain improved customer engagement, operations are optimized for better efficiency, finance gains the ability to detect and prevent fraud in real time, and manufacturing and service utilize artificial intelligence (AI)/machine learning (ML) algorithms fed with streaming machine data that is integrated with customer, product, service, and warranty data to better predict component failure and schedule optimized service and maintenance.
Data Moves
Data is an essential asset to every business. With the proper management and accessibility of these solutions, organizations are better equipped to act on situations in real time.
Ashwin Ramachandran, senior product manager, Syncsort, says data integration tools serve as the backbone of the enterprise, tasked with the responsibility of ensuring that raw data can become insight. “Enterprises constantly collect data across a variety of applications and data stores, but these stores are ultimately siloed. These assets represent latent value and potential for derived insight, but if they are unable to talk to each other, they also represent a missed opportunity. Data integration’s role is then to not only connect these disparate assets, but to restructure, convert, and harmonize them for the downstream applications like business intelligence applications, advanced analytics, or machine learning that will bring value to the business.”
“Data integration is imperative for companies to thrive in the digital age,” states Thrift. He says creating a fabric of connectivity is fundamental to unlocking productivity and that means connecting applications, people, devices, and processes and breaking down data silos in today’s hybrid IT environment. “Modernizing the IT infrastructure is a priority for many organizations that want to operate faster, reduce costs, and drive innovation.”
Historically, data integration tools eliminate siloed, duplicate, or inconsistent datasets to help improve business performance and make information more readily available. Unstructured information from outside of a firewall—such as social media and internet data and other streaming data sources like the Internet of Things (IoT) add to the complexity, admits Jake Freivald, VP, Information Builders. “These data points contain critical information and can change how businesses view their data if looked at in tandem with traditional enterprise information. That’s why it is even more imperative that organizations break down data silos and integrate information to drive valuable and actionable insights across the entire organization.”
Beyond the basic use of data integration tools, they are also a primary vehicle for businesses looking to adapt to the cloud. “Data integration—specifically replication, is the most useful technology to move data to the cloud,” offers Brooks-WIlliams.
Brooks-Williams adds that the cloud is an obvious growth area and companies are turning to the cloud for savings and elasticity, which means that companies have access to a larger volume of their data—big data. “Integration tools need to be able to handle that volume as well as the various cloud platforms coming to market.”
Kim Kaluba, senior product manager, SAS, says data integration tools are currently pigeon-holed into the functions of moving and transporting data from one place to another. However, that is the easy part—integration is difficult. “One of the main reasons data integration is difficult is because data is everywhere across the organization—in different systems, with different schemas, and with different data dependencies,” she offers.
As the speed of business increases, those with access to real-time insights and can move data at the pace of change maintain a competitive advantage, says Ankorion. “The data-driven economy demands that organizations across every industry efficiently and consistently unlock more value from their data through a constant flow of insights to business users. Modern data integration software is able to make this information available for cloud and analytics in an agile and real-time approach so that enterprise information is available and meaningful.”
Evolution
The way businesses collect, store, manage, and leverage data is constantly evolving. Therefore, the tools that support these processes are as well.
A myriad of emerging new technologies generate new forms of data. “These will need to be integrated into our customers’ business processes including new applications—and microapplications—that leverage blockchain, augmented and virtual reality, voice, hand gestures, and 5G,” shares Thrift.
Data integration will need to evolve to support AI and ML processes in order to assist with the integration of data based on past human decisions. “The tools will need to incorporate how a human—or human-made process—integrated data in the past, learn from these decisions, and apply the learnings to the data across the organization. The removal of highly manual efforts will transition and data will eventually be able to integrate itself based on what it has learned, sharing its learnings with machines and man,” says Kaluba.
Thrift adds that AI is only as good as the data available to analyze. “An integration platform is essential for delivering the requisite data pulled for a variety of sources. However, it’s a two-way street. After the data is analyzed, it’s equally as important to return it to applications including ERP and CRM systems or business intelligence dashboards—or it would just be another data silo.”
Data integration tools should provide social collaboration methods to facilitate workstream communications and reduce bottlenecks. When coupled with AI/ML automation and guided user interactions, process efficiency improves exponentially, offers Haddad.
Haddad notes that it is clear AI needs data, but it is important to remember that data also needs AI. “With the incredibly volume and variety of data, and the intensive labor involved with ingesting, integrating, and cleansing all of that data, it is no surprise that there is simply more work than can be done by humans. The only way to address the growing burden of manual tasks is with intelligent automation.”
Freivald points out that enterprises already use a variety of methods using ETL, search technology, operation data access, and process integration to link together systems to support business intelligence and analytics. However, as data architectures become more sophisticated and trends like big data and IoT continue to proliferate, these methods—and the solutions that support them—are rapidly changing. As a result, more modern integration tools emerge to support the growth of this unstructured information. “Importantly, as these technologies develop, they also converge. Data integration as a low-level topic—microservices, connectors, big data, SQL-on-Hadoop data quality, and MDM—gets subsumed into a higher level vision of data integration in which these technologies are put in service of broader business goals that require a complete, dependable, flexible, and high-quality view of a business derived from every data source. Modern data integration increasingly becomes a high-level topic that encompasses all the rest.”
Brooks-Williams agrees, adding that companies are generally looking to data integration tools to handle increasingly complex data environments. “Trends like AI and the IoT add to further proliferation of data—both the supply of and the demand for data. With IoT, a lot of data is generated at the edge, on the factory floor at a jet’s wind turbines, for example. That data gets moved into a central area for deeper consolidated analysis, which impacts bandwidth and latency, which then further improves the impact of the AI. In both cases, data integration tools need to accommodate this complexity, which only increase in the future.”
Organizations are increasingly look to invest more time and resources into getting the most value out of their data, not necessarily integration itself. “This means enabling more citizen integrators through simplified user experiences and machine learning-guided development,” says Ramachandran. This also means that data integration tools are expected to leverage metadata in order to build the data integration pipeline. “Furthermore, as the growth of the cloud and enterprise migration to the cloud accelerates, data integration tools are continually tasked with making sense of increasing varieties and volumes of data under stricter service-level agreements.
Data integration tools are expected to become more autonomous, seamlessly running behind the scenes. “Expectations, especially of decision makers are that the data will just be there. This is already happening—an increasing number of IT-savvy people are sitting around the executive table that want to run their own analysis or create their own dashboards, and they want instant access to the company’s edge for the factory floor. They don’t want to submit a request to IT to configure a solution or grant access to run on the source system. Businesses want real-time data analysis when they want it,” states Brooks-Williams.
Ankorion feels that the shift to modern data architectures that contain cloud, data lakes, and streaming technologies requires a new approach to data integration. Traditional technologies like ETL are outdated and ineffective in addressing the agility, efficiency, and latency requirements needed. He says data integration tools have moved away from manual scripting to self-service automation and from batch to real time. “The pace of business demand is in minutes, not days, making it vital to have information delivered in real time. Data integration solutions based on change data capture technology allow businesses to access data quickly and support emerging data lakes, streaming, and cloud platforms as well.”
“The speed at which organizations can gather insights is a critical factor for success,” says Freivald. “Only when information is integrated, cleansed, and mastered in an automated way can businesses ensure that users are empowered with timely, accurate, and complete data for better decision making.”
Winning with Data
Data integration tools are critical for accessing, integrating, cleansing, mastering, and governing enterprise data for many strategic initiatives. “As our world becomes increasingly dependent on data, organizations face a significant challenge managing and making sense of massive amounts of data across a myriad of disparate sources,” says Haddad. He says all of this data must be integrated for data scientists and business analysts to turn data into insights and action.
As data becomes increasingly complex, businesses look to management and integration tools to make sense of it all. While larger organizations benefit the most, any modern business can find success with better data access and intelligence.
Nov2019, Software Magazine
