Home Solutions Real-Time Data Integration

Real-Time Data Integration

Continuous Real-Time Data Integration

Why batch updates are no longer efficient for real-time analysis and how HVR can enable continuous data integration for your modern environment

Until recently, organizations have used traditional extract, transform, and load (ETL) approaches to consolidate all the data they need to perform analysis. These solutions are designed to bulk load data from enterprise applications into a data warehouse or data lake for analysis.

In the past, this batch data integration served enterprises well by incrementally updating historical data sources during quiet periods—such as at the end of the business day. Today, many organizations perform these batch updates between one and four times a day, and are looking to reduce the update window to less than an hour.

Unfortunately, older ETL solutions were never designed to support real-time analysis. These solutions have several characteristics that make them unsuitable for this use case. With traditional ETL solutions, organizations typically run their business on applications that use data managed by a transaction-oriented RDBMS system such as Oracle or Ingres. At pre-determined intervals—often overnight—the organization updates the data warehouse with data processed the previous day from one or more applications.

Challenges Batch Processes Present to Modern Organizations

These batch processes present several challenges to modern organizations:

Growing volumes of data are making it increasingly difficult to load all data within the available quiet period. This means data available for analysis can be out of synch with operational data.
Decisions have to be made based on the analysis of historical data alone.
As organizations increasingly serve customers 24×7, time windows available to update data on a batch basis have decreased. Many organizations have reached a point where they can no longer stop their operations for any amount of time to update the data warehouse.

HVR is a solution to these challenges. HVR is a data integration product designed to handle large volumes of data while performing in complex and heterogeneous environments. HVR is able to move data fast and efficiently because it makes real-time changes on the source with its log-based change data capture functionality.

How HVR Enables Continuous, Real-Time Data Integration

The Answer: Log-Based Change Data Capture

HVR is a purpose-built real-time data integration solution provides continuous integration through log-based change data capture. This capability allows the solution to transfer and integrate changes to the data incrementally as they occur, rather than making larger updates all at once.

As a result, it:

Reduces the risk that data in production databases and data warehouses will be out of sync
Ensures that decisions can be made based on analysis of the latest data
Supports 24×7 operations

The biggest benefits of log-based CDC include:

Minimal Impact: Log-based CDC has less impact on the database because it reads directly from the logs without directly impacting the transaction. In contrast, trigger-based CDC creates triggers on tables that require change data capture, and firing these slows down transactions.

Fast performance: HVR directly reads the logs on the file system allowing highly efficient change data capture, supporting large volumes of data.

More flexibility: log-based capture supports more data operations such as truncates, and enables support for DDL capture

Learn more about HVR’s Change Data Capture Functionality and how HVR is a complete data integration solution.

HVR: Support for Disparate Systems Enables a Continuous Data Integration Strategy

Today’s organizations want to integrate any data from any source, whether it’s stored on-premises, in the cloud, or is generated as a stream. While most organizations continue to use on-premise applications, they are also increasingly adopting software as a service (SaaS) applications in the cloud. And they’re looking to integrate these systems. Some of this need for integration is strategic, and some is driven by shadow IT.

Organizations need to be able to integrate these data sources where it makes sense and to manage that integration in a hybrid fashion. But while it is possible to use traditional ETL solutions to integrate data from cloud-based solutions, they were not designed for this purpose. Similarly, traditional ETL systems are unable to take advantage of native functionality in streaming environments.

HVR was designed from the ground up to support the environments the organization needs to integrate will be more streamlined and require less manual coding than one that bolts this functionality onto a traditional data integration
product.

HVR supports the most popular relational, columnar, document storage and streaming data sources, Hadoop targets and file locations. HVR can also bridge the gap between on-premise applications and the cloud as well as between clouds from different service providers.





		Redshift

Learn more about Continuous Data Integration and How it Can Transform Your Business

Download our Whitepaper

Achieve Greater Business Agility with Real-Time Analytics

In this whitepaper, you will learn:

Why traditional extract, transform, and load (ETL) data integration approaches are insufficient to meet the demands of real-time analytics
How organizations can achieve business agility through real-time analytics by taking advantage of solutions that integrate data in a highly flexible manner
How to keep up with growing data volumes and shrinking processing windows