Real-Time Data Integration

Continuous Real-Time Data Integration

Why batch updates are no longer efficient for real-time analysis and how HVR can enable continuous cloud data integration for your modern environment.



Transform or Die: Continuous Data Integration for the Modern Enterprise

Featuring Matt Aslett of 451 Research

Until recently, organizations have used traditional extract, transform, and load (ETL) approaches to consolidate all the data they need to perform analysis.

These solutions are designed to bulk load data from enterprise applications into a data warehouse or data lake for analysis.

In the past, this batch data integration served enterprises well by incrementally updating historical data sources during quiet periods—such as at the end of the business day. Today, many organizations perform these batch updates between one and four times a day, and are looking to reduce the update window to less than an hour.

Unfortunately, older ETL solutions were never designed to support real-time analysis. These solutions have several characteristics that make them unsuitable for this use case.

Challenges Batch Processes Present to Modern Organizations

These batch processes present several challenges to modern organizations:

  • Growing volumes of data are making it increasingly difficult to load all data within the available quiet period. This means data available for analysis can be out of synch with operational data.
  • Decisions have to be made based on the analysis of historical data alone.
  • As organizations increasingly serve customers 24×7, time windows available to update data on a batch basis have decreased. Many organizations have reached a point where they can no longer stop their operations for any amount of time to update the data warehouse.

HVR is a solution to these challenges. HVR is a data integration product designed to handle large volumes of data while performing in complex and heterogeneous environments. HVR is able to move data fast and efficiently because it makes real-time changes on the source with its log-based change data capture functionality.

 

How HVR Enables Continuous, Real-Time Data Integration

The Answer: Log-Based Change Data Capture

HVR is a purpose-built real-time data integration solution provides continuous integration through log-based change data capture. This capability allows the solution to transfer and integrate changes to the data incrementally as they occur, rather than making larger updates all at once.

As a result, it:

  • Reduces the risk that data in production databases and data warehouses will be out of sync
  • Ensures that decisions can be made based on analysis of the latest data
  • Supports 24×7 operations

The biggest benefits of log-based CDC include:

  • Minimal Impact: Log-based CDC has less impact on the database because it reads directly from the logs without directly impacting the transaction. In contrast, trigger-based CDC creates triggers on tables that require change data capture, and firing these slows down transactions.
  • Fast performance: HVR directly reads the logs on the file system allowing highly efficient change data capture, supporting large volumes of data.
  • More flexibility: log-based capture supports more data operations such as truncates, and enables support for DDL capture

real-time data integration

Learn more about HVR’s Change Data Capture Functionality and how HVR is a complete data integration solution.

HVR: Support for Disparate Systems Enables a Continuous Data Integration Strategy


supported_platforms_source_targets_data_integration

Today’s organizations want to integrate any data from any source, whether it’s stored on-premises, in the cloud, or is generated as a stream. While most organizations continue to use on-premise applications, they are also increasingly adopting software as a service(SaaS) applications in the cloud. And they’re looking to integrate these systems. Some of this need for integration is strategic, and some is driven by shadow IT.

Organizations need to be able to integrate these data sources where it makes sense and to manage that integration in a hybrid fashion. But while it is possible to use traditional ETL solutions to integrate data from cloud-based solutions, they were not designed for this purpose. Similarly, traditional ETL systems are unable to take advantage of native functionality in streaming environments.

HVR was designed from the ground up to support the environments the organization need to integrate will be more streamlined and require less manual coding than one that bolts this functionality onto a traditional data integration product.

HVR supports the most popular relational, columnar, document storage and streaming data sources, Hadoop targets and file locations. HVR can also bridge the gap between on-premise applications and the cloud as well as between clouds from different service providers.

 

Webinar: GoEasy’s Road to Real-Time Analytics

GoEasy, a Financial Services company based in Canada, provides consumer loans with a promise of fast decision making. They needed a faster way of integrating data between their SQL Server systems.

In this webinar, Anu Oladele, Director of Data Science & Business Insights—GoEasy and Joe DeBuzna, VP of Field Engineering—HVR, discuss how they worked together to architect a solution that enabled GoEasy to achieve real-time business insights and faster loan processing speeds. They discuss the business use case, their process, and share a technical overview of the solution developed as well as plans for scaling their solution.



Case Study – CACI: Real-Time Data Integration into a Data Warehouse


Data Warehouse Example

About: CACI provides information solutions and services in support of national security missions and government transformation for Intelligence, Defense, and Federal Civilian customers.

Challenge:

  • Integrate student administration data with government systems
  • Provide real-time data integration for CACI and third- party data to a data warehouse.

Databases:

  • Oracle
  • Microsoft SQL Server

Use Case: On-premises and cloud-based data integration

Whitepaper: Continuous Real-Time Integration for Real-Time Analytics

Continuous Data Integration for Real-Time Analytics

Understand Why ETL Data Integration Approaches are Insufficient and How Log-Based CDC Enables Continual Integration

Because companies once made business decisions based on reports of historical data, they could only address events after the fact. Today, business leaders are increasingly looking to improve responsiveness by using real-time analytics to understand and address business challenges in real-time. To achieve real-time analytics, businesses need to integrate data from a variety of sources in real-time.

In This Whitepaper You Will Learn

Why traditional extract, transform, and load (ETL) data integration approaches are insufficient to meet the demands of real-time analytics
How organizations can achieve business agility through real-time analytics by taking advantage of solutions that integrate data in a highly flexible manner
How to keep up with growing data volumes and shrinking processing windows
Why Log-Based Change Data Capture is an optimal solution for real-time data integration

© 2019 HVR

Try now Contact us