From 24 Hour To 30 Minute Data Warehouse Updates

One of HVR’s customers is a global industrial conglomerate where we have multiple deployments of our software. In this blog, we want to highlight a near real-time analytics use-case that we developed for one of their divisions. Like many large organizations, this division is a committed SAP user and, like many SAP users, makes extensive use of the SAP Business Objects Data Services ETL product as well as SAP’s ERP suite.

Moving To A Faster Data Warehouse Platform Was Not The Complete Answer

One of the division’s business and IT goals was to improve decision making by providing faster querying and reporting performance using close to real-time data instead of data updated on a daily basis. However, as they pursued this goal, the limitations of their SAP-based architecture became apparent. In an effort to change this, our customer adopted Actian Vectorwise (now called Actian Vector) as the database platform for their data warehouse and, while this had some positive impact, it revealed that many of the performance issues existed much earlier in the data pipeline.

Data Extraction Was At The Root Of The Problem

These problems occurred as a result of several different issues, for example:

  • SAP Business Objects Data Services is a traditional ETL tool with no real-time capability that generates ABAP code to extract data out of cluster and pool tables.
  • When used to extract large amounts of data from hundreds of tables, this process proved to be so slow and to impose such a performance overhead on the SAP applications that it could only be run on a daily basis.

Having identified these issues, the customer started the process of evaluating how best to solve these problems.

Solving The Problem With HVR

The customer evaluated a number of options to solve this problem including options from SAP and HVR. After an intensive evaluation process that lasted 12 weeks, the customer was convinced that the HVR product and HVR’s proactive approach to working with them to solve the problem, was the right way for them to go. The HVR solution resulted in simplified and streamlined the overall data architecture in which:

  • Data is extracted from both SAP cluster/pool tables using an ETL process that runs every 30 minutes.
  • Data is extracted in real-time from SAP transparent tables.
  • These two data sets are combined, processed and loaded into the Vectorwise data warehouse the data in which is never more than 30 minutes old.

Results And A Hidden Benefit

HVR’s solution enabled the data in the Vectorwise data warehouse to be updated every 30 minutes instead of every day. This provided the users of the system, who predominantly use Tableau for reports, queries and charts, the following benefits:

  • Faster reporting on more up-to-date data
  • Faster query times
  • 50x less data being moved across the network
  • No overhead or loss of performance to the SAP ERP application

The solution also had a hidden benefit: it revealed that the original assumptions about how far back data is being restated were incorrect resulting in data getting out of sync over time.

Summary

The business and IT goals of the project were to improve decision making by providing faster querying and reporting performance based on close to real-time data. HVR’s ability to meet these goals has led to many subsequent projects for this customer, including real-time data integration into cloud-based database technologies.

If you would like to see how our software may be able to provide your business with solutions, feel free to contact us. We’d be happy to work with you!

About Mark

Mark Van de Wiel is the CTO for HVR. He has a strong background in data replication as well as real-time Business Intelligence and analytics.

Discussion

© 2019 HVR

Live Demo Contact Us