1TB of Data Changes Per Day? No problem. How HVR enables high volume data replication from systems such as SAP into Snowflake   Snowflake is the first data warehouse built for the cloud. With storage separated from compute resources, it…

The Major Types of Data Integration (& When To Use Them) Introduction Data is currency. Clean, prepared, useful data represents value, just like cold cash (if not more so). Poorly-sourced, irresponsibly-stored, or corrupted data is as worthless as counterfeit bills…

An Agile Approach to Cloud Data Warehouse Adoption   With the volume of data across every enterprise growing at hyper speed, organizations are seeking to modernize their existing data warehouse infrastructure in order to quickly scale and deliver value to…

Optimizing Your Network Topology   What does “topology” mean? The word topology is derived from the Greek word topos which means place, region, or space. The term is used across different sciences and technologies. In the context of replication, the…

Data Validation – the Value of Live Compare For years organizations have built data warehouses, analytical systems and, more recently, data lakes. Due to the nature of the query workload – “can we get fast query response times please?” –…

Introducing HVR 5.6: Cloud Connectivity Release Summer is upon us and its peak travel time. For all of us familiar with road trips and air travel, we all want to get to our destinations quickly, with minimal discomfort, and with…

7 major benefits of using log-based CDC for data replication from SAP What a Global Water Technology Company Achieved with Real-Time Data Replication  The world’s water resources are stretched beyond all previous limits and the toughest industrial water process challenges…

What is Streaming Data? “Streaming data“ is data that is continuously produced by a source and is processed without having access to all of the data (since streaming started). Radio and television are classic examples of streaming data. Nowadays it…

Remember Collaborate! HVR is an official IOUG partner and was also an exhibitor at Collaborate this year. Here is a recap of my experience at #C19TX. It had been a few years since I had attended Collaborate.  It was an…

Cloud Terminology Defining Common Cloud Acronomyns: SaaS, PaaS, IaaS, DBaaS In part one of this blog, I mentioned that the cloud has been around for quite some time but we just didn’t call it the cloud. In this post, I focus…

Cloud Data Replication There are many options to “get” your data to the cloud. Real-Time Data Replication is one of them. In this post, get an overview of what it is and get access to resources that can help you…

Data Integration Architecture: Understanding Agents To use agents, or to not use agents, that is the question The question of whether or not to use an agent when performing data integration, especially around use cases with log-based Change Data Capture…

SAP Data Integration How to Unlock Your SAP Data, Continuously and in Real-Time SAP is a popular platform for organizations large and small. Available since 1973, it’s used for everything from Collaborative Lifecycle Management (CLM) to supply chain management to industrial…

SQL Data Replication  This post is a technical overview of how HVR supports SQL Data Replication. The SQL Server Database is a very commonly used database, often running very demanding and mission-critical workloads. SQL Server is a common data source…

Data Lake Integration At recent IT industry events, we polled IT leaders about their plans for a data lakes. Data lakes have emerged from being known as “Hadoop” to more of a concept, it is not so much about what…

What are your backup plans for your big data databases? Big Data is the “shiny new toy” that everyone is playing around with at the moment. Companies have embraced these new technologies (e.g HDFS, S3, Hive etc)–something which was once…

Comparing your Data Across Multiple Source and Targets When you have data stored in multiple applications and want to consolidate it into a central data warehouse or data lake, you need to make sure the data in the source and…

Prescriptive Analytics: Driving The Transportation Industry Racing down the tracks with the equivalent of 170 Boeing 747s in tow, the 440,000 pound (200,000kg) next-generation locomotive is about to break. Some of the hundreds of sensors pulling in over 150,000 data…

Exploding Data Volumes. Real-Time Demand. System Overload. What to do about it. Most organizations have a need to consolidate data from multiple systems to get the information required to be successful. Over the last few decades, organizations built Operational Data…

A few weeks ago I wrote about latency in the context of long-running transactions. Latency is generally considered to be the time it takes for a transaction committed on the source system to be reflected (i.e. committed) on the target system….

Worried about Vendor Lock-in? Unlike Open source technologies such as PostgreSQL, most Database vendors (think Oracle, Microsoft, IBM, SAP) offer a suite of products, and obviously they will try to convince you to use more components in their technology stack….

Could have prevented a late night interruption… Monday, 2 AM. A brief, loud chirping sound wakes me up. A few minutes later I hear the same noise. Concerned that the noise may wake up the entire house I decide to…

The Real-Time Data Reality Today’s world can truly be described as a real-time world. From 24×7 news to social media to stock trading to traffic updates, everyone expects the latest information to be available on demand on their smartphone. With…

2 Key Differences And the impact on real-time reporting 1. Databases using columnar storage have a major benefit over equivalent databases using row-based storage: query performance for analytical queries that crunch through lots of data are much faster. Data is only…

Tracking Database Changes To understand what happened in your database is very powerful for multiple reasons: You want to look at the old state of the database – i.e. you want to go back in time. An issue came up,…

Data Center Migration: A Practical Approach In today’s data-focused business environment, almost all organizations face data migration challenges. While it is possible to meet these challenges using script-based or manual approaches, many organizations prefer to use a proven real-time replication…

Generally data replication products will not break transaction boundaries, unless configured to do so. Also, most real-time data replication tools don’t start applying transactions to the destination system until the source system commits the change (I am aware of one…

In the ideal world data replication is only in real time, is turned on once, and never has any problems. Most real-world replication has yet to reach the ideal world, and as a result there are many reason why it…

An audit table is a table that contains the full history of rows. I.e. based on the primary key of a row in the source table one can query the full history of the row in the audit table and…

A few years ago I worked with an ultra-fast analytical columnar database Vectorwise (now called Vector). At the time I thought Vectorwise could beat any row-based database on query performance thanks to the extremely efficient software. Then one day a…

Data Warehouse Replication Most readers will be familiar with the proverb “If you have a hammer, then everything looks like a nail” to indicate that one may be inclined to use the wrong tool to accomplish a task simply due…

What is Active/Active? Active/active replication is a setup in which multiple databases with the same application tables operate independently with changes synchronized between them. Geographical replication is a popular use case for active/active replication, but it may also be of…

High Availability (HA) has many angles. An obvious angle in the context of data replication technology is high availability of a replicated database. However, there is more around HA, especially in the context of heterogeneous replication, because in such a…

And what does it cost? The simplified view of application availability is binary: the application is available, or it is not. There is however a large grey area related to availability to indicate whether the application performs as well as…

MMP Databases Massively-Parallel Processing (MPP) databases such as Teradata, Greenplum or Actian Matrix (formerly ParAccel) achieve ultimate performance through parallel execution across multiple servers. In order for this approach to work well data must be distributed as evenly as possible,…

How can we speed up replication? Every now and again I come across the question: how can we speed up the replication? Irrespective of the target database the desirable answer for the person responsible for the replication implementation may be:…

Selecting the Right Data Replication Software for Your Business There are a million reasons you might need a real-time data replication solution. Maybe you’re migrating applications to the cloud or looking for a hybrid cloud solution. Or maybe you’re implementing…

More Important Than Ever as Real-Time Analytics Catches Fire According to a TDWI Best Practices Report on Real Time Data, BI, and Analytics, real-time analytics is taking off. A survey of BI and data warehousing professionals cited in the report…

Five Industries That Are Being Transformed By Real-Time Analytics In past blogs, I’ve defined real-time analytics and discussed alternative architectural approaches to implementing real-time analytic solutions. I’ve also touched on how some of our customers, such as Lufthansa, Aspen Marketing Services (a division of…

Oracle to SQL Server in 10 Minutes For some time I have been showing HVR demos because I think it is more interesting than showing PowerPoint slides. This applies both to the audience, and to me. Of course things can…

Real-Time Analytics Real-time analytics is a widely used IT term that can mean different things to different people. To some, it means extremely fast query response times, even if the data being queried resides in a data warehouse that hasn’t…

Is too much flexibility a bad thing? Flexibility in software often leads to complexity. However power users want the knobs to change things for optimum results and squeeze the last few percent of performance out of the system. Complexity of…

Best-Practices for Database Migrations Data migration, even when dealing with a single database is rarely as simple as it sounds especially if the database that is being migrated supports one or more production applications. In this blog, we will discuss…

Three Lessons One Global Financial Institution Learned When Building a Real-Time Enterprise Many organizations talk about the need to be a “real-time enterprise”, but very few organizations achieve this model when faced with the business, cultural, and technical challenges that go with…

The idea is simple: instead of placing data in a purpose-built data store, you move it into a data lake in its original format. This eliminates the upfront costs of data ingestion, like transformation. Once data is placed into the…

Challenges and Solutions for Geographical Replication Still one of the most common real-time data integration scenarios is on-premises real-time reporting and Business Intelligence. Such a scenario involves one or more source databases (e.g. Oracle, SQL Server, DB2) and one target…

Whoever said watching TV rots the brain has never tried watching TV at my house. Determining which boxes to turn on, following the proper sequence, and finding a specific show or movie is a strenuous mind exercise. Memory, applied logic,…

Forgive us, we get pretty excited about what we can do with SQL Server There are numerous options if you are looking for data replication out of and into SQL Server. One of these has been (and still is) HVR,…

 SNAP Mirror Replication vs Log-Based Data Replication Tools I came across the suggestion this week to use SAN-based SNAP Mirror for database replication instead of the type of log-based replication that other database replication tools, including HVR, perform. Differences between…