Data Integration Architecture: Understanding Agents To use agents, or to not use agents, that is the question The question of whether or not to use an agent when performing data integration, especially around use cases with log-based Change Data Capture…

SAP Data Integration How to Unlock Your SAP Data, Continuously and in Real-Time SAP is a popular platform for organizations large and small. Available since 1973, it’s used for everything from Collaborative Lifecycle Management (CLM) to supply chain management to industrial…

SQL Data Replication  This post is a technical overview of how HVR supports SQL Data Replication. The SQL Server Database is a very commonly used database, often running very demanding and mission-critical workloads. SQL Server is a common data source…

Redshift Data Warehouse  Best Practices for Integrating into your cloud-based data warehouse for Real-Time Analytics Amazon Redshift is a managed data warehouse service in the cloud. Redshift originated in 2012 as a fork of the ParAccel database following Amazon’s investment…

Data Lake Integration At recent IT industry events, we polled IT leaders about their plans for a data lakes. Data lakes have emerged from being known as “Hadoop” to more of a concept, it is not so much about what…

Cloud Data Replication. Facts and Figures. You don’t have to look very far to learn that all bets (and dollars) are on the cloud for data storage and management. At recent IT Industry events that we sponsored, we set to…

What are your backup plans for your big data databases? Big Data is the “shiny new toy” that everyone is playing around with at the moment. Companies have embraced these new technologies (e.g HDFS, S3, Hive etc)–something which was once…

Comparing your Data Across Multiple Source and Targets When you have data stored in multiple applications and want to consolidate it into a central data warehouse or data lake, you need to make sure the data in the source and…

Prescriptive Analytics: Driving The Transportation Industry Racing down the tracks with the equivalent of 170 Boeing 747s in tow, the 440,000 pound (200,000kg) next-generation locomotive is about to break. Some of the hundreds of sensors pulling in over 150,000 data…

Exploding Data Volumes. Real-Time Demand. System Overload. What to do about it. Most organizations have a need to consolidate data from multiple systems to get the information required to be successful. Over the last few decades, organizations built Operational Data…

Abandoning one for the other may not be the answer Over the last few decades we’ve seen an explosion in the use of Data Warehousing within the IT industry; it was all the rage for doing analytics and reporting. This…

A few weeks ago I wrote about latency in the context of long-running transactions. Latency is generally considered to be the time it takes for a transaction committed on the source system to be reflected (i.e. committed) on the target system….

Worried about Vendor Lock-in? Unlike Open source technologies such as PostgreSQL, most Database vendors (think Oracle, Microsoft, IBM, SAP) offer a suite of products, and obviously they will try to convince you to use more components in their technology stack….

Could have prevented a late night interruption… Monday, 2 AM. A brief, loud chirping sound wakes me up. A few minutes later I hear the same noise. Concerned that the noise may wake up the entire house I decide to…

The Real-Time Data Reality Today’s world can truly be described as a real-time world. From 24×7 news to social media to stock trading to traffic updates, everyone expects the latest information to be available on demand on their smartphone. With…

2 Key Differences And the impact on real-time reporting 1. Databases using columnar storage have a major benefit over equivalent databases using row-based storage: query performance for analytical queries that crunch through lots of data are much faster. Data is only…

Tracking Database Changes To understand what happened in your database is very powerful for multiple reasons: You want to look at the old state of the database – i.e. you want to go back in time. An issue came up,…

Data Center Migration: A Practical Approach In today’s data-focused business environment, almost all organizations face data migration challenges. While it is possible to meet these challenges using script-based or manual approaches, many organizations prefer to use a proven real-time replication…

Generally data replication products will not break transaction boundaries, unless configured to do so. Also, most real-time data replication tools don’t start applying transactions to the destination system until the source system commits the change (I am aware of one…

In the ideal world data replication is only in real time, is turned on once, and never has any problems. Most real-world replication has yet to reach the ideal world, and as a result there are many reason why it…

An audit table is a table that contains the full history of rows. I.e. based on the primary key of a row in the source table one can query the full history of the row in the audit table and…

A few years ago I worked with an ultra-fast analytical columnar database Vectorwise (now called Vector). At the time I thought Vectorwise could beat any row-based database on query performance thanks to the extremely efficient software. Then one day a…

Data Warehouse Replication Most readers will be familiar with the proverb “If you have a hammer, then everything looks like a nail” to indicate that one may be inclined to use the wrong tool to accomplish a task simply due…

What is Active/Active? Active/active replication is a setup in which multiple databases with the same application tables operate independently with changes synchronized between them. Geographical replication is a popular use case for active/active replication, but it may also be of…

High Availability (HA) has many angles. An obvious angle in the context of data replication technology is high availability of a replicated database. However, there is more around HA, especially in the context of heterogeneous replication, because in such a…

And what does it cost? The simplified view of application availability is binary: the application is available, or it is not. There is however a large grey area related to availability to indicate whether the application performs as well as…

Three Updates with our SQL Server Replication Software   There are numerous options if you are looking for data replication out of and into SQL Server. One of these has been HVR, enabling replication in both homogeneous (SQL Server to SQL…

Is your car faster than mine? Maybe it is, but if the speed limit is 65 or 70 miles per hour, does it really matter? Also note that there are dependencies. For example under what conditions do you measure? On…

MMP Databases Massively-Parallel Processing (MPP) databases such as Teradata, Greenplum or Actian Matrix (formerly ParAccel) achieve ultimate performance through parallel execution across multiple servers. In order for this approach to work well data must be distributed as evenly as possible,…

How can we speed up replication? Every now and again I come across the question: how can we speed up the replication? Irrespective of the target database the desirable answer for the person responsible for the replication implementation may be:…

Selecting the right Data Replication Software for Your Business There are a million reasons you might need a real-time data replication solution. Maybe you’re migrating applications to the cloud. Or maybe you’re implementing a real-time analytics solution. Or perhaps you…

More Important Than Ever as Real-Time Analytics Catches Fire According to a TDWI Best Practices Report on Real Time Data, BI, and Analytics, real-time analytics is taking off. A survey of BI and data warehousing professionals cited in the report…

Five Industries That Are Being Transformed By Real-Time Analytics In past blogs, I’ve defined real-time analytics and discussed alternative architectural approaches to implementing real-time analytic solutions. I’ve also touched on how some of our customers, such as Lufthansa, Aspen Marketing Services (a division of…

Oracle to SQL Server in 10 Minutes For some time I have been showing HVR demos because I think it is more interesting than showing PowerPoint slides. This applies both to the audience, and to me. Of course things can…

5 Benefits of Real-Time Data Warehousing Going from an infrequently updated data warehouse or data mart environment to a near real-time data warehouse has a number of benefits: 1. FASTER DECISIONS: Make decisions quicker based on more current and more…

Real-Time Analytics Real-time analytics is a widely used IT term that can mean different things to different people. To some, it means extremely fast query response times, even if the data being queried resides in a data warehouse that hasn’t…

Is too much flexibility a bad thing? Flexibility in software often leads to complexity. However power users want the knobs to change things for optimum results and squeeze the last few percent of performance out of the system. Complexity of…

Best-Practices for Database Migrations Data migration, even when dealing with a single database is rarely as simple as it sounds especially if the database that is being migrated supports one or more production applications. In this blog, we will discuss…

Three Lessons One Global Financial Institution Learned When Building a Real-Time Enterprise Many organizations talk about the need to be a “real-time enterprise”, but very few organizations achieve this model when faced with the business, cultural, and technical challenges that go with…

The idea is simple: instead of placing data in a purpose-built data store, you move it into a data lake in its original format. This eliminates the upfront costs of data ingestion, like transformation. Once data is placed into the…

Challenges and Solutions for Geographical Replication Still one of the most common real-time data integration scenarios is on-premises real-time reporting and Business Intelligence. Such a scenario involves one or more source databases (e.g. Oracle, SQL Server, DB2) and one target…

Whoever said watching TV rots the brain has never tried watching TV at my house. Determining which boxes to turn on, following the proper sequence, and finding a specific show or movie is a strenuous mind exercise. Memory, applied logic,…

Forgive us, we get pretty excited about what we can do with SQL Server There are numerous options if you are looking for data replication out of and into SQL Server. One of these has been (and still is) HVR,…

 SNAP Mirror Replication vs Log-Based Data Replication Tools I came across the suggestion this week to use SAN-based SNAP Mirror for database replication instead of the type of log-based replication that other database replication tools, including HVR, perform. Differences between…