Achieving maximum data replication performance is not easy. You run into many dependencies. Some are due to the environment and related to the source technology or target technology. Others are related to what you want to replicate. And finally, how...
After several acquisitions, a leading e-tailer expanded its portfolio of brands and wants to unite the data collected across the enterprise. The entire portfolio is represented on the eCommerce site, however, things look very different on the backend. The data...
The first blog in this series discussed a general approach to achieving maximum performance. In the second blog, I highlighted HVR’s distributed architecture and its performance and scalability benefits. This post focuses on the initial—or one time—load: Refresh. Subsequent posts...
In the first blog of this series, I shared general concepts about data replication performance. This second post is about recommended performance architecture when deploying HVR. I will use real-world analogies to illustrate why the architecture is ideal for optimal...
This post is the first in a series of five to discuss data replication performance. This blog post focuses on performance objectives and general aspects of data performance. In the second post, we discuss HVR’s performance architecture. The remaining three...
With limited face-to-face interactions due to the COVID-19 pandemic and an increasingly competitive landscape, retail companies are working to digitally transform to maintain a viable business and keep customers happy. You may have heard the term “Digital Transformation” being thrown...
In the current rapidly changing market and environment, businesses are struggling to determine the best way to remain viable. Organizations have taken the opportunity to assess their business and find operational efficiencies that can improve their overall fiscal health. Because...
The term “hybrid cloud” has no single definition. When it was first used, it typically meant a combination of a private and public cloud. Since then, the term has arguably taken on a broader meaning, including (public) cloud offerings from...
The COVID-19 pandemic required businesses, especially those in the financial services industry, to reevaluate their digital transformation strategy, find ways to optimize operational efficiencies, and better serve customers. In the inaugural session of the “Digital Transformation Storytelling Series,” Talend’s RVP...
To maintain a competitive advantage, quickly respond to customer needs, and manage supply and demand, companies in the consumer goods industry need to evaluate their digital transformation strategy. But what does it mean to transform digitally? Digital Transformation is about...
Back in the day, there were only a couple of ways to move day around an organization. With Change Data Capture (CDC), instead of uploading the whole file again with quarterly or weekly sales, you would only transport and load the...
What is Digital Transformation? /ˈdijidl/ /ˌtran(t)sfərˈmāSH(ə)n/ The process of using digital technologies to create new — or modify existing — business processes, culture, and customer experiences to meet changing business and market requirements. This reimaging of business in the digital...
“Replication in computing involves sharing information so as to ensure consistency between redundant resources, such as software or hardware components, to improve reliability, fault-tolerance, or accessibility.” Use cases for database replication include high availability, the geographical distribution of data, migration...
Helping customers architect their data management infrastructure is one of the most exciting aspects of my job. When organizations first engage with HVR, they are often in the early stages of their digital transformation journey: the realization that data assets...
Managing data flows today and into the future The unlimited scalability of Snowflake’s cloud data platform and HVR’s unique change data capture technology, provide customers with an effective solution for continuous, high-volume data replication and integration. During the webinar, Achieving...
Data is everywhere. But data by itself is not very valuable. When Relational Database Management Systems (RDBMS) became popular, the amount of usable information exploded. Businesses in almost every industry took advantage of being able to organize their data in...
When is it time to switch to (near) real-time? As a leader in real-time data replication technology, we are often involved in helping our clients make the transition from a scheduled batch approach to a modern continuous streaming method. Some...
How the leading gift provider is leveraging HVR and Snowflake to create a single cloud-based data warehouse for broader analytics across its family of brands 1-800-FLOWERS.COM, Inc. is a one-stop gifting destination featuring a family of more than...
Why Organizations May Want to Consider a Multi-Cloud Integration Approach “Don’t put all your eggs in one basket” this saying, attributed to Cervantes, holds true 400 years later. Many organizations don’t rely on just one IT vendor to deliver all...
As organizations seek to gain business efficiencies through broader analytics, the need to move data between systems is greater now than ever before. To do this, organizations look to build an optimized analytical data processing environment, one that combines data...
HVR + Log-Based CDC for Oracle I remember the very first time I heard the phrase Change Data Capture (CDC), specifically it was Oracle CDC. I was in El Segundo, CA, right across from LAX airport. I was teaching an...
Log-Based Replication for SQL Server Always On Systems According to DE-Engine’s list of the most popular database technologies, Microsoft SQL Server remains in the top three for worldwide deployment year after year, only behind Oracle and MySQL. With that, it’s...
How One of the Largest Mutual Fund Houses in India Integrated Oracle with PostgreSQL for Real-Time Analytics Background TO THE NEW is a digital technology company that provides end-to-end product development services. They help businesses digitally transform by leveraging the...
What to Look for in a Data Validation Solution for Replication Many of us have experience moving to a new home or city. Personally, I have moved across continents and within the contiguous United States. In any moving process, it’s...
Learn how HVR retrieves up-to-date definitions for pool and cluster tables using log-based change data capture The Ubiquity of SAP across the Enterprise What does 3M, BMW, The Coca-Cola Company, DHL, Ford Motor Company, Airbus, The Dow Chemical Company, and...
Exploring Oracle GoldenGate Alternatives Data replication, specifically, data replication using log-based Change Data Capture, has been a data integration option for organizations for over a decade. It is a solution that serves several traditional and modern use cases: distributing load...
Several TB of Data Changes Per Day? No problem. How HVR enables high volume data replication from systems such as SAP into Snowflake Snowflake is a modern data platform built for any cloud. With storage separate from computing resources, it...
The Major Types of Data Integration (& When To Use Them) Introduction Data is currency. Clean, prepared, useful data represents value, just like cold cash (if not more so). Poorly-sourced, irresponsibly-stored, or corrupted data is as worthless as counterfeit bills...
How CDC Powers Real-time Reporting with Tableau For nearly two decades, Tableau has been the gold standard for data visualization platforms. Tableau provides a highly intuitive graphical user interface that makes it easy for non-technical data analysts to go...
Using CDC to Power Real-Time Analytics on Snowflake Snowflake is the first data warehouse and analytics service to be built for the cloud. Since Snowflake’s introduction in 2012, companies have flocked to it in order to minimize costs, simplify...
In this post, I talk about the benefits of using log-based CDC for a real-time data warehouse…But first, let’s start with a short history of Data Warehouses Data warehouses have been around for decades. Teradata, the grandfather of the data...
7 Major Benefits of Using Log-Based CDC for Data Replication from SAP What a Global Water Technology Company Achieved with Real-Time Data Replication The world’s water resources are stretched beyond all previous limits and the toughest industrial water process challenges...
Common Challenges in Cloud Reporting When embarking on a Cloud initiative, the first use case most organizations consider is reporting, and more and more organizations see the benefits of real-time reporting and analytics. Relative to on-premises real-time reporting, using the...
Nine Tips for Choosing Cloud Data Migration Software Key considerations for choosing a cloud data migration software for the near and long-term As your organization strategically moves applications, architectures, and data platforms to the cloud, determining the best tool to...
Can your data replication technology keep up? Let’s look at some factors for consideration. Think about it this way…Is your car faster than mine? Maybe it is, but if the speed limit is 65 or 70 miles per hour, does...
Five Common Cloud Integration Challenges ...and How to Overcome Them Companies are adopting the cloud at an unprecedented rate. While the benefits of cloud computing are undeniable and well-publicized, data integration presents a unique set of challenges, particularly in hybrid...
Three Key Requirements for Continuous Integration in Cloud Environments In my previous blog post, I discussed hybrid cloud computing. The phrase Hybrid Cloud Computing is used to describe the coexistence of multiple environments, at least one of which is a...
Hybrid Cloud Computing My previous blog post covered motivations why organizations adopt the cloud, and why IT is now embracing cloud computing. However, a move to the cloud can be a complex undertaking, especially for organizations running many applications or...
The 10 Most Common Customer Questions About Real-Time Data Replication Business users are demanding access to real-time data to improve decision making across the organization. Whether they’re looking to react quickly to customer needs, differentiate from the competition, or respond...
3 Best Practices for AWS Cloud Data Integration Tools | HVR Organizations interested in leveraging the power of AWS should consider the following best practices when designing solutions for their cloud integration software. Early cloud adopters were often startups attracted...
Understanding Managed File Transfers. Considerations and Challenges. In many organizations, a large part of the exchange and distribution of information is realized by the copy and transfer of data files. As the number of files is ever increasing, the results...
SAP Data Integration How to Unlock Your SAP Data, Continuously and in Real-Time SAP is a popular platform for organizations large and small. Available since 1973, it’s used for everything from Collaborative Lifecycle Management (CLM) to supply chain management to industrial...
Redshift Data Warehouse Best Practices for Integrating into your cloud-based data warehouse for Real-Time Analytics Amazon Redshift is a managed data warehouse service in the cloud. Redshift originated in 2012 as a fork of the ParAccel database following Amazon’s investment...
Six Steps Toward Successful Cloud Data Migrations …And minimize downtime This post provides a set of best practices to facilitate cloud migrations with minimal downtime and minimal risk, irrespective of the type of cloud service you plan to adopt, focusing...
Change Data Capture – Impact on Database Processing Clients often ask: “What is HVR’s impact on database processing?” The question is simple, the answer, not so much. There are, however, a number of considerations. Supplemental Logging Most databases focused on transactional...
HVR 5.2: Data Lake HVR 5.2 provides even faster, more secure, and accurate data movement in a modern environment. Latest enhancements include features that help customers optimize their adoption of data lake technologies. Read on to learn more… Defining Data Lakes...
Prescriptive Analytics: Driving The Transportation Industry Racing down the tracks with the equivalent of 170 Boeing 747s in tow, the 440,000 pound (200,000kg) next-generation locomotive is about to break. Some of the hundreds of sensors pulling in over 150,000 data...
Streaming Data How important are “transactions”? Streaming Data: A Definition Amazon Web Services defines Streaming Data as “data that is generated continuously by thousands of data sources, which typically send in the data records simultaneously, and in small sizes (order...
Using Change Data Capture to Augment your ELT/ETL Solutions Reporting on near real-time data is a must in today’s landscape. To stay competitive, companies are now implementing more of an operational BI strategy for day-to-day tactical decision making to increase...
Exploding Data Volumes. Real-Time Demand. System Overload. What to do about it. Most organizations have a need to consolidate data from multiple systems to get the information required to be successful. Over the last few decades, organizations built Operational Data...
Similarities and Differences between Oracle and SQL Transaction Logs In preparation for a presentation session about transaction log fundamentals at a past Silicon Valley SQL Saturday event in Mountain View, I looked into understanding more detail about the SQL Server...
Microsoft to Microsoft and Oracle to Microsoft We thought that, instead of writing about some deeply technical issue in this blog, we would answer a couple of the first questions that we get asked by prospective new customers. Those questions...
ACID in the context of database technology is the abbreviation for Atomicity, Consistency, Isolation, and Durability. These concepts date back to the 1970s, and the first software developed to implement these properties automatically is by now literally decades old. Today’s...
Problems don’t always originate where they surface Most software is pretty complex to ensure it works well irrespective of the many ways it can be used. This applies particularly to flexible platform technologies such as operating systems, databases, and also...
APIs are a great way to share data, and – especially in Software-as-a-Service (SaaS) environments – APIs have become a de-facto “standard” to integrate data. I put “standard” within quotes because there aren’t necessarily a lot of standards around APIs....
Worried about Vendor Lock-in? Unlike Open source technologies such as PostgreSQL, most Database vendors (think Oracle, Microsoft, IBM, SAP) offer a suite of products, and obviously they will try to convince you to use more components in their technology stack....
2 Key Differences And the impact on real-time reporting 1. Databases using columnar storage have a major benefit over equivalent databases using row-based storage: query performance for analytical queries that crunch through lots of data are much faster. Data is only...
Data Center Migration: A Practical Approach In today’s data-focused business environment, almost all organizations face data migration challenges. While it is possible to meet these challenges using script-based or manual approaches, many organizations prefer to use a proven real-time replication...
How Do You Manage Large Tables? Some of HVR’s largest clients store hundreds of millions of rows in tables with many columns. Dealing with tables this size is not necessarily an issue from an ongoing real-time data replication perspective –...
Real-time data replication implementations cover a number of use cases that you find on this website. Amongst the most popular use cases are Real-Time Business Intelligence (real-time reporting) and High Availability. These use cases are popular because nowadays almost every...
Alternative Architectural Approaches to Implementing Real-Time Analytics In other posts, I’ve mentioned several different architectural approaches that can be employed to access and analyze near real-time operational data. Because this is a frequent topic of conversation that we have with customers...
Generally, data replication products will not break transaction boundaries unless configured to do so. Also, most real-time data replication tools don’t start applying transactions to the destination system until the source system commits the change (I am aware of one...
Database transaction logs are very complex. A transaction commit only succeeds when the change has been written into and persisted by the transaction log (the data for the transaction may still sit in memory). Transaction logs allow a database to...
A few years ago I worked with an ultra-fast analytical columnar database Vectorwise (now called Vector). At the time I thought Vectorwise could beat any row-based database on query performance thanks to the extremely efficient software. Then one day a...
Imagine your employer decided to gradually migrate away from your current primary database technology to an alternative technology. And you are the one who has to make this happen. Initially, the alternative technology will only be used for real-time reporting,...
What is Active/Active? Active/active replication is a setup in which multiple databases with the same application tables operate independently with changes synchronized between them. Geographical replication is a popular use case for active/active replication, but it may also be of...
Near Real-Time Data Warehouses: A Primer Business leaders are constantly on the lookout for ways to improve responsiveness to customers. And new technologies are helping them meet that objective. Take data warehousing technology. Because data warehouses once only operated in...
High Availability (HA) has many angles. An obvious angle in the context of data replication technology is high availability of a replicated database. However, there is more around HA, especially in the context of heterogeneous replication, because in such a...
SQL Server Replication Software Knowing your options If you’re looking for data replication out of and into SQL Server, one option is HVR, which enables replication in both homogeneous (SQL Server to SQL Server) and heterogeneous environments. HVR version 5 introduced...
Four Methods of Change Data Capture As its name suggests, Change Data Capture (CDC) techniques are used to identify changes. CDC can be the basis to synchronize another system with the same incremental changes, or to store an audit trail...
How can you perform high-performance real-time Business Intelligence on a columnar database? Columnar databases – in contrast to row-based databases – physically store data for columns together in data blocks (both on disk and in-memory). Queries retrieving data from many...
MMP Databases Massively-Parallel Processing (MPP) databases such as Teradata, Greenplum or Actian Matrix (formerly ParAccel) achieve ultimate performance through parallel execution across multiple servers. In order for this approach to work well data must be distributed as evenly as possible,...
Selecting the Right Data Replication Software for Your Business There are a million reasons you might need a real-time data replication solution. Maybe you’re migrating applications to the cloud or looking for a hybrid cloud solution. Or maybe you’re implementing...
SQL Server Data Replication Done Fast and Reliably Data Replication for SQL Server and Heterogeneous Environments An overview, how-to videos and a case study to see how easy it can be to use HVR for SQL Server Replication Scenarios in Heterogeneous...
Why use a Third-Party Tool over SQL Server Replication? (updated April 22, 2019) Most database vendors provide their own native transactional replication solution. Microsoft is no exception. If your use case is database replication between two Microsoft SQL Server...
Five Industries That Are Being Transformed By Real-Time Analytics In past blogs, I’ve defined real-time analytics and discussed alternative architectural approaches to implementing real-time analytic solutions. I’ve also touched on how some of our customers, such as Lufthansa, Aspen Marketing Services (a division of...
Do You Need It, and How Do You Do It? The term “real-time” is used extensively in the information technology industry, but the definition is somewhat ambiguous. Since we describe our product offering as “real-time data replication software”, we thought...
5 Benefits of Real-Time Data Warehousing Going from an infrequently updated data warehouse or data mart environment to a near real-time data warehouse has a number of benefits: 1. FASTER DECISIONS: Make decisions quicker based on more current and more...
Three Lessons One Global Financial Institution Learned When Building a Real-Time Enterprise Many organizations talk about the need to be a “real-time enterprise”, but very few organizations achieve this model when faced with the business, cultural, and technical challenges that go with...
Common Challenges in Cloud Reporting When embarking on a Cloud initiative, the first use case most organizations consider is reporting, and more and more organizations see the benefits of real-time reporting and analytics. Relative to on-premises real-time reporting, using the...
Challenges and Solutions for Geographical Replication Still one of the most common real-time data integration scenarios is on-premises real-time reporting and Business Intelligence. Such a scenario involves one or more source databases (e.g. Oracle, SQL Server, DB2) and one target...
SNAP Mirror Replication vs Log-Based Data Replication Tools I came across the suggestion this week to use SAN-based SNAP Mirror for database replication instead of the type of log-based replication that other database replication tools, including HVR, perform. Differences between...
Worry about the impact of the Log-Based CDC? Many DBAs are – rightfully – worried about the impact log-based Change Data Capture (CDC) will have on the mission-critical transaction processing database. There are many factors that influence CDC performance including...
Cloud Migration in the Forecast IT continues its inexorable march to the cloud. Whether we’re talking about infrastructure, middleware, applications or business process services, Gartner reports that by 2021 approximately 28% of all IT spending will be for cloud-based services—up...