Mark Van de Wiel

Common Challenges in Cloud Reporting

When embarking on a cloud initiative, the first use case most organizations consider is reporting and analysis. Why is that?

  • An analytical environment deals with a lot of data. Cloud-based technologies provide scale using a pay-for-use model. To set up an analytical environment on-premises requires a large upfront investment.
  • The analytical environment consolidates data from multiple systems, some of which may be on-premises and some in the cloud.
  • Relative to transaction processing applications, the analytical environments are less crucial to the primary business process. Organizations still exploring cloud-based environments are more likely to start with less mission-critical deployments.
  • Organizations are at different stages of digital transformation. Analytical deployments are more likely new initiatives.

More and more organizations recognize the benefits such as reporting and analytics in real-time.

Challenge 1: Network Connectivity & Latency
Cloud technologies evolve rapidly and there are many options.

  • Cloud-based equivalents of traditional databases, available as a service— think database service equivalents of SQL Server, Oracle, PostgreSQL, and MySQL. The major cloud vendors AWS, Azure, and GCP all provide multiple options.
  • Technologies evolving in the cloud only, but with strong ties back to – once upon a time – on-premises technologies. Examples include the Amazon Aurora offering, compatible with PostgreSQL or MySQL, Redshift, Azure Synapse Analytics, and the Oracle Autonomous Data Warehouse.
  • Cloud-only platforms focused on analytics, including Snowflake, Google BigQuery, and Databricks Delta Lake.
  • Data lake stores like AWS S3, Azure Data Lake Store (ADLS), and Google Cloud Storage (GCS), supporting Hadoop technologies like Hive to present a SQL-compatible layer.
  • Streaming data technologies, including Kafka, Amazon Kinesis, Azure Event Hub, or Google Cloud Dataflow as the starting point for continuous analysis.

Which technology, or combination of technologies, makes the most sense for your use case?

The pay-for-use model for most cloud technologies allows for a relatively low-risk evaluation. Except you must invest time to load your data. How can you maintain flexibility across clouds, across technologies? And how do you minimize the load on systems, should you (temporarily) need a combination of technologies?

Challenge 2: Data Transfer Efficiency
Relative to connections in an on-premises environment, network bandwidth into the cloud may be a fair amount lower, and latency is likely a lot higher. How can you make the best use of this constraint resource?
Use best practices like Change Data Capture (CDC), but also compression, large block transfer, and other optimizations in the communication protocol to minimize sensitivity for high latency.

Challenge 3: Data Security
HVR provides low-impact, log-based CDC from numerous transactional systems. The technology supports capture once, deliver many, with support for a rich set of cloud-based analytical destination technologies.
The best practice architecture we employ in the cloud uses an HVR installation as close as possible to the source database and destination technology.

With support for database and data store technologies across clouds, HVR can address all three challenges for reporting and analytics in the cloud. Data compression and large block transfers are default, and encryption is easy to set up.

Find the platforms you rely on at https://www.hvr-software.com/product/platform-support/#.

About Mark

Mark Van de Wiel is the CTO for HVR. He has a strong background in data replication as well as real-time Business Intelligence and analytics.

Test drive
Contact us