Moving from Batch Processing to Real-Time Analytics
When is it time to switch to (near) real-time? As a leader in real-time data replication technology, we are often involved in helping our clients make the transition from a scheduled batch approach to a modern continuous streaming method.
Some indicators that batch processing may no longer be suiting your organization can include: growing data volumes, inability to meet SLAs, miscommunication among employees, vendors, greater inventory yield variance, lack of competitive advantage, and more.
Also for some organizations, having a complete copy of their data once a day is not enough, as they struggle to identify what has changed throughout the day. This might mean they have to rely on timestamped records, which may or may not be updated each and every time. It also brings into question how you track and handle deletes. A real-time solution like HVR detects and replicates all changes and even validates the data so that you can have complete trust in it.
For some organizations, the path to adopting a real-time architecture is not always clear, so in this blog, I highlight some of the key drivers that we come across.
As new cloud-based services emerge for improving the customer experience, we see a large proportion of our clients needing to feed their new applications with key data from their existing applications (which is still typically on-premises). Users of the new applications expect a 24/7 service which depicts their current status and activity, updating it once a day is not an option.
goeasy, a financial service company, determined that requiring customers to wait up to 24-hours to receive a decision on online loan applications was no longer acceptable. Their BI team architected a real-time replication and integration strategy with HVR and Azure that enables them to respond faster to loan inquiries and grow their customer base.
Growing Data Volumes
For analytical use cases, again we are seeing these being spun up in the cloud on a plethora of technologies, and due to end-user demand, you could be looking to facilitate (near) real-time analytics. But what exactly does that mean? You request some information and you get yesterday’s data immediately? For true real-time analytics, you need the freshest data being continually fed into your chosen analytics platform.
In most organizations, the traditional overnight batch volumes are increasing, putting extra pressure on the processing window. These pressures drive some organizations away from a batch-led approach. The implementation of a new analytics platform is a common activity that allows us to revisit the approach and move to a more modern real-time strategy that delivers many business benefits.
Even moving to intraday reporting via batch processing can push some of the complexities back on to the core processing systems that hold key data that needs to be fed into the ecosystem. Since these systems are critical to the core business function, you cannot afford any overhead on these throughout the day, as it negatively impacts the primary users.
For example, an HVR water resources customer found that their growing data volumes meant that data didn’t always flow into the analytics systems as the users needed. In the beginning, analytics sapped (no pun intended) resources on the source SAP system.
This clearly didn’t scale, so ETL processes (Business Objects Data Services – BODS) were used to periodically extract large volumes of data and transport it to another Oracle database reporting instance. This impacted daily operations as the source was designed for many small parallel transactions, not large batch processes, however, analytic results slowly dripped out.
The customer now uses HVR to deliver data in real-time to the analytics systems. HVR’s log-based Change Data Capture only captures the data that has changed, so the impact on the source system is minimal and data delivery is fast.
Change Data Capture Feeds Event-Driven Architectures
Finally, any event-driven architecture requires real-time events to truly function. If you are not in a position to modify your core applications but require them to inform you immediately when customer x transacts, or inventory y drops below 10, then using a real-time integration solution such as HVR to deliver these events on behalf of the application is essential. A real-time approach like HVR detects and replicates all changes as they happen.
Our customer, Hop! Airlines, uses an event-driven architecture to better serve their employees. Quickly communicating flight and schedule changes to customers is extremely important, but so is the communication to flight staff. Their schedules get disrupted as well. HVR is a solution that supports their adoption of Kafka, delivering real-time notifications of schedule changes to in-flight teams.