Exciting Times for Data Integration
More Important Than Ever as Real-Time Analytics Catches Fire
According to a TDWI Best Practices Report on Real Time Data, BI, and Analytics, real-time analytics is taking off. A survey of BI and data warehousing professionals cited in the report found that 41% of respondents update their data warehouses multiple times each day, and 30% continuously update their data warehouses in near real time.
Data integration has always been the biggest challenge associated with real time-analytics, and three major trends are making it even more challenging.
- More data sources in more places – A decade ago, most data warehouses were sourced from on-premises ERP systems, legacy applications, and websites. Those systems are still hanging around at most companies, but they’ve been augmented with a plethora of cloud applications and data centers that are geographically distributed all over the planet.
- More diverse data types and schemas – Until recently, data warehouse architects, designers, and developers only worried about moving OLTP data from relational databases and flat files into relational data warehouses with dimensional schemas. Today, they deal with Hadoop HDFS and similar file systems, columnar databases for analytics, NoSQL databases, machine data, and other data types. It can be structured, semi-structured, or unstructured.
- Exploding data volumes – As we all know, things like Hadoop, cloud computing, mobile devices, and Internet of Things produce lots of data. So much so that a cottage industry has emerged focused solely on citing amazing facts and statistics that tell us just how much data we’re generating. Who knew that Google’s 16 data centers housed almost a million servers and consume .01% of the world’s energy? Or that an Exabyte of data could fill 250 million DVDs? Or that every two days, the world generates the total amount of data generated from the dawn of civilization until 2003?
These are exciting times for everyone involved with big data and business intelligence. And as game changing new technologies and innovative analytic applications come online at light speed, the importance of data capture, management and integration is greater than ever.