Lazy Minimal Data Warehouse Refresh

Abstract

Data warehousing is a technique for integrating data from several source systems to enable pervasive data analytics. Data updates in the source systems result in data refreshes down the data stream. These data refreshes are potentially both computationally and financially costly, even if the refreshed data is not utilized before the next refresh. In this study, we present an accessible approach for selecting tables, views and materialized views for data refreshes in situations where entities have complex dependencies. While such solutions have been proposed in scientific literature in the past, their practical applications have been few and far between. We speculate that this gap between theory and practice stems from the highly theoretical presentation of such solutions. In this study, we aim to address this gap from a practice-oriented industry perspective. Utilizing this approach may, depending on the structure of the database and how it is used, present considerable improvements for the efficient utilization of, e.g., computation and networking resources, as well as energy efficiency.