Enhancing Data Interoperability in Multi-platform Lakehouses with Apache Iceberg

Abstract

Managing data across diverse platforms poses significant challenges, including data duplication, vendor lock-in, and inconsistent governance. Lack of a unified table format often leads to complex pipelines, increased storage costs, and hindered interoperability. Apache Iceberg, with its platform-agnostic design, presents a solution by providing a consistent table format for large-scale analytical workloads while addressing cross-platform data accessibility. In this paper, we study the use of Apache Iceberg as a unified table format to enable interoperability between Snowflake and Databricks, with data stored on Amazon S3. Experimental setups include accessing Snowflake-managed Iceberg tables in Databricks and vice versa. Key focus areas include examining query performance, metadata synchronization, and the challenges of managing consistent data across platforms. Optimization strategies, specifically data reordering, were applied to test improvements in query performance for various workloads. The results show that Iceberg reduces the complexity of data management by automating metadata handling and synchronization, ensuring real-time data consistency. Query performance showed improvement in medium-complexity queries with optimized Iceberg tables, while highlighting potential areas for further optimization in full-table scans. These findings underscore Iceberg’s potential as a scalable, efficient solution for modern data lake architectures.