Should we adopt a Data Lakehouse architecture or stick with a traditional Data Warehouse?

All answers to this question.

The Lakehouse architecture, popularized by technologies like Delta Lake and Apache Iceberg, is definitely ready for prime time. The core benefit is that it brings ACID transactions and schema enforcement to the low-cost storage of a Data Lake. In the past, you would have to move data from HDFS to a Warehouse like Redshift for BI, creating data silos and latency. With a Lakehouse, your data scientists can access raw files while your analysts run SQL queries on the same storage layer. This "single source of truth" significantly reduces data engineering overhead and ensures everyone is looking at the same version of the data.

   Answered 2024-03-16 by Kimberly Adams

his sounds great in theory, but what about the performance of complex joins? Can a Lakehouse truly match the optimized query engines of a dedicated warehouse like Snowflake or BigQuery when handling billions of rows?

   Answered 2024-03-20 by Robert Taylor

• Robert, that’s where the "Engine" comes in. While the storage is a Lakehouse, we use Trino as our query engine. It allows us to perform high-speed joins across different storage formats. We’ve found that with proper partitioning and Z-ordering on our Delta tables, the performance gap is negligible for 90% of our use cases. The cost savings on storage alone made the switch worth it for us, and our data governance is much easier now that we aren't managing two separate environments.

     Commented 2024-03-22 by Christopher Evans

Start with a pilot. Move one high-volume data stream into a Lakehouse format and see if your BI tools can consume it without performance degradation before committing fully.

   Answered 2024-03-25 by Jennifer White

• I agree with Jennifer. We did a 30-day POC with Databricks and discovered that the "Auto-loader" feature saved our engineers hours of manual ETL work.

     Commented 2024-03-27 by Sarah Miller