How do we choose between a Data Lake and a Data Warehouse for a new Big Data architecture?

All answers to this question.

The choice depends on your "Schema-on-Read" vs. "Schema-on-Write" requirements. Data Warehouses are great for structured data and high performance but are expensive to scale for unstructured raw data. Data Lakes (like S3 or HDFS) allow you to store everything cheaply, but they can quickly turn into "Data Swamps" without proper metadata management. I strongly suggest looking into the Data Lakehouse pattern using technologies like Delta Lake or Apache Iceberg. This gives you the low-cost storage of a lake with the ACID transactions and schema enforcement of a warehouse. It is the modern standard for balancing flexibility with data integrity.

   Answered 2024-03-16 by Kimberly Adams

Have you evaluated your team's' existing skill set? Moving to a Lakehouse or a pure Data Lake requires heavy Spark or Python knowledge, whereas a Warehouse is usually accessible to anyone who knows standard SQL.

   Answered 2024-03-20 by Robert Taylor

• Robert, that’s a fair point. Our analysts are SQL-heavy, so we chose Snowflake as our warehouse but kept an S3 bucket for raw ingestion. We use dbt (data build tool) to transform the raw data into structured tables. This way, the data scientists get their raw files, and the business users get their clean SQL tables without us having to manage two completely separate, disconnected environments.

     Commented 2024-03-22 by Christopher Evans

Start with the "Why." If you don't need real-time machine learning, a simple cloud data warehouse is often more than enough and much easier to maintain than a complex lake.

   Answered 2024-03-25 by Jennifer White

• Totally agree, Jennifer. Over-engineering a Big Data stack is a common trap. If your data volume is only in the low Terabytes, a full Hadoop cluster is just unnecessary overhead.

     Commented 2024-03-27 by Sarah Miller