Should I use Python or SQL for the majority of my feature engineering in large-scale projects?

All answers to this question.

The general rule is to push as much computation to the database as possible. SQL is designed for set-based operations and is incredibly efficient at joining and aggregating large tables. If you can do your joins and basic filters in SQL, you save memory in your Python environment. However, for complex features like rolling window calculations, text embeddings, or custom mathematical transformations, Python is far more flexible. If your data exceeds 10GB, Pandas will likely crash, and that is when you should transition to PySpark or Polars instead of relying solely on SQL.

   Answered 2025-01-24 by Stephanie Long

Does your current infrastructure allow for dbt (data build tool) to manage these transformations, or are you writing raw scripts?

   Answered 2025-01-27 by Andrew Ford

• Andrew, we are still using raw scripts, which is becoming a nightmare for version control. If we switch to dbt, does it make it easier for Data Scientists to collaborate with Data Engineers, or does it add another layer of complexity that slows down the experimental phase of the project?

     Commented 2025-01-29 by Charles Hudson

SQL is king for data retrieval and basic cleaning. Save Python for the heavy lifting of statistical modeling and advanced feature creation.

   Answered 2025-02-01 by Pamela Knight

• Pamela is spot on. Using SQL for the "heavy lifting" of data movement ensures that your Python environment stays lean and focused on the actual science.

     Commented 2025-02-02 by Scott Peterson