Is the "Small Data" movement making Big Data tools like Spark obsolete?

All answers to this question.

Spark isn't obsolete, but it is no longer the "default" for everything, which is a good thing. We went through a period of "Resume-Driven Development" where people used a 10-node cluster to process a 5GB file. In late 2024, "Cost-Efficiency" is the top priority for CTOs. If you can do the same job on a single large EC2 instance using DuckDB for $10 instead of a $500 Databricks job, you are a hero. You should still learn Spark for the scale it provides, but a truly senior DE knows when not to use it. That judgment is what separates the pros from the learners.

   Answered 2024-11-05 by Cynthia Barrett

Cynthia, I totally agree on the cost aspect. But for someone looking to get hired at a FAANG or a large enterprise, will they even care about DuckDB, or do they still expect to see "Spark Expert" on the resume regardless of the project size?

   Answered 2024-11-18 by Ryan Fitzgerald

• Ryan, FAANG still uses Spark/Flink because they actually have the scale. But if you walk into a Mid-Market interview and explain how you saved the company 40% on cloud costs by migrating a legacy Spark job to a Polars-based lambda function, you’ll get the job immediately. It shows you care about the business's money, not just playing with cool tech. The trend for 2025 is "Right-Sized Engineering"—using the simplest tool that solves the problem reliably and cheaply.

     Commented 2024-11-28 by Eric Sullivan

I think of it as "Macro vs. Micro." Spark for the enterprise lake, DuckDB for the local analysis and the "Edge" computing. You need both in your toolkit to be versatile.

   Answered 2024-12-05 by Jordan Bell

• Spot on, Jordan. Versatility is the key. Being a "One-Tool" engineer is a dangerous place to be in this fast-moving market.

     Commented 2024-12-12 by Cynthia Barrett