Is R still relevant for Big Data analytics or should I focus entirely on Python and PySpark?

All answers to this question.

The "R vs Python" debate has shifted significantly. While R remains the gold standard for deep statistical analysis and beautiful visualizations (thanks to ggplot2), Python has definitely won the battle for production environments. If you are dealing with "Big Data," PySpark is almost mandatory because of its ability to handle distributed computing. However, don't abandon R. Many pharmaceutical and research firms still rely on it. My advice is to become "bilingual." Use R for exploratory data analysis and Python for building your scalable machine learning pipelines.

   Answered 2025-02-08 by Kimberly Hayes

Do you find that the integration between R and Spark (via SparkR) is sufficient for your needs, or do you feel the libraries in Python are more mature?

   Answered 2025-02-11 by Tyler Anderson

• Tyler, in my experience, SparkR is okay but PySpark has a much larger community and better documentation. When you run into a complex bug in a production pipeline, you’re much more likely to find a solution on Stack Overflow for Python. Plus, the deep learning libraries like PyTorch and TensorFlow have far better support in the Python ecosystem, which is a major factor today.

     Commented 2025-02-14 by Lawrence Baker

Python is definitely the leader for general-purpose data science, but R’s Shiny is still unbeatable for creating quick, interactive dashboards for non-technical stakeholders.

   Answered 2025-02-16 by Megan Sullivan

• That’s a great point, Megan. Shiny allows for a level of customization in data reporting that takes much longer to achieve with Python-based frameworks like Dash or Streamlit.

     Commented 2025-02-18 by Justin Moore