How can I optimize Cloud Storage costs for large datasets in a Data Science project?

All answers to this question.

You should look into S3 Intelligent-Tiering immediately. It automatically moves your data between "Frequent Access" and "Infrequent Access" tiers based on your usage patterns without any performance impact. For your older datasets that are rarely touched but must be kept for compliance, move them to "Glacier Deep Archive." The cost difference is massive—Standard is about $0.023 per GB, while Deep Archive is $0.00099. Also, make sure you are using VPC Endpoints so you aren't paying data transfer fees when your ML models on EC2 pull data from your S3 buckets; that is a hidden cost many people overlook.

   Answered 2025-06-10 by Dorothy Hall

How difficult is it to retrieve data from the archive tiers if a data scientist suddenly needs to re-train a model on three-year-old data?

   Answered 2025-06-20 by William Young

• William, retrieving from Deep Archive can take 12 to 48 hours. If you need it faster, "Glacier Instant Retrieval" is a better middle ground. It gives you the low storage cost but allows for millisecond access, though you pay a higher fee for the actual retrieval of the data itself.

     Commented 2025-06-28 by Charles King

We saved 30% just by cleaning up incomplete multipart uploads. Sometimes the simplest lifecycle rules for "AbortIncompleteMultipartUpload" make a huge difference.

   Answered 2025-07-05 by Joseph Allen

• Great tip, Joseph. I often forget about those orphaned fragments. They really do add up over time when you are constantly uploading large datasets for training models.

     Commented 2025-07-12 by Sarah Lewis