How do we manage storage lifecycle policies to automatically move "cold" data to cheaper tiers?

All answers to this question.

Make sure you tag your data first. You can apply different lifecycle rules to different tags, which gives you much more granular control than just moving everything in a bucket.

   Answered 2024-09-15 by Dorothy Scott

• Tagging is a lifesaver, Dorothy. We tag by "Project ID," so when a project ends, all associated storage automatically moves to the cheapest archive tier.

     Commented 2024-09-16 by Margaret King

Automated tiering is the easiest way to see an immediate drop in your cloud bill. On AWS, you use S3 Lifecycle Rules. You can set a rule that says "if a file hasn't been accessed in 30 days, move it to S3 Standard-IA, and after 90 days, move it to Glacier Deep Archive." The best part is that it's all handled by the cloud provider. Just be careful with "Retrieval Fees." If you move something to Deep Archive but then suddenly need to download it next week, you will pay a significant premium. Use "S3 Intelligent-Tiering" if your access patterns are unpredictable—it handles the movement for you automatically.

   Answered 2024-09-17 by Margaret King

Do you have a clear definition of what "Cold" means for your specific company? Is it 30 days without access, or 90 days?

   Answered 2024-09-20 by Anthony Walker

• Anthony, defining "Cold" is harder than it sounds! In our case, our developers thought logs were cold after a month, but our security team needed them for 90 days for threat hunting. We ended up setting a compromise at 60 days. We also discovered that for very small files, the metadata cost of moving them to Glacier can actually be higher than just keeping them in Standard storage. You really have to look at your file sizes and access frequency before you turn on the automation, or you might get a surprise bill.

     Commented 2024-09-23 by Steven Hall