How can we reduce high egress fees when moving large datasets between different cloud providers?

All answers to this question.

You might want to look into Cloudflare R2. They don't charge any egress fees, which makes them an excellent intermediary "buffer" for moving data between the major providers.

   Answered 2024-03-15 by Sarah Parker

• I completely agree with Sarah. R2 has been a game-changer for our video streaming assets. We store the masters there and pull them to various compute clouds without worrying about the bill.

     Commented 2024-03-16 by Robert Miller

Egress fees are the "silent killer" of cloud budgets. To tackle this, you should first look into a dedicated interconnect like AWS Direct Connect or Azure ExpressRoute combined with a neutral colocation facility. This allows for lower, more predictable data transfer rates compared to the public internet. Another strategy is to implement "Data Gravity" principles—process the data as close to its storage location as possible using serverless functions or containers. If you must move data, consider using a specialized cloud storage provider like Wasabi or Backblaze for your "hot" storage, as they often have zero egress fee policies when integrated with certain CDNs.

   Answered 2024-03-18 by Jennifer Jenkins

Have you tried implementing a compression layer like Zstandard or Gzip before the transfer, or are the datasets already in a compressed format like Parquet?

   Answered 2024-03-22 by Michael Thompson

• Michael, compression is a great start! However, for petabyte-scale data, the CPU overhead for compression can sometimes outweigh the savings. I’ve found that using a "Delta Lake" architecture helps more. Instead of moving the whole dataset, you only sync the changed bits. This significantly reduces the total gigabytes transferred and has saved my current firm about 40% on our cross-cloud sync costs over the last year. It's definitely the way to go for modern data engineering.

     Commented 2024-03-25 by David Wilson