How do we implement effective Data Governance and lineage in a distributed Big Data environment?

All answers to this question.

Governance in Big Data is a three-legged stool: Metadata, Lineage, and Security. For Hadoop-based stacks, Apache Atlas is the standard for metadata and lineage, while Apache Ranger handles fine-grained access control. You can tag a column as "PII" in Atlas, and Ranger will automatically apply a masking policy so only authorized users see the actual data. For modern cloud stacks, you might look at Atlan or Collibra. The key is to automate the metadata collection; if you rely on manual documentation, your lineage will be out of date within a week. Integration with your ETL tools like Airflow is vital to capture "in-flight" changes.

   Answered 2024-01-07 by Patricia Moore

This sounds like a nightmare for GDPR compliance. How are you handling "Right to be Forgotten" requests if you don't have a clear map of every place a user's' email address is stored in the cluster?

   Answered 2024-01-10 by Thomas Clark

• Thomas, it was a nightmare until we implemented a "Master Data Management" (MDM) layer. We now use a global ID for users across all systems. If a delete request comes in, we trigger an automated Spark job that uses our Atlas lineage map to find every table and file containing that ID. It's not perfect yet, but it’s much better than the "search and hope" method we were using before.

     Commented 2024-01-12 by Daniel Lewis

Data catalogs like AWS Glue or Amundsen are great for discovery, but remember that governance is about "people and process" more than just the software tools you install.

   Answered 2024-01-15 by Barbara Taylor

• Spot on, Barbara. We had to appoint "Data Stewards" for each business unit to ensure they actually label their data correctly before it ever touches the central lake.

     Commented 2024-01-17 by Jessica Williams