Is HDFS Federation the right solution for scaling a NameNode that has reached its memory limit?

All answers to this question.

HDFS Federation is designed exactly for your situation. While hardware upgrades (more RAM) can provide temporary relief, the NameNode is ultimately a single-threaded bottleneck for namespace operations. Federation allows you to add multiple NameNodes, each managing a separate portion of the file system (e.g., /user, /data, /tmp). These NameNodes are independent and don't require coordination, but they share the same pool of DataNodes. This horizontally scales the metadata layer and provides better isolation, though it does require a client-side mount table or ViewFS to navigate the namespaces.

   Answered 2024-09-17 by Margaret Hall

Does HDFS Federation also provide High Availability automatically, or do we still need to set up Standby NameNodes for each individual namespace we create?

   Answered 2024-09-19 by Richard White

• Federation and High Availability (HA) are separate features, Richard. Federation gives you multiple "Active" NameNodes for scaling, but each of those should ideally have its own "Standby" NameNode and Quorum Journal Manager setup. So, if you have two namespaces, you would technically have four NameNode processes running to ensure both scalability and 24/7 fault tolerance for each part of the file system.

     Commented 2024-09-20 by Joseph Moore

We implemented Federation last year and used ViewFS to make it transparent to our users. It was complex to set up but solved our NameNode GC pressure immediately.

   Answered 2024-09-21 by Mary Young

• That’s encouraging to hear, Mary. The ViewFS layer seems to be the key to making the multi-namespace setup usable for the end-user developers and data scientists.

     Commented 2024-09-25 by Jennifer Lee