Configuring Fair Scheduler in YARN

1. Here is the Sample fair-scheduler.xml file

<allocations>

  <queue name="root">

    <minResources>10000 mb,0vcores</minResources>

    <maxResources>90000 mb,0vcores</maxResources>

    <maxRunningApps>50</maxRunningApps>

    <maxAMShare>0.1</maxAMShare>

    <weight>2.0</weight>

    <schedulingPolicy>fair</schedulingPolicy>

    <queue name="dev">

      <aclSubmitApps>*</aclSubmitApps>

      <minResources>5000 mb,0vcores</minResources>

    </queue>

<queue name="test">

      <aclSubmitApps>*</aclSubmitApps>

      <minResources>5000 mb,0vcores</minResources>

    </queue>

<queue name="prod">

      <aclSubmitApps>*</aclSubmitApps>

      <minResources>5000 mb,0vcores</minResources>

    </queue>

  </queue>

  <queueMaxAMShareDefault>0.5</queueMaxAMShareDefault>

  <!-- Queue 'secondary_group_queue' is a parent queue and may have

       user queues under it -->

  <queue name="secondary_group_queue" type="parent">

  <weight>3.0</weight>

  </queue>

  <user name="mapr">

    <maxRunningApps>30</maxRunningApps>

  </user>

  <userMaxAppsDefault>5</userMaxAppsDefault>

  <queuePlacementPolicy>

    <rule name="specified" />

    <rule name="primaryGroup" create="false" />

    <rule name="nestedUserQueue">

        <rule name="secondaryGroupExistingQueue" create="false" />

    </rule>

    <rule name="default" queue="sample_queue"/>

  </queuePlacementPolicy>

</allocations>

You need to copy this file as per your policies and place it in resource manager node.

2. Add below properties in yarn-site.xml

  <property>
       <name>yarn.resourcemanager.scheduler.class</name>
       <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
  </property>

  <property>

       <name>yarn.scheduler.fair.allocation.file</name>

       <value>/path/to/allocationfile</value>
  </property>

3. Restart the Resource manager(s)

       Now you should be able to see the queues running on the cluster, in Resource manager's web UI.