Hadoop Admin's Blog

#Edit the logback.xml under the configuration dirctory #Modify / add the below property <logger name="org.apache.drill" additivity="false">     <level value="DEBUG" />     <appender-ref ref="FILE" />   </logger>   <logger name="query.logger" additivity="false">     <level value="DEBUG" />     <appender-ref ref="QUERY" />     <!--     <appender-ref ref="SOCKET" /> -->   </logger>     #Modify the logger level as debug  #Kill / restart the drill-bits   #Now you must be able to see the drillbits generating debug log

Some steps are specific to MapR Cluster. Ignore those steps. 1. Setup Yum repositories for MapR 2. Check for the packages           yum list mapr-httpfs 3. Install ==> yum install mapr-httpfs 4. Configure the core-site.xml and httpfs-site.xml properties to add impersonations                                                 <property>         <name>hadoop.proxyuser.#HTTPFSUSER#.hosts</name>         <value>httpfs-host.foo.com</value>        </property>         <property>             <name>hadoop.proxyuser.#HTTPFSUSER#.groups</name>     ...

1. Install dependencies httpd php gcc, glibc-common gd,gd-devel make net-snmp 2. Create user and group for Nagios useradd nagios groupadd nagcmd usermod -G nagcmd nagios usermod -G nagcmd apache 3. Download and extract nagios core 4. configure the nagios by running below ./configure --with-command-group=nagcmd make all make install 5.  make install-init   ==> To add services in init scripts 6.  make install-commandmode   ==> To use nagios in command mode 7.  make install-config   ==> To install the configuration files 8.  Configuration file locations /usr/local/nagios/etc/* /usr/local/nagios/etc/objects/contacts.cfg -> for contact options 9.  make install-webconf   => For installing httpd server configurations 10. Adding users to nagios htpasswd -s -c /usr/local/nagios/etc/htpasswd.users nagiosadmin 11. start the httpd server service httpd start systemctl start...

#Pre-requisites 1. OOzie must be up and running 2. Working zookeepers 3. Common database for both oozie instances ==================================================================== #Installing and configuring HA proxy ==================================================================== yum install haproxy #Edit the configuration vim "/etc/haproxy/haproxy.cfg" #Add following property # This is the setup for oozie server. Clients will connect to load_balancer_host:11000. # HAProxy will balance connections among the list of servers listed below. listen oozieha     bind 10.10.71.15:11001     mode tcp     option tcplog     balance roundrobin     server webserver01 10.10.71.13:11000 check     server webserver02 10.10.71.14:11000 check #start haproxy service service haproxy start service will be running in 11001 port ====================================================================

1. Here is the Sample fair-scheduler.xml file <allocations>   <queue name="root">     <minResources>10000 mb,0vcores</minResources>     <maxResources>90000 mb,0vcores</maxResources>     <maxRunningApps>50</maxRunningApps>     <maxAMShare>0.1</maxAMShare>     <weight>2.0</weight>     <schedulingPolicy>fair</schedulingPolicy>     <queue name="dev">       <aclSubmitApps>*</aclSubmitApps>       <minResources>5000 mb,0vcores</minResources>     </queue> <queue name="test">       <aclSubmitApps>*</aclSubmitApps>       <minResources>5000 mb,0vcores</minResources>     </queue> <queue name="prod">       <aclSubmitApps>*...

1. Create a storage plugin for maprDB in drill Web UI {     "type" : "hbase",      "enabled" : true,               config" : { "hbase.table.namespace.mappings" : "*:/tables" #Change the path where you have mapr tables } } 2. Connect to drill ./sqline !connect jdbc:drill:zk=10.10.71.12:5181   sqlline> !connect jdbc:drill:zk=10.10.71.12:5181 scan complete in 406ms Enter username for jdbc:drill:zk=10.10.71.12:5181: mapr Enter password for jdbc:drill:zk=10.10.71.12:5181: **** 0: jdbc:drill:zk=10.10.71.12:5181> 3. Change the storage plugin to maprdb (or the name you have given) 0: jdbc:drill:zk=10.10.71.12:5181> use maprdb; +-------+-------------------------------------+ |  ok   |               summary               | +-------+-------------------------------------+ | true ...

1. Configure kerberos client Copy the krb5.conf from to client /etc/krb5.conf (or) Edit the parameters as example shown below [mapr@srini-node1 bin]$ cat /etc/krb5.conf [logging] default = FILE:/var/log/krb5libs.log kdc = FILE:/var/log/krb5kdc.log admin_server = FILE:/var/log/kadmind.log [libdefaults] default_realm = EXAMPLE.COM dns_lookup_realm = false dns_lookup_kdc = false ticket_lifetime = 24h renew_lifetime = 7d forwardable = true [realms] EXAMPLE.COM = { kdc = krbserv.example.com admin_server = krbserv.example.com } [domain_realm] .example.com = EXAMPLE.COM krbserv.example.com = EXAMPLE.COM 2. Add the host principal for client in kerberos server addprinc mapr addprinc mapr/srini-node1.example.com Enter the password and confirm List the principal by using listprincs and make sure the principal present 3. Check in client ...