KAFKA commands - 1

-

A) Create KAFKA Topics

Command Format

#kafka-topics --create --topic [TOPIC NAME] --zookeeper [ZOOKEEPER HOSTNAME] --replication-factor [NUMBER OF REPLICAS] --partitions [NUMBER OF PARTITIONS]


Example:

#kafka-topics --create --topic mytesttopic --zookeeper myzkhost.cluster.com --replication-factor 2 --partitions 3

Here:-

[TOPIC NAME = the name of the topic which is to be created ],
[ZOOKEEPER HOSTNAME = zookeeper hostname ]
[NUMBER OF REPLICAS = number of replications for each partitions, It should not be greater than the number of Brokers ]
[NUMBER OF PARTITIONS = Number of partitions for a topic ]


B) Describe a topic

 Command Format

 #kafka-topics --describe --topic [ TOPIC NAME ] --zookeeper [ZOOKEEPER HOSTNAME ]

Example

#kafka-topics --describe --topic mytesttopic --zookeeper myzkhost.cluster.com


Topic:mytesttopic PartitionCount:3 ReplicationFactor:2 Configs:
Topic: mytesttopic Partition: 0 Leader: 10 Replicas: 10,30 Isr: 10,30
Topic: mytesttopic Partition: 1 Leader: 20 Replicas: 20,10 Isr: 20,10
Topic: mytesttopic Partition: 2 Leader: 30 Replicas: 30,20 Isr: 30,20
Here in this output:

Leader = Leader partition residing on the particular broker ID
Replicas= Other brokers where the particular partition copies are hosted
ISR= In-Sync-Replicas, (the brokers which hosts the In-Sync copy of the same partition)

Location: Chennai, Tamil Nadu, India

Comments

Post a Comment

Popular posts from this blog

Querying MapR DB tables from Drill

-
1. Create a storage plugin for maprDB in drill Web UI {     "type" : "hbase",      "enabled" : true,               config" : { "hbase.table.namespace.mappings" : "*:/tables" #Change the path where you have mapr tables } } 2. Connect to drill ./sqline !connect jdbc:drill:zk=10.10.71.12:5181   sqlline> !connect jdbc:drill:zk=10.10.71.12:5181 scan complete in 406ms Enter username for jdbc:drill:zk=10.10.71.12:5181: mapr Enter password for jdbc:drill:zk=10.10.71.12:5181: **** 0: jdbc:drill:zk=10.10.71.12:5181> 3. Change the storage plugin to maprdb (or the name you have given) 0: jdbc:drill:zk=10.10.71.12:5181> use maprdb; +-------+-------------------------------------+ |  ok   |               summary               | +-------+-------------------------------------+ | true ...
Read more

OpenSSL Certificate creations

-
Create Self Signed Certificate Here are the steps involved in creating self signed certificate a) Create a key b) Generate Certificate Sign request .csr c) Sign your certificate (As it is self signed) a) Create a key Create a directory first (to avoid confusions) #mkdir certificates #cd certificates Generate key file #openssl genrsa 1024 > private.key b) Generate Certificate Sign request .csr #openssl req -new -key private.key > certrequest.csr c) Sign your certificate (As it is self Signed) #openssl x509 -in certrequest.csr -out selfsigned.crt -req -signkey private.key -days 365 Here the -days is the number of days denotes the validity of certificate Once after this we can get the certificate as file "selfsigned.crt"
Read more

HTTP Fs Installation

-
Some steps are specific to MapR Cluster. Ignore those steps. 1. Setup Yum repositories for MapR 2. Check for the packages           yum list mapr-httpfs 3. Install ==> yum install mapr-httpfs 4. Configure the core-site.xml and httpfs-site.xml properties to add impersonations                                                 <property>         <name>hadoop.proxyuser.#HTTPFSUSER#.hosts</name>         <value>httpfs-host.foo.com</value>        </property>         <property>             <name>hadoop.proxyuser.#HTTPFSUSER#.groups</name>     ...
Read more