Setting Up Spark Development Environment on IntelliJ IDE

-
Here are the Steps which can be followed to Setup Spark Development environment on IntelliJ IDEA IDE 
  
Pre-requisites:

Setup Procedure:

  • Create New Project by File >> New Project >> And Choose "Python Project"
  • If you are not seeing "Python" as an option, try to install plugin by Clicking "More Via Plugins" on bottom left and search for "Python"


  • Creating the project with above said procedure will automatically create Project Structure as below



  • Open Terminal from the IDE (Bottom left Icon) and Activate the Virtual environment
  • Install pyspark package from the requirements.txt file

  • Once necessary packages installed, run "deactivate" on the same terminal
  • Create new Python File under the root of project





Comments

Post a Comment

Popular posts from this blog

Querying MapR DB tables from Drill

-
1. Create a storage plugin for maprDB in drill Web UI {     "type" : "hbase",      "enabled" : true,               config" : { "hbase.table.namespace.mappings" : "*:/tables" #Change the path where you have mapr tables } } 2. Connect to drill ./sqline !connect jdbc:drill:zk=10.10.71.12:5181   sqlline> !connect jdbc:drill:zk=10.10.71.12:5181 scan complete in 406ms Enter username for jdbc:drill:zk=10.10.71.12:5181: mapr Enter password for jdbc:drill:zk=10.10.71.12:5181: **** 0: jdbc:drill:zk=10.10.71.12:5181> 3. Change the storage plugin to maprdb (or the name you have given) 0: jdbc:drill:zk=10.10.71.12:5181> use maprdb; +-------+-------------------------------------+ |  ok   |               summary               | +-------+-------------------------------------+ | true ...
Read more

OpenSSL Certificate creations

-
Create Self Signed Certificate Here are the steps involved in creating self signed certificate a) Create a key b) Generate Certificate Sign request .csr c) Sign your certificate (As it is self signed) a) Create a key Create a directory first (to avoid confusions) #mkdir certificates #cd certificates Generate key file #openssl genrsa 1024 > private.key b) Generate Certificate Sign request .csr #openssl req -new -key private.key > certrequest.csr c) Sign your certificate (As it is self Signed) #openssl x509 -in certrequest.csr -out selfsigned.crt -req -signkey private.key -days 365 Here the -days is the number of days denotes the validity of certificate Once after this we can get the certificate as file "selfsigned.crt"
Read more

HTTP Fs Installation

-
Some steps are specific to MapR Cluster. Ignore those steps. 1. Setup Yum repositories for MapR 2. Check for the packages           yum list mapr-httpfs 3. Install ==> yum install mapr-httpfs 4. Configure the core-site.xml and httpfs-site.xml properties to add impersonations                                                 <property>         <name>hadoop.proxyuser.#HTTPFSUSER#.hosts</name>         <value>httpfs-host.foo.com</value>        </property>         <property>             <name>hadoop.proxyuser.#HTTPFSUSER#.groups</name>     ...
Read more