Setting Up Spark Development Environment on IntelliJ IDE
Here are the Steps which can be followed to Setup Spark Development environment on IntelliJ IDEA IDE Pre-requisites: Download and Install IntelliJ IDE on Mac / Windows Environment https://www.jetbrains.com/idea/download/?section=mac Install Java JDK on Mac or Download tar archive and keep in some path Setup Procedure: Create New Project by File >> New Project >> And Choose "Python Project" If you are not seeing "Python" as an option, try to install plugin by Clicking "More Via Plugins" on bottom left and search for "Python" Creating the project with above said procedure will automatically create Project Structure as below Create requirements.txt file with Desired pyspark versions. Available pyspark releases can be found here https://pypi.org/project/pyspark/#history Open Terminal from the IDE (Bottom left Icon) and Activate the Virtual environment Install pyspark package from the requirements.txt file Once necessary packages ins...