Concept
In this project my two main objectives were:
- Determine the industry factors that are most important in predicting the salary amounts for these data.
- Determine the factors that distinguish job categories and titles from each other; for example, can required skills accurately predict job title?
This required scraping and preparing the data.
Data
Firstly, I scraped the data from the site using BeautifulSoup4 to get job listings for the titles "Data Scientist", "Data Analyst" and "Data Engineer", ending up with a data frame of shape 3210 × 5. The 5 columns are Title, Company, Salary, State, and Description. As most of the data did not have salary values, those rows needed to be dropped because the problem statement is based on salary. After loading into the DataFrame, I started analysing the data through EDA and feature engineering, ending up with a clean dataset of size 285 × 5.
In the exploratory data analysis phase, I found that most of the jobs are from NSW when compared to all other states. The highest salary in the dataset is for the title "Head of Data Science", paying 300K PA and based in NSW. The median salary is 95K PA. Most of the job postings were of title Data Analyst, followed by Data Science and Data Engineer. I feature engineered the title column to extract senior, mid, and junior levels, and added a skills column by extracting skills from the description.
Process
In order to analyze the data and generate insights out of it, I followed a process that looked at the data using the following visualizations:
- Job postings across all states
- Exploratory Data Analysis using Tableau
- Boxplots to see outliers
- Confusion matrix
- Scatter plot
- Bipartite graph
- ROC curve