Lab1: Intro to Data and Python Data Visualization using Iris Data¶

In this notebook we'll play with IRIS data, an introductory datset for Data Learners. Here we will plot Iris data using python. Get your hands dirty!

Data Set Information:¶

This is perhaps the best known database to be found in the data science literature. The data set contains 3 classes of 50 instances each, where each class refers to a type of iris plant. One class is linearly separable from the other 2; the latter are NOT linearly separable from each other.

Attribute Information:¶

1. sepal length in cm
2. sepal width in cm
3. petal length in cm
4. petal width in cm
5. class: -- Iris Setosa -- Iris Versicolour -- Iris Virginica

Check wikipedia for details - https://en.wikipedia.org/wiki/Iris_flower_data_set

We'll use three libraries for this tutorial: pandas, matplotlib, and seaborn.¶

Geting Data¶

First Import libraries¶

Read data¶

Take a look at Data¶

This is the first thing you should do when you start to work on any dataset. Just one line of code!

Dig some information on features, rows and columns¶

Here we've total 150 rows and 5 columns.

Check more

Descriptive Statistics¶

Observe data from statistics

Data preprocessing¶

1. Handling missing values¶

In this step, we aim to address missing values within the numerical columns of our dataset.

• Please employ the DataFrame.fillna() method, leveraging the mean of each column as a replacement for its missing values.

2. Outlier Detection¶

In this step, we will focus on identifying and removing outliers from the numerical columns of our dataset using the Local Outlier Factor (LOF) model. LOF is an unsupervised algorithm that detects outliers by measuring the local deviation of a given data point with respect to its neighbors. The LocalOutlierFactor class from sklearn.neighbors provides the functionality to detect outliers in a dataset.

• First, Read through the provided documentation, especially focusing on the "Fit the model for outlier detection (default)" section and the usage of the fit_predict method.

• Then, apply what you've learned: Use the LOF model to first identify outliers in your dataset and then remove them.

3. Data normalization¶

StandardScaler from sklearn.preprocessing is a powerful and easy-to-use tool for applying z-score normalization to your dataset.

• Please apply z-score normalization to the numerical columns in your dataset.

Is it a balanced dataset? Let's count example of each species in dataset.

You'll see 3 species ['versicolor', 'virginica', 'setosa'] and 50, 46, 45 examples respectively.

Let's Visualize Iris data and get some insight on our dataset!¶

Python has lots of visualization tools from Pandas, Matplotlib and Seaborn Libraries. We'll use couples of them to visualize data. You'll find data visualization quite interesting. The interesting fact is that we will be able to make intuition on Data without calculating/applying any algorthim. Let's start!

1. Exploring the Relationship between Sepal Length and Width¶

One of the first aspects we'll explore is the relationship between Sepal Length and Width among the different Iris species. We'll use a scatter plot for this purpose, which is excellent for spotting correlations between two variables.

Pandas offers a straightforward approach to creating scatter plots directly from a DataFrame. The DataFrame.plot.scatter() method is designed for this very purpose. Here's a quick reference to the documentation: Pandas DataFrame.plot.scatter.

• Please read the documentation of DataFrame.plot.scatter(), and then analyze the relationship between Sepal Length and Width using a scatter plot.

We'll check similar type plotting using seaborn after some moments!

2. Delving into Histograms¶

Next, we dive into histograms to get a distribution overview of the Iris features. Histograms are excellent for understanding the underlying distribution of data points across different intervals. Understanding Histograms：

1. Bins: The range of values is divided into a series of intervals — these are the bins. Bins are usually specified as consecutive, non-overlapping intervals of a variable.
2. Frequency: The number of data points that fall within each bin's range. In a histogram, the frequency for each bin is represented by the height of the bar, so a taller bar means a higher frequency of data points in that interval.

DataFrame.hist() method in Pandas helps creat histograms from a DataFrame. Here's a quick reference to the documentation: Pandas DataFrame.hist.

• Please read the documentation of DataFrame.hist(), and then plot the histograms.

3. Discovering Box plots¶

Let's try another cool pandas plotting. Yes, BOX PLOT! Boxplots provide a compact representation of data distributions, highlighting the median, quartiles, and outliers.

DataFrame.boxplot() method in Pandas makes a box plot from DataFrame columns.Here's a quick reference to the documentation: Pandas DataFrame.boxplot.

• Please read the documentation of DataFrame.boxplot(), and then plot the box plots for each numerical column.

4. Let's try Seaborn for jointplot and pairplot¶

Seaborn has many nice functions for plotting. You can create jointplot, hisplot, pairplot, boxplot, violinplot (it's damn cool), heatmap polt etc. I recommend seaborn personally cause it has many cool ploting techniques!

seaborn jointplot on sepal_length vs sepal_width¶

sns.jointplot in the Seaborn library allows users to simultaneously display the distribution of two variables along with their relationship. This provides an intuitive perspective for analyzing the relationship between two variables while also showcasing their individual univariate distributions.

Here's a quick reference to the documentation: Seaborn seaborn.jointplot.

Pairplot which shows the bivariate relation among features¶

sns.pairplot from the Seaborn library designed for plotting pairwise relationships in a dataset. When you have a DataFrame with multiple quantitative variables, sns.pairplot can be an invaluable tool for quickly visualizing how each pair of variables relates to one another across the entire dataset. Additionally, it can also plot the univariate distribution of each variable on the diagonal.

Here's a quick reference to the documentation: Seaborn seaborn.pairplot.

5. Parallel coordinates - high dimensional data visualization¶

Parallel coordinates are a powerful multivariate visualization technique. This method allows us to plot each feature of our dataset on a separate column and then connects these features with lines for each data sample. This approach is particularly useful for visualizing and understanding high-dimensional data, enabling us to spot patterns and relationships across multiple dimensions simultaneously.

We can use plotting.parallel_coordinates method in pandas library to make parallel coordinate plot. Here's a quick reference to the documentation: pandas.plotting.parallel_coordinates.

6. Correlations among feautres¶

Understanding how features in your dataset relate to each other is crucial for many analytical tasks.

The Pearson correlation coefficient is a measure that helps in quantifying the linear relationship between variables. In Pandas, the .corr() method conveniently calculates the Pearson correlation coefficient between every pair of numerical features in a DataFrame, producing a correlation matrix.

A heatmap is an excellent tool for visually representing the correlation matrix. It uses color coding to represent different values, making it easier to discern the strength and direction of the relationships between features at a glance. For creating heatmaps, we turn to the seaborn.heatmap() menthod in Seaborn library. Here's a quick reference to the documentation: seaborn.heatmap.

From above figure it's clear that Sepal Length and Sepal Width show weak correlations whereas Petal Width and Petal Length show strong correlations. It suggests that the Species can be identified better using Petal compared to Sepal. We can verify it using Machine Learning!

Self Practice¶

Try to repeat the above steps for data preprocessing and visualization on the Penguins dataset. You can find the dataset in the seaborn package.