Lab2: Dimensionality Reduction in Face Recognition¶

forms the foundation of any machine learning algorithm, without it, Data Science can not happen. Sometimes, it can contain a huge number of features, some of which are not even required. Such redundant information makes modeling complicated. Furthermore, interpreting and understanding the data by visualization gets difficult because of the high dimensionality. This is where dimensionality reduction comes into play.

In this lab you will engage in a series of tasks:

1. TASK 1: Visualize sample face images
2. TASK 2: Dimensionality reduction and visualization
3. TASK 3: Analyzing PCA's Role in Face Recognition Through Image Reconstruction
4. [Optional] Task 4: Evaluating the Impact of Dimensionality Reduction on Face Recognition Performance
5. [Optional] Task 5: Carry out PCA without scikit learn

Please note that areas in your notebook requiring code implementation are marked with # TODO.

Loading the dataset¶

You will use the Labeled Faces in the Wild (LFW) dataset, which is a widely-used benchmark dataset for studying face recognition problems. The dataset contains face images of various individuals collected from the internet, with variations in pose, lighting, and expression.

Fallback for dataset loading¶

If you encounter issues loading the dataset directly via the API, you can download it from the e-learning platform and then load it from a local file:

TASK 1: Visualize sample face images¶

To get a sense of the data we are working with, let's visualize the first 10 face images along with their corresponding labels.

Task Requirements:¶

• Figure Creation: Set up a figure that can comfortably fit 10 images. A recommended layout for this would be 2 rows by 5 columns.
• Image Iteration: Go through the first 10 images in the dataset, displaying each one on the figure with its associated label.
• Image Display: Render each image in grayscale and use the individual's name from the dataset as the title for its subplot.

Implementation Tips:¶

• The figsize parameter in the matplotlib.pyplot.figure() function allows you to specify the overall size of the figure. Adjust this to ensure that all subplots fit well within the display area. Check out the documentation for matplotlib.pyplot.figure for more guidance.

• To efficiently create a figure with a set of subplots, matplotlib.pyplot.subplots() can be a handy function. This not only initializes the figure but also returns an array of axes objects that can be used to plot individual subplots. Check out the documentation for matplotlib.pyplot.subplots for more guidance.

TASK 2: Dimensionality reduction and visualization¶

In this section, we aim to apply dimensionality reduction techniques learned from the previous class to our dataset.

Data preparation¶

Let's first create a pandas DataFrame named df that contains our features X and the target y. Then, we'll filter this DataFrame to retain only the data points corresponding to specific targets of interest, in this case, targets 0 and 5.

You are required perform dimensionality reduction on the prepared data and visualize the data points. Complete the code to achieve the following objectives:

1. PCA Transformation:

   • Extract the feature data from the DataFrame df (i.e., omitting the 'label' column)
   • Implement the following algorithms to reduce the feature data to two dimensions: (1) Principal Component Analysis (PCA), (2) Multidimensional Scaling (MDS), (3) Locally Linear Embedding (LLE), (4) Laplacian Eigenmaps and (5) t-Distributed Stochastic Neighbor Embedding (t-SNE).
   • [Optional] Optionally, to make computation more efficient for methods like t-SNE, you can first reduce the dimensionality using PCA.

2. Data Visualization:

   • Display the data points in a two-dimensional space, as transformed by each dimensionality reduction method. Use different colors to distinguish between the classes (label 0 and label 5).

Task 3: Analyzing PCA's Role in Face Recognition Through Image Reconstruction¶

In this section, we will investigate how PCA can be used for dimensionality reduction in face recognition and its impact on image reconstruction quality.

Your objective is to use PCA for reducing the dimensionality of a dataset of images and then to reconstruct these images from the reduced-dimensional space.

Basically, PCA operates as $$Z=WX.$$

For reconstruction, the process is $$W^{-1}Z=X.$$

Given that the matrix $W$ is orthonormal (they are the eigenvectors!), it simplifies to $W^{T}Z=X,$ where $W^{T}Z$ represents the reconstructed image.

Specifically, the inverse_transform method of the PCA object will be utilized to revert the PCA-transformed data back to its original space.

1. Dimensionality reduction via PCA:

   • First, you'll need to determine the number of principal components to retain, then fit PCA on your data and transform it. You can try to change the number of components, see what happens to the reconstructed image.

2. Image reconstruction:

   • Use the inverse_transform method of the PCA object to revert the PCA-transformed data back to its original space, then reshape it to obtain the reconstructed images. You can refer to the document: sklearn.decomposition.PCA.inverse_transform()

3. Visualize reconstructed images:

   • Compare one of the original images with the reconstructed one by visualizing them side by side.

[Optional] Task 4: Evaluating the Impact of Dimensionality Reduction on Face Recognition Performance¶

In this optional task, you will assess how dimensionality reduction affects the efficiency of a face recognition classifier. Follow these steps to conduct your evaluation:

1. Apply dimensionality reduction: Utilize dimensionality reduction methods to transform the high-dimensional face data into a more manageable, lower-dimensional representation. For this task, consider using PCA.

2. Data Splitting: Properly partition your dataset into training and testing sets to effectively assess the model's performance.

2. Classifier training & performance comparison: Implement and train a Support Vector Machine (SVM) classifier (or any classifier of your choice) on both the original high-dimensional and reduced-dimensionality data to see the impact of dimensionality reduction on learning. Evaluate the classifier's performance on each data set by comparing accuracy and running time.

Example code is provided for training SVM with original high-dimensional data, focusing on accuracy and running time. Extend this to include training on reduced-dimensional data and report accuracy and running time. By completing these two tasks, you'll gain insights into the trade-offs between computational efficiency and classifier performance brought about by dimensionality reduction in face recognition tasks.

By comparing the classification results of the original data and the PCA-reduced data, we can draw the following conclusions:

1. Impact of Dimensionality Reduction on Classification Performance: If the classification accuracy of the PCA-reduced data is close to or exceeds that of the original data, this indicates that sufficient information has been retained for effective classification while reducing the dimensions.

2. Efficiency Improvement via Dimensionality Reduction: Dimensionality reduction is especially useful for handling high-dimensional data, as it can reduce computational complexity and avoid overfitting.

[Optional] Task 5: Carry out PCA without scikit learn¶

In this section, we will implement PCA from scratch without using scikit-learn. This will help you understand the underlying mathematics and concepts behind PCA.

To manually implement the PCA algorithm, you need to understand the basic steps of PCA:

1. Centering:

• Calculate the mean of each feature.
• Subtract this mean from each feature to center the data around zero.
• This ensures that PCA deals with variance and covariance without being affected by the absolute values of features.

2. Covariance Matrix:

• Compute the covariance matrix of the features.
• You can use np.cov() to compute the covariance matrix of the centered features.

3. Eigenvalues and Eigenvectors:

• Solve for the eigenvalues and corresponding eigenvectors of the covariance matrix.
• You can utilize np.linalg.eigh() to compute the eigenvalues and eigenvectors of the covariance matrix.

4. Select Principal Components:

• Order the eigenvalues from highest to lowest to prioritize the most significant features.
• Select the top k eigenvectors based on the sorted eigenvalues; you can use np.argsort().
• These selected eigenvectors define your new feature space, capturing the most variance.

5. Project Data:

• Project the original data onto the new feature space to get the reduced-dimension data.
• You can use np.dot() to compute the product of two arrays

Let's do it!

This code reduces the dataset X to n_components principal components. You can adjust the number of principal components as needed.

Now, let's compare the image before using the manual implemented PCA and after using the manual implemented PCA.