Project 3

Part 3.1: Defining Correspondences

In this part, we define keypoints/correspondences on two faces, which can be thought of as facial details that we would like the warping mechanism to pay special attention to. The goal is to create a midway, and then later a morph, between the following two images of Robert Deniro and George Clooney:

[The images are taken from the photographs by Martin Schoeller. While it was recommended to do the first few parts with your own face, I found it much simpler to take a photo which had enough detail and was aligned well to focus on the face. Photographs by the same artist means similarity, and easier warping]
A total of 60 points were chosen to build the correspondence, giving arrays \(\text{arr1}\) and \(\text{arr2}\) of length 60 each. We compute the average between the two image keypoints as \(\frac{1}{2}( \text{arr1} + \text{arr2} ) \). We then use scipy.spatial.Delaunay to compute a triangulation of the averaged points, which gives us the average shape between the two faces. I implemented a custom labelling tool using ginput. In the next part, we use this average triangulation to compute the midway face.
To see results, click on the following link:
Part 3.1 Results

Part 3.2: Computing the Midway face

Computing Affine Transformations between simplices

Let \(T = T_1 \sqcup \cdots \sqcup T_n \) and \(T' = T_1' \sqcup \cdots \sqcup T_n'\) be triangulations of two images. Then between triangles \(T_i\) and \(T_i'\), we wish to find an affine transformation \(A_i: T_i \to T_i'\) such that the vertices of \(T_i\) are mapped to the vertices of \(T_i'\). If \((x_j, y_j)\) and \((x_j', y_j')\) for \(j=1, 2, 3\) are the vertices of \(T_i\) and \(T_i'\) respectively, then \[ \begin{bmatrix} x_1' & x_2' & x_3' \\ y_1' & y_2' & y_3' \\ 1 & 1 & 1 \end{bmatrix} = \underbrace{\begin{bmatrix} a & b & c \\ d & e & f \\ 0 & 0 & 1 \end{bmatrix}}_{=:A_i} \begin{bmatrix} x_1 & x_2 & x_3 \\ y_1 & y_2 & y_3 \\ 1 & 1 & 1 \end{bmatrix} . \] Since the points form a triangle, the matrices other than \(A_i\) are invertible, and this is how we compute the affine transformation. In reality, we will store \[ A_i^{-1} = \begin{bmatrix} x_1 & x_2 & x_3 \\ y_1 & y_2 & y_3 \\ 1 & 1 & 1 \end{bmatrix} \begin{bmatrix} x_1' & x_2' & x_3' \\ y_1' & y_2' & y_3' \\ 1 & 1 & 1 \end{bmatrix}^{-1} \] since that will be used for the inverse warp later.

Inverse Warp

Let \(f\) be an image and let \(T = T_1 \sqcup \cdots \sqcup T_n\) be a triangulation for it. Let \(T' = T_1' \sqcup \cdots \sqcup T_n'\) be the triangulation we want to warp \(f\) into. We have transformations \(A_1^{-1}, \ldots, A_n^{-1}\) where \(A_i^{-1} : T_i' \to T_i\), available for the inverse warp. To compute the warped image \(f'\) at coordinates \((x', y') \in T_i'\), we have \[ f'(x', y') = f(T_i^{-1}(x', y')). \] To address the case that \(T_i^{-1}(x', y')\) is not an integer, we use interpolation. We loop over all triangulations \(T_i\), evaluating \(f'\) on all points inside the triangulation. To implement this partition of space into triangulations, we will use polygon masks using skimage.draw.polygon applied to the image to piece-by-piece compute the inverse warped image.

Results

After inverse warping across all simplices, we have our warped image, which in this case is the midway face for Goerge Clooney and Robert Deniro. To see results, click the following link:
Part 3.2 Results

Part 3.3: Morph Sequences

Introduction

Any particular instance of the morph sequence, as described in the project outline, is given by the function morph_fixed(im1, pts_arr1, im2, pts_arr2, simplices, warp_frac, dissolve_frac, channel), and produces a morphing .gif file that takes im1 to im2 as warp_frac and dissolve_frac go from 0 to 1. In future parts, we will also explore the impact of differently varying warp_frac and dissolve_frac.

Results

We present a morph of Robert Deniro into George Clooney. Click the following link for results.
Part 3.3 Results

Part 3.4: "Mean Face" of a population.

We use the FEI Face Database, which is a Brazilian Face Database consisting of 400 images of 200 distinct people: each person has one image with a neutral face, and then another with a smiling face. We generate both the mean face and mean shape of the population.
After that, we will show several faces from the FEI Face Database and my own face getting warped into the mean face. We will then see a much cooler morphing effect, which looks goofy for George Clooney due to images being off-aligned.
Click the following link for results.
Part 3.4 Results

Part 3.5: Caricatures.

In a certain sense, the difference image between myself and the average neutral expression woman is the features that I am missing to be a woman. Thus by adding those features with a greater weight \(\alpha\), we can generate caricatures.
Click the following link for results.
Part 3.5 Results